diff options
| author | Linus Torvalds <torvalds@linux-foundation.org> | 2022-12-13 11:59:58 -0800 |
|---|---|---|
| committer | Linus Torvalds <torvalds@linux-foundation.org> | 2022-12-13 11:59:58 -0800 |
| commit | a594533df0f6ca391da003f43d53b336a2d23ffa (patch) | |
| tree | ec984c693b0bfc208519c43134f21365797f90ee /drivers/gpu/drm/i915 | |
| parent | cdb9d3537711939e4d8fd0de2889c966f88346eb (diff) | |
| parent | 66efff515a6500d4b4976fbab3bee8b92a1137fb (diff) | |
| download | linux-a594533df0f6ca391da003f43d53b336a2d23ffa.tar.bz2 | |
Merge tag 'drm-next-2022-12-13' of git://anongit.freedesktop.org/drm/drm
Pull drm updates from Dave Airlie:
"The biggest highlight is that the accel subsystem framework is merged.
Hopefully for 6.3 we will be able to line up a driver to use it.
In drivers land, i915 enables DG2 support by default now, and nouveau
has a big stability refactoring and initial ampere support, AMD
includes new hw IP support and should build on ARM again. There is
also an ofdrm driver to take over offb on platforms it's used.
Stuff outside my tree, the dma-buf patches hit a few places, the vc4
firmware changes also do, and i915 has some interactions with MEI for
discrete GPUs. I think all of those should have been acked/reviewed by
relevant parties.
New driver:
- ofdrm - replacement for offb
fbdev:
- add support for nomodeset
fourcc:
- add Vivante tiled modifier
core:
- atomic-helpers: CRTC primary plane test fixes, fb access hooks
- connector: TV API consistency, cmdline parser improvements
- send connector hotplug on cleanup
- sort makefile objects
tests:
- sort kunit tests
- improve DP-MST tests
- add kunit helpers to create a device
sched:
- module param for scheduling policy
- refcounting fix
buddy:
- add back random seed log
ttm:
- convert ttm_resource to size_t
- optimize pool allocations
edid:
- HFVSDB parsing support fixes
- logging/debug improvements
- DSC quirks
dma-buf:
- Add unlocked vmap and attachment mapping
- move drivers to common locking convention
- locking improvements
firmware:
- new API for rPI firmware and vc4
xilinx:
- zynqmp: displayport bridge support
- dpsub fix
bridge:
- adv7533: Remove dynamic lane switching
- it6505: Runtime PM support, sync improvements
- ps8640: Handle AUX defer messages
- tc358775: Drop soft-reset over I2C
panel:
- panel-edp: Add INX N116BGE-EA2 C2 and C4 support.
- Jadard JD9365DA-H3
- NewVision NV3051D
amdgpu:
- DCN support on ARM
- DCN 2.1 secure display
- Sienna Cichlid mode2 reset fixes
- new GC 11.x firmware versions
- drop AMD specific DSC workarounds in favour of drm code
- clang warning fixes
- scheduler rework
- SR-IOV fixes
- GPUVM locking fixes
- fix memory leak in CS IOCTL error path
- flexible array updates
- enable new GC/PSP/SMU/NBIO IP
- GFX preemption support for gfx9
amdkfd:
- cache size fixes
- userptr fixes
- enable cooperative launch on gfx 10.3
- enable GC 11.0.4 KFD support
radeon:
- replace kmap with kmap_local_page
- ACPI ref count fix
- HDA audio notifier support
i915:
- DG2 enabled by default
- MTL enablement work
- hotplug refactoring
- VBT improvements
- Display and watermark refactoring
- ADL-P workaround
- temp disable runtime_pm for discrete-
- fix for A380 as a secondary GPU
- Wa_18017747507 for DG2
- CS timestamp support fixes for gen5 and earlier
- never purge busy TTM objects
- use i915_sg_dma_sizes for all backends
- demote GuC kernel contexts to normal priority
- gvt: refactor for new MDEV interface
- enable DC power states on eDP ports
- fix gen 2/3 workarounds
nouveau:
- fix page fault handling
- Ampere acceleration support
- driver stability improvements
- nva3 backlight support
msm:
- MSM_INFO_GET_FLAGS support
- DPU: XR30 and P010 image formats
- Qualcomm SM6115 support
- DSI PHY support for QCM2290
- HDMI: refactored dev init path
- remove exclusive-fence hack
- fix speed-bin detection
- enable clamp to idle on 7c3
- improved hangcheck detection
vmwgfx:
- fb and cursor refactoring
- convert to generic hashtable
- cursor improvements
etnaviv:
- hw workarounds
- softpin MMU fixes
ast:
- atomic gamma LUT support
- convert to SHMEM
lcdif:
- support YUV planes
- Increase DMA burst size
- FIFO threshold tuning
meson:
- fix return type of cvbs mode_valid
mgag200:
- fix PLL setup on some revisions
sun4i:
- A100 and D1 support
udl:
- modesetting improvements
- hot unplug support
vc4:
- support PAL-M
- fix regression preventing 4K @ 60Hz
- fix NULL ptr deref
v3d:
- switch to drm managed resources
renesas:
- RZ/G2L DSI support
- DU Kconfig cleanup
mediatek:
- fixup dpi and hdmi
- MT8188 dpi support
- MT8195 AFBC support
tegra:
- NVDEC hardware on Tegra234 SoC
hdlcd:
- switch to drm managed resources
ingenic:
- fix registration error path
hisilicon:
- convert to drm_mode_init
maildp:
- use managed resources
mtk:
- use drm_mode_init
rockchip:
- use drm_mode_copy"
* tag 'drm-next-2022-12-13' of git://anongit.freedesktop.org/drm/drm: (1397 commits)
drm/amdgpu: fix mmhub register base coding error
drm/amdgpu: add tmz support for GC IP v11.0.4
drm/amdgpu: enable GFX Clock Gating control for GC IP v11.0.4
drm/amdgpu: enable GFX Power Gating for GC IP v11.0.4
drm/amdgpu: enable GFX IP v11.0.4 CG support
drm/amdgpu: Make amdgpu_ring_mux functions as static
drm/amdgpu: generally allow over-commit during BO allocation
drm/amd/display: fix array index out of bound error in DCN32 DML
drm/amd/display: 3.2.215
drm/amd/display: set optimized required for comp buf changes
drm/amd/display: Add debug option to skip PSR CRTC disable
drm/amd/display: correct DML calc error of UrgentLatency
drm/amd/display: correct static_screen_event_mask
drm/amd/display: Ensure commit_streams returns the DC return code
drm/amd/display: read invalid ddc pin status cause engine busy
drm/amd/display: Bypass DET swath fill check for max clocks
drm/amd/display: Disable uclk pstate for subvp pipes
drm/amd/display: Fix DCN2.1 default DSC clocks
drm/amd/display: Enable dp_hdmi21_pcon support
drm/amd/display: prevent seamless boot on displays that don't have the preferred dig
...
Diffstat (limited to 'drivers/gpu/drm/i915')
303 files changed, 10734 insertions, 5414 deletions
diff --git a/drivers/gpu/drm/i915/Kconfig.profile b/drivers/gpu/drm/i915/Kconfig.profile index 39328567c200..7cc38d25ee5c 100644 --- a/drivers/gpu/drm/i915/Kconfig.profile +++ b/drivers/gpu/drm/i915/Kconfig.profile @@ -57,10 +57,28 @@ config DRM_I915_PREEMPT_TIMEOUT default 640 # milliseconds help How long to wait (in milliseconds) for a preemption event to occur - when submitting a new context via execlists. If the current context - does not hit an arbitration point and yield to HW before the timer - expires, the HW will be reset to allow the more important context - to execute. + when submitting a new context. If the current context does not hit + an arbitration point and yield to HW before the timer expires, the + HW will be reset to allow the more important context to execute. + + This is adjustable via + /sys/class/drm/card?/engine/*/preempt_timeout_ms + + May be 0 to disable the timeout. + + The compiled in default may get overridden at driver probe time on + certain platforms and certain engines which will be reflected in the + sysfs control. + +config DRM_I915_PREEMPT_TIMEOUT_COMPUTE + int "Preempt timeout for compute engines (ms, jiffy granularity)" + default 7500 # milliseconds + help + How long to wait (in milliseconds) for a preemption event to occur + when submitting a new context to a compute capable engine. If the + current context does not hit an arbitration point and yield to HW + before the timer expires, the HW will be reset to allow the more + important context to execute. This is adjustable via /sys/class/drm/card?/engine/*/preempt_timeout_ms diff --git a/drivers/gpu/drm/i915/Makefile b/drivers/gpu/drm/i915/Makefile index cea00aaca04b..01974b82d205 100644 --- a/drivers/gpu/drm/i915/Makefile +++ b/drivers/gpu/drm/i915/Makefile @@ -127,9 +127,11 @@ gt-y += \ gt/intel_sseu.o \ gt/intel_sseu_debugfs.o \ gt/intel_timeline.o \ + gt/intel_wopcm.o \ gt/intel_workarounds.o \ gt/shmem_utils.o \ gt/sysfs_engines.o + # x86 intel-gtt module support gt-$(CONFIG_X86) += gt/intel_ggtt_gmch.o # autogenerated null render state @@ -183,8 +185,7 @@ i915-y += \ i915_trace_points.o \ i915_ttm_buddy_manager.o \ i915_vma.o \ - i915_vma_resource.o \ - intel_wopcm.o + i915_vma_resource.o # general-purpose microcontroller (GuC) support i915-y += gt/uc/intel_uc.o \ @@ -209,6 +210,9 @@ i915-y += gt/uc/intel_uc.o \ # graphics system controller (GSC) support i915-y += gt/intel_gsc.o +# graphics hardware monitoring (HWMON) support +i915-$(CONFIG_HWMON) += i915_hwmon.o + # modesetting core code i915-y += \ display/hsw_ips.o \ @@ -244,6 +248,7 @@ i915-y += \ display/intel_global_state.o \ display/intel_hdcp.o \ display/intel_hotplug.o \ + display/intel_hti.o \ display/intel_lpe_audio.o \ display/intel_modeset_verify.o \ display/intel_modeset_setup.o \ @@ -310,15 +315,18 @@ i915-y += \ i915-y += i915_perf.o -# Protected execution platform (PXP) support -i915-$(CONFIG_DRM_I915_PXP) += \ +# Protected execution platform (PXP) support. Base support is required for HuC +i915-y += \ pxp/intel_pxp.o \ + pxp/intel_pxp_tee.o \ + pxp/intel_pxp_huc.o + +i915-$(CONFIG_DRM_I915_PXP) += \ pxp/intel_pxp_cmd.o \ pxp/intel_pxp_debugfs.o \ pxp/intel_pxp_irq.o \ pxp/intel_pxp_pm.o \ - pxp/intel_pxp_session.o \ - pxp/intel_pxp_tee.o + pxp/intel_pxp_session.o # Post-mortem debug and GPU hang state capture i915-$(CONFIG_DRM_I915_CAPTURE_ERROR) += i915_gpu_error.o diff --git a/drivers/gpu/drm/i915/display/g4x_dp.c b/drivers/gpu/drm/i915/display/g4x_dp.c index e3e3d27ffb53..3593938dcd87 100644 --- a/drivers/gpu/drm/i915/display/g4x_dp.c +++ b/drivers/gpu/drm/i915/display/g4x_dp.c @@ -8,6 +8,7 @@ #include <linux/string_helpers.h> #include "g4x_dp.h" +#include "i915_reg.h" #include "intel_audio.h" #include "intel_backlight.h" #include "intel_connector.h" diff --git a/drivers/gpu/drm/i915/display/g4x_dp.h b/drivers/gpu/drm/i915/display/g4x_dp.h index e1f50263a725..a38b3e1e01d3 100644 --- a/drivers/gpu/drm/i915/display/g4x_dp.h +++ b/drivers/gpu/drm/i915/display/g4x_dp.h @@ -8,7 +8,7 @@ #include <linux/types.h> -#include "i915_reg.h" +#include "i915_reg_defs.h" enum pipe; enum port; diff --git a/drivers/gpu/drm/i915/display/g4x_hdmi.c b/drivers/gpu/drm/i915/display/g4x_hdmi.c index 2b73f5ff0d02..121caeaa409b 100644 --- a/drivers/gpu/drm/i915/display/g4x_hdmi.c +++ b/drivers/gpu/drm/i915/display/g4x_hdmi.c @@ -6,6 +6,7 @@ */ #include "g4x_hdmi.h" +#include "i915_reg.h" #include "intel_audio.h" #include "intel_connector.h" #include "intel_crtc.h" @@ -78,6 +79,18 @@ static bool intel_hdmi_get_hw_state(struct intel_encoder *encoder, return ret; } +static int g4x_hdmi_compute_config(struct intel_encoder *encoder, + struct intel_crtc_state *crtc_state, + struct drm_connector_state *conn_state) +{ + struct drm_i915_private *i915 = to_i915(encoder->base.dev); + + if (HAS_PCH_SPLIT(i915)) + crtc_state->has_pch_encoder = true; + + return intel_hdmi_compute_config(encoder, crtc_state, conn_state); +} + static void intel_hdmi_get_config(struct intel_encoder *encoder, struct intel_crtc_state *pipe_config) { @@ -543,7 +556,7 @@ void g4x_hdmi_init(struct drm_i915_private *dev_priv, "HDMI %c", port_name(port)); intel_encoder->hotplug = intel_hdmi_hotplug; - intel_encoder->compute_config = intel_hdmi_compute_config; + intel_encoder->compute_config = g4x_hdmi_compute_config; if (HAS_PCH_SPLIT(dev_priv)) { intel_encoder->disable = pch_disable_hdmi; intel_encoder->post_disable = pch_post_disable_hdmi; @@ -585,7 +598,7 @@ void g4x_hdmi_init(struct drm_i915_private *dev_priv, } else { intel_encoder->pipe_mask = ~0; } - intel_encoder->cloneable = 1 << INTEL_OUTPUT_ANALOG; + intel_encoder->cloneable = BIT(INTEL_OUTPUT_ANALOG); intel_encoder->hpd_pin = intel_hpd_pin_default(dev_priv, port); /* * BSpec is unclear about HDMI+HDMI cloning on g4x, but it seems @@ -593,7 +606,7 @@ void g4x_hdmi_init(struct drm_i915_private *dev_priv, * only one port anyway, nothing is lost by allowing it. */ if (IS_G4X(dev_priv)) - intel_encoder->cloneable |= 1 << INTEL_OUTPUT_HDMI; + intel_encoder->cloneable |= BIT(INTEL_OUTPUT_HDMI); dig_port->hdmi.hdmi_reg = hdmi_reg; dig_port->dp.output_reg = INVALID_MMIO_REG; diff --git a/drivers/gpu/drm/i915/display/hsw_ips.c b/drivers/gpu/drm/i915/display/hsw_ips.c index a5be4af792cb..83aa3800245f 100644 --- a/drivers/gpu/drm/i915/display/hsw_ips.c +++ b/drivers/gpu/drm/i915/display/hsw_ips.c @@ -104,8 +104,7 @@ static bool hsw_ips_need_disable(struct intel_atomic_state *state, * Disable IPS before we program the LUT. */ if (IS_HASWELL(i915) && - (new_crtc_state->uapi.color_mgmt_changed || - new_crtc_state->update_pipe) && + intel_crtc_needs_color_update(new_crtc_state) && new_crtc_state->gamma_mode == GAMMA_MODE_MODE_SPLIT) return true; @@ -146,8 +145,7 @@ static bool hsw_ips_need_enable(struct intel_atomic_state *state, * Re-enable IPS after the LUT has been programmed. */ if (IS_HASWELL(i915) && - (new_crtc_state->uapi.color_mgmt_changed || - new_crtc_state->update_pipe) && + intel_crtc_needs_color_update(new_crtc_state) && new_crtc_state->gamma_mode == GAMMA_MODE_MODE_SPLIT) return true; @@ -155,7 +153,7 @@ static bool hsw_ips_need_enable(struct intel_atomic_state *state, * We can't read out IPS on broadwell, assume the worst and * forcibly enable IPS on the first fastset. */ - if (new_crtc_state->update_pipe && old_crtc_state->inherited) + if (intel_crtc_needs_fastset(new_crtc_state) && old_crtc_state->inherited) return true; return !old_crtc_state->ips_enabled; diff --git a/drivers/gpu/drm/i915/display/i9xx_plane.c b/drivers/gpu/drm/i915/display/i9xx_plane.c index 5afbe3e98ee8..ecaeb7dc196b 100644 --- a/drivers/gpu/drm/i915/display/i9xx_plane.c +++ b/drivers/gpu/drm/i915/display/i9xx_plane.c @@ -8,6 +8,9 @@ #include <drm/drm_blend.h> #include <drm/drm_fourcc.h> +#include "i915_irq.h" +#include "i915_reg.h" +#include "i9xx_plane.h" #include "intel_atomic.h" #include "intel_atomic_plane.h" #include "intel_de.h" @@ -15,7 +18,6 @@ #include "intel_fb.h" #include "intel_fbc.h" #include "intel_sprite.h" -#include "i9xx_plane.h" /* Primary plane formats for gen <= 3 */ static const u32 i8xx_primary_formats[] = { diff --git a/drivers/gpu/drm/i915/display/icl_dsi.c b/drivers/gpu/drm/i915/display/icl_dsi.c index ed4d93942dbd..d16b30a2dded 100644 --- a/drivers/gpu/drm/i915/display/icl_dsi.c +++ b/drivers/gpu/drm/i915/display/icl_dsi.c @@ -29,6 +29,7 @@ #include <drm/drm_atomic_helper.h> #include <drm/drm_mipi_dsi.h> +#include "i915_reg.h" #include "icl_dsi.h" #include "icl_dsi_regs.h" #include "intel_atomic.h" @@ -1974,16 +1975,8 @@ static void icl_dsi_add_properties(struct intel_connector *connector) { const struct drm_display_mode *fixed_mode = intel_panel_preferred_fixed_mode(connector); - u32 allowed_scalers; - allowed_scalers = BIT(DRM_MODE_SCALE_ASPECT) | - BIT(DRM_MODE_SCALE_FULLSCREEN) | - BIT(DRM_MODE_SCALE_CENTER); - - drm_connector_attach_scaling_mode_property(&connector->base, - allowed_scalers); - - connector->base.state->scaling_mode = DRM_MODE_SCALE_ASPECT; + intel_attach_scaling_mode_property(&connector->base); drm_connector_set_panel_orientation_with_quirk(&connector->base, intel_dsi_get_panel_orientation(connector), @@ -1993,7 +1986,6 @@ static void icl_dsi_add_properties(struct intel_connector *connector) void icl_dsi_init(struct drm_i915_private *dev_priv) { - struct drm_device *dev = &dev_priv->drm; struct intel_dsi *intel_dsi; struct intel_encoder *encoder; struct intel_connector *intel_connector; @@ -2018,7 +2010,7 @@ void icl_dsi_init(struct drm_i915_private *dev_priv) connector = &intel_connector->base; /* register DSI encoder with DRM subsystem */ - drm_encoder_init(dev, &encoder->base, &gen11_dsi_encoder_funcs, + drm_encoder_init(&dev_priv->drm, &encoder->base, &gen11_dsi_encoder_funcs, DRM_MODE_ENCODER_DSI, "DSI %c", port_name(port)); encoder->pre_pll_enable = gen11_dsi_pre_pll_enable; @@ -2042,12 +2034,10 @@ void icl_dsi_init(struct drm_i915_private *dev_priv) encoder->is_clock_enabled = gen11_dsi_is_clock_enabled; /* register DSI connector with DRM subsystem */ - drm_connector_init(dev, connector, &gen11_dsi_connector_funcs, + drm_connector_init(&dev_priv->drm, connector, &gen11_dsi_connector_funcs, DRM_MODE_CONNECTOR_DSI); drm_connector_helper_add(connector, &gen11_dsi_connector_helper_funcs); connector->display_info.subpixel_order = SubPixelHorizontalRGB; - connector->interlace_allowed = false; - connector->doublescan_allowed = false; intel_connector->get_hw_state = intel_connector_get_hw_state; /* attach connector to encoder */ @@ -2055,9 +2045,9 @@ void icl_dsi_init(struct drm_i915_private *dev_priv) intel_bios_init_panel(dev_priv, &intel_connector->panel, NULL, NULL); - mutex_lock(&dev->mode_config.mutex); + mutex_lock(&dev_priv->drm.mode_config.mutex); intel_panel_add_vbt_lfp_fixed_mode(intel_connector); - mutex_unlock(&dev->mode_config.mutex); + mutex_unlock(&dev_priv->drm.mode_config.mutex); if (!intel_panel_preferred_fixed_mode(intel_connector)) { drm_err(&dev_priv->drm, "DSI fixed mode info missing\n"); diff --git a/drivers/gpu/drm/i915/display/icl_dsi_regs.h b/drivers/gpu/drm/i915/display/icl_dsi_regs.h index f78f28b8dd94..d4845ac65acc 100644 --- a/drivers/gpu/drm/i915/display/icl_dsi_regs.h +++ b/drivers/gpu/drm/i915/display/icl_dsi_regs.h @@ -6,7 +6,7 @@ #ifndef __ICL_DSI_REGS_H__ #define __ICL_DSI_REGS_H__ -#include "i915_reg_defs.h" +#include "intel_display_reg_defs.h" /* Gen11 DSI */ #define _MMIO_DSI(tc, dsi0, dsi1) _MMIO_TRANS((tc) - TRANSCODER_DSI_0, \ diff --git a/drivers/gpu/drm/i915/display/intel_atomic.c b/drivers/gpu/drm/i915/display/intel_atomic.c index 18f0a5ae3bac..6621aa245caf 100644 --- a/drivers/gpu/drm/i915/display/intel_atomic.c +++ b/drivers/gpu/drm/i915/display/intel_atomic.c @@ -252,6 +252,11 @@ intel_crtc_duplicate_state(struct drm_crtc *crtc) if (crtc_state->hw.gamma_lut) drm_property_blob_get(crtc_state->hw.gamma_lut); + if (crtc_state->pre_csc_lut) + drm_property_blob_get(crtc_state->pre_csc_lut); + if (crtc_state->post_csc_lut) + drm_property_blob_get(crtc_state->post_csc_lut); + crtc_state->update_pipe = false; crtc_state->disable_lp_wm = false; crtc_state->disable_cxsr = false; @@ -274,6 +279,9 @@ static void intel_crtc_put_color_blobs(struct intel_crtc_state *crtc_state) drm_property_blob_put(crtc_state->hw.degamma_lut); drm_property_blob_put(crtc_state->hw.gamma_lut); drm_property_blob_put(crtc_state->hw.ctm); + + drm_property_blob_put(crtc_state->pre_csc_lut); + drm_property_blob_put(crtc_state->post_csc_lut); } void intel_crtc_free_hw_state(struct intel_crtc_state *crtc_state) diff --git a/drivers/gpu/drm/i915/display/intel_atomic_plane.c b/drivers/gpu/drm/i915/display/intel_atomic_plane.c index aaa6708256d5..10e1fc9d0698 100644 --- a/drivers/gpu/drm/i915/display/intel_atomic_plane.c +++ b/drivers/gpu/drm/i915/display/intel_atomic_plane.c @@ -425,6 +425,47 @@ static bool intel_plane_do_async_flip(struct intel_plane *plane, return DISPLAY_VER(i915) < 13 || old_crtc_state->uapi.async_flip; } +static bool i9xx_must_disable_cxsr(const struct intel_crtc_state *new_crtc_state, + const struct intel_plane_state *old_plane_state, + const struct intel_plane_state *new_plane_state) +{ + struct intel_plane *plane = to_intel_plane(new_plane_state->uapi.plane); + bool old_visible = old_plane_state->uapi.visible; + bool new_visible = new_plane_state->uapi.visible; + u32 old_ctl = old_plane_state->ctl; + u32 new_ctl = new_plane_state->ctl; + bool modeset, turn_on, turn_off; + + if (plane->id == PLANE_CURSOR) + return false; + + modeset = intel_crtc_needs_modeset(new_crtc_state); + turn_off = old_visible && (!new_visible || modeset); + turn_on = new_visible && (!old_visible || modeset); + + /* Must disable CxSR around plane enable/disable */ + if (turn_on || turn_off) + return true; + + if (!old_visible || !new_visible) + return false; + + /* + * Most plane control register updates are blocked while in CxSR. + * + * Tiling mode is one exception where the primary plane can + * apparently handle it, whereas the sprites can not (the + * sprite issue being only relevant on VLV/CHV where CxSR + * is actually possible with a sprite enabled). + */ + if (plane->id == PLANE_PRIMARY) { + old_ctl &= ~DISP_TILED; + new_ctl &= ~DISP_TILED; + } + + return old_ctl != new_ctl; +} + static int intel_plane_atomic_calc_changes(const struct intel_crtc_state *old_crtc_state, struct intel_crtc_state *new_crtc_state, const struct intel_plane_state *old_plane_state, @@ -482,17 +523,9 @@ static int intel_plane_atomic_calc_changes(const struct intel_crtc_state *old_cr if (turn_on) { if (DISPLAY_VER(dev_priv) < 5 && !IS_G4X(dev_priv)) new_crtc_state->update_wm_pre = true; - - /* must disable cxsr around plane enable/disable */ - if (plane->id != PLANE_CURSOR) - new_crtc_state->disable_cxsr = true; } else if (turn_off) { if (DISPLAY_VER(dev_priv) < 5 && !IS_G4X(dev_priv)) new_crtc_state->update_wm_post = true; - - /* must disable cxsr around plane enable/disable */ - if (plane->id != PLANE_CURSOR) - new_crtc_state->disable_cxsr = true; } else if (intel_wm_need_update(old_plane_state, new_plane_state)) { if (DISPLAY_VER(dev_priv) < 5 && !IS_G4X(dev_priv)) { /* FIXME bollocks */ @@ -504,6 +537,10 @@ static int intel_plane_atomic_calc_changes(const struct intel_crtc_state *old_cr if (visible || was_visible) new_crtc_state->fb_bits |= plane->frontbuffer_bit; + if (HAS_GMCH(dev_priv) && + i9xx_must_disable_cxsr(new_crtc_state, old_plane_state, new_plane_state)) + new_crtc_state->disable_cxsr = true; + /* * ILK/SNB DVSACNTR/Sprite Enable * IVB SPR_CTL/Sprite Enable @@ -720,7 +757,7 @@ void intel_plane_update_noarm(struct intel_plane *plane, { struct intel_crtc *crtc = to_intel_crtc(crtc_state->uapi.crtc); - trace_intel_plane_update_noarm(&plane->base, crtc); + trace_intel_plane_update_noarm(plane, crtc); if (plane->update_noarm) plane->update_noarm(plane, crtc_state, plane_state); @@ -732,7 +769,7 @@ void intel_plane_update_arm(struct intel_plane *plane, { struct intel_crtc *crtc = to_intel_crtc(crtc_state->uapi.crtc); - trace_intel_plane_update_arm(&plane->base, crtc); + trace_intel_plane_update_arm(plane, crtc); if (crtc_state->do_async_flip && plane->async_flip) plane->async_flip(plane, crtc_state, plane_state, true); @@ -745,7 +782,7 @@ void intel_plane_disable_arm(struct intel_plane *plane, { struct intel_crtc *crtc = to_intel_crtc(crtc_state->uapi.crtc); - trace_intel_plane_disable_arm(&plane->base, crtc); + trace_intel_plane_disable_arm(plane, crtc); plane->disable_arm(plane, crtc_state); } @@ -1005,7 +1042,7 @@ intel_prepare_plane_fb(struct drm_plane *_plane, */ if (intel_crtc_needs_modeset(crtc_state)) { ret = i915_sw_fence_await_reservation(&state->commit_ready, - old_obj->base.resv, NULL, + old_obj->base.resv, false, 0, GFP_KERNEL); if (ret < 0) @@ -1039,8 +1076,7 @@ intel_prepare_plane_fb(struct drm_plane *_plane, struct dma_fence *fence; ret = i915_sw_fence_await_reservation(&state->commit_ready, - obj->base.resv, NULL, - false, + obj->base.resv, false, i915_fence_timeout(dev_priv), GFP_KERNEL); if (ret < 0) diff --git a/drivers/gpu/drm/i915/display/intel_audio.c b/drivers/gpu/drm/i915/display/intel_audio.c index aacbc6da84ef..98c3322b4549 100644 --- a/drivers/gpu/drm/i915/display/intel_audio.c +++ b/drivers/gpu/drm/i915/display/intel_audio.c @@ -250,7 +250,7 @@ static const struct hdmi_aud_ncts hdmi_aud_ncts_36bpp[] = { /* get AUD_CONFIG_PIXEL_CLOCK_HDMI_* value for mode */ static u32 audio_config_hdmi_pixel_clock(const struct intel_crtc_state *crtc_state) { - struct drm_i915_private *dev_priv = to_i915(crtc_state->uapi.crtc->dev); + struct drm_i915_private *i915 = to_i915(crtc_state->uapi.crtc->dev); const struct drm_display_mode *adjusted_mode = &crtc_state->hw.adjusted_mode; int i; @@ -260,17 +260,17 @@ static u32 audio_config_hdmi_pixel_clock(const struct intel_crtc_state *crtc_sta break; } - if (DISPLAY_VER(dev_priv) < 12 && adjusted_mode->crtc_clock > 148500) + if (DISPLAY_VER(i915) < 12 && adjusted_mode->crtc_clock > 148500) i = ARRAY_SIZE(hdmi_audio_clock); if (i == ARRAY_SIZE(hdmi_audio_clock)) { - drm_dbg_kms(&dev_priv->drm, + drm_dbg_kms(&i915->drm, "HDMI audio pixel clock setting for %d not found, falling back to defaults\n", adjusted_mode->crtc_clock); i = 1; } - drm_dbg_kms(&dev_priv->drm, + drm_dbg_kms(&i915->drm, "Configuring HDMI audio for pixel clock %d (0x%08x)\n", hdmi_audio_clock[i].clock, hdmi_audio_clock[i].config); @@ -304,96 +304,67 @@ static int audio_config_hdmi_get_n(const struct intel_crtc_state *crtc_state, return 0; } -static bool intel_eld_uptodate(struct drm_connector *connector, - i915_reg_t reg_eldv, u32 bits_eldv, - i915_reg_t reg_elda, u32 bits_elda, - i915_reg_t reg_edid) +/* ELD buffer size in dwords */ +static int g4x_eld_buffer_size(struct drm_i915_private *i915) { - struct drm_i915_private *dev_priv = to_i915(connector->dev); - const u8 *eld = connector->eld; u32 tmp; - int i; - - tmp = intel_de_read(dev_priv, reg_eldv); - tmp &= bits_eldv; - - if (!tmp) - return false; - tmp = intel_de_read(dev_priv, reg_elda); - tmp &= ~bits_elda; - intel_de_write(dev_priv, reg_elda, tmp); + tmp = intel_de_read(i915, G4X_AUD_CNTL_ST); - for (i = 0; i < drm_eld_size(eld) / 4; i++) - if (intel_de_read(dev_priv, reg_edid) != *((const u32 *)eld + i)) - return false; - - return true; + return REG_FIELD_GET(G4X_ELD_BUFFER_SIZE_MASK, tmp); } static void g4x_audio_codec_disable(struct intel_encoder *encoder, const struct intel_crtc_state *old_crtc_state, const struct drm_connector_state *old_conn_state) { - struct drm_i915_private *dev_priv = to_i915(encoder->base.dev); - u32 eldv, tmp; - - tmp = intel_de_read(dev_priv, G4X_AUD_VID_DID); - if (tmp == INTEL_AUDIO_DEVBLC || tmp == INTEL_AUDIO_DEVCL) - eldv = G4X_ELDV_DEVCL_DEVBLC; - else - eldv = G4X_ELDV_DEVCTG; + struct drm_i915_private *i915 = to_i915(encoder->base.dev); + struct intel_crtc *crtc = to_intel_crtc(old_crtc_state->uapi.crtc); /* Invalidate ELD */ - tmp = intel_de_read(dev_priv, G4X_AUD_CNTL_ST); - tmp &= ~eldv; - intel_de_write(dev_priv, G4X_AUD_CNTL_ST, tmp); + intel_de_rmw(i915, G4X_AUD_CNTL_ST, + G4X_ELD_VALID, 0); + + intel_crtc_wait_for_next_vblank(crtc); + intel_crtc_wait_for_next_vblank(crtc); } static void g4x_audio_codec_enable(struct intel_encoder *encoder, const struct intel_crtc_state *crtc_state, const struct drm_connector_state *conn_state) { - struct drm_i915_private *dev_priv = to_i915(encoder->base.dev); + struct drm_i915_private *i915 = to_i915(encoder->base.dev); + struct intel_crtc *crtc = to_intel_crtc(crtc_state->uapi.crtc); struct drm_connector *connector = conn_state->connector; - const u8 *eld = connector->eld; - u32 eldv; - u32 tmp; - int len, i; + const u32 *eld = (const u32 *)connector->eld; + int eld_buffer_size, len, i; - tmp = intel_de_read(dev_priv, G4X_AUD_VID_DID); - if (tmp == INTEL_AUDIO_DEVBLC || tmp == INTEL_AUDIO_DEVCL) - eldv = G4X_ELDV_DEVCL_DEVBLC; - else - eldv = G4X_ELDV_DEVCTG; + intel_crtc_wait_for_next_vblank(crtc); - if (intel_eld_uptodate(connector, - G4X_AUD_CNTL_ST, eldv, - G4X_AUD_CNTL_ST, G4X_ELD_ADDR_MASK, - G4X_HDMIW_HDMIEDID)) - return; + intel_de_rmw(i915, G4X_AUD_CNTL_ST, + G4X_ELD_VALID | G4X_ELD_ADDRESS_MASK, 0); - tmp = intel_de_read(dev_priv, G4X_AUD_CNTL_ST); - tmp &= ~(eldv | G4X_ELD_ADDR_MASK); - len = (tmp >> 9) & 0x1f; /* ELD buffer size */ - intel_de_write(dev_priv, G4X_AUD_CNTL_ST, tmp); + eld_buffer_size = g4x_eld_buffer_size(i915); + len = min(drm_eld_size(connector->eld) / 4, eld_buffer_size); - len = min(drm_eld_size(eld) / 4, len); for (i = 0; i < len; i++) - intel_de_write(dev_priv, G4X_HDMIW_HDMIEDID, - *((const u32 *)eld + i)); + intel_de_write(i915, G4X_HDMIW_HDMIEDID, eld[i]); + for (; i < eld_buffer_size; i++) + intel_de_write(i915, G4X_HDMIW_HDMIEDID, 0); - tmp = intel_de_read(dev_priv, G4X_AUD_CNTL_ST); - tmp |= eldv; - intel_de_write(dev_priv, G4X_AUD_CNTL_ST, tmp); + drm_WARN_ON(&i915->drm, + (intel_de_read(i915, G4X_AUD_CNTL_ST) & G4X_ELD_ADDRESS_MASK) != 0); + + intel_de_rmw(i915, G4X_AUD_CNTL_ST, + 0, G4X_ELD_VALID); } static void hsw_dp_audio_config_update(struct intel_encoder *encoder, const struct intel_crtc_state *crtc_state) { - struct drm_i915_private *dev_priv = to_i915(encoder->base.dev); - struct i915_audio_component *acomp = dev_priv->display.audio.component; + struct drm_i915_private *i915 = to_i915(encoder->base.dev); + struct i915_audio_component *acomp = i915->display.audio.component; enum transcoder cpu_transcoder = crtc_state->cpu_transcoder; enum port port = encoder->port; const struct dp_aud_n_m *nm; @@ -403,12 +374,12 @@ hsw_dp_audio_config_update(struct intel_encoder *encoder, rate = acomp ? acomp->aud_sample_rate[port] : 0; nm = audio_config_dp_get_n_m(crtc_state, rate); if (nm) - drm_dbg_kms(&dev_priv->drm, "using Maud %u, Naud %u\n", nm->m, + drm_dbg_kms(&i915->drm, "using Maud %u, Naud %u\n", nm->m, nm->n); else - drm_dbg_kms(&dev_priv->drm, "using automatic Maud, Naud\n"); + drm_dbg_kms(&i915->drm, "using automatic Maud, Naud\n"); - tmp = intel_de_read(dev_priv, HSW_AUD_CFG(cpu_transcoder)); + tmp = intel_de_read(i915, HSW_AUD_CFG(cpu_transcoder)); tmp &= ~AUD_CONFIG_N_VALUE_INDEX; tmp &= ~AUD_CONFIG_PIXEL_CLOCK_HDMI_MASK; tmp &= ~AUD_CONFIG_N_PROG_ENABLE; @@ -420,9 +391,9 @@ hsw_dp_audio_config_update(struct intel_encoder *encoder, tmp |= AUD_CONFIG_N_PROG_ENABLE; } - intel_de_write(dev_priv, HSW_AUD_CFG(cpu_transcoder), tmp); + intel_de_write(i915, HSW_AUD_CFG(cpu_transcoder), tmp); - tmp = intel_de_read(dev_priv, HSW_AUD_M_CTS_ENABLE(cpu_transcoder)); + tmp = intel_de_read(i915, HSW_AUD_M_CTS_ENABLE(cpu_transcoder)); tmp &= ~AUD_CONFIG_M_MASK; tmp &= ~AUD_M_CTS_M_VALUE_INDEX; tmp &= ~AUD_M_CTS_M_PROG_ENABLE; @@ -433,15 +404,15 @@ hsw_dp_audio_config_update(struct intel_encoder *encoder, tmp |= AUD_M_CTS_M_PROG_ENABLE; } - intel_de_write(dev_priv, HSW_AUD_M_CTS_ENABLE(cpu_transcoder), tmp); + intel_de_write(i915, HSW_AUD_M_CTS_ENABLE(cpu_transcoder), tmp); } static void hsw_hdmi_audio_config_update(struct intel_encoder *encoder, const struct intel_crtc_state *crtc_state) { - struct drm_i915_private *dev_priv = to_i915(encoder->base.dev); - struct i915_audio_component *acomp = dev_priv->display.audio.component; + struct drm_i915_private *i915 = to_i915(encoder->base.dev); + struct i915_audio_component *acomp = i915->display.audio.component; enum transcoder cpu_transcoder = crtc_state->cpu_transcoder; enum port port = encoder->port; int n, rate; @@ -449,7 +420,7 @@ hsw_hdmi_audio_config_update(struct intel_encoder *encoder, rate = acomp ? acomp->aud_sample_rate[port] : 0; - tmp = intel_de_read(dev_priv, HSW_AUD_CFG(cpu_transcoder)); + tmp = intel_de_read(i915, HSW_AUD_CFG(cpu_transcoder)); tmp &= ~AUD_CONFIG_N_VALUE_INDEX; tmp &= ~AUD_CONFIG_PIXEL_CLOCK_HDMI_MASK; tmp &= ~AUD_CONFIG_N_PROG_ENABLE; @@ -457,25 +428,25 @@ hsw_hdmi_audio_config_update(struct intel_encoder *encoder, n = audio_config_hdmi_get_n(crtc_state, rate); if (n != 0) { - drm_dbg_kms(&dev_priv->drm, "using N %d\n", n); + drm_dbg_kms(&i915->drm, "using N %d\n", n); tmp &= ~AUD_CONFIG_N_MASK; tmp |= AUD_CONFIG_N(n); tmp |= AUD_CONFIG_N_PROG_ENABLE; } else { - drm_dbg_kms(&dev_priv->drm, "using automatic N\n"); + drm_dbg_kms(&i915->drm, "using automatic N\n"); } - intel_de_write(dev_priv, HSW_AUD_CFG(cpu_transcoder), tmp); + intel_de_write(i915, HSW_AUD_CFG(cpu_transcoder), tmp); /* * Let's disable "Enable CTS or M Prog bit" * and let HW calculate the value */ - tmp = intel_de_read(dev_priv, HSW_AUD_M_CTS_ENABLE(cpu_transcoder)); + tmp = intel_de_read(i915, HSW_AUD_M_CTS_ENABLE(cpu_transcoder)); tmp &= ~AUD_M_CTS_M_PROG_ENABLE; tmp &= ~AUD_M_CTS_M_VALUE_INDEX; - intel_de_write(dev_priv, HSW_AUD_M_CTS_ENABLE(cpu_transcoder), tmp); + intel_de_write(i915, HSW_AUD_M_CTS_ENABLE(cpu_transcoder), tmp); } static void @@ -488,33 +459,48 @@ hsw_audio_config_update(struct intel_encoder *encoder, hsw_hdmi_audio_config_update(encoder, crtc_state); } +/* ELD buffer size in dwords */ +static int hsw_eld_buffer_size(struct drm_i915_private *i915, + enum transcoder cpu_transcoder) +{ + u32 tmp; + + tmp = intel_de_read(i915, HSW_AUD_DIP_ELD_CTRL(cpu_transcoder)); + + return REG_FIELD_GET(IBX_ELD_BUFFER_SIZE_MASK, tmp); +} + static void hsw_audio_codec_disable(struct intel_encoder *encoder, const struct intel_crtc_state *old_crtc_state, const struct drm_connector_state *old_conn_state) { - struct drm_i915_private *dev_priv = to_i915(encoder->base.dev); + struct drm_i915_private *i915 = to_i915(encoder->base.dev); + struct intel_crtc *crtc = to_intel_crtc(old_crtc_state->uapi.crtc); enum transcoder cpu_transcoder = old_crtc_state->cpu_transcoder; - u32 tmp; - mutex_lock(&dev_priv->display.audio.mutex); + mutex_lock(&i915->display.audio.mutex); /* Disable timestamps */ - tmp = intel_de_read(dev_priv, HSW_AUD_CFG(cpu_transcoder)); - tmp &= ~AUD_CONFIG_N_VALUE_INDEX; - tmp |= AUD_CONFIG_N_PROG_ENABLE; - tmp &= ~AUD_CONFIG_UPPER_N_MASK; - tmp &= ~AUD_CONFIG_LOWER_N_MASK; - if (intel_crtc_has_dp_encoder(old_crtc_state)) - tmp |= AUD_CONFIG_N_VALUE_INDEX; - intel_de_write(dev_priv, HSW_AUD_CFG(cpu_transcoder), tmp); + intel_de_rmw(i915, HSW_AUD_CFG(cpu_transcoder), + AUD_CONFIG_N_VALUE_INDEX | + AUD_CONFIG_UPPER_N_MASK | + AUD_CONFIG_LOWER_N_MASK, + AUD_CONFIG_N_PROG_ENABLE | + (intel_crtc_has_dp_encoder(old_crtc_state) ? + AUD_CONFIG_N_VALUE_INDEX : 0)); /* Invalidate ELD */ - tmp = intel_de_read(dev_priv, HSW_AUD_PIN_ELD_CP_VLD); - tmp &= ~AUDIO_ELD_VALID(cpu_transcoder); - tmp &= ~AUDIO_OUTPUT_ENABLE(cpu_transcoder); - intel_de_write(dev_priv, HSW_AUD_PIN_ELD_CP_VLD, tmp); + intel_de_rmw(i915, HSW_AUD_PIN_ELD_CP_VLD, + AUDIO_ELD_VALID(cpu_transcoder), 0); + + intel_crtc_wait_for_next_vblank(crtc); + intel_crtc_wait_for_next_vblank(crtc); - mutex_unlock(&dev_priv->display.audio.mutex); + /* Disable audio presence detect */ + intel_de_rmw(i915, HSW_AUD_PIN_ELD_CP_VLD, + AUDIO_OUTPUT_ENABLE(cpu_transcoder), 0); + + mutex_unlock(&i915->display.audio.mutex); } static unsigned int calc_hblank_early_prog(struct intel_encoder *encoder, @@ -626,178 +612,190 @@ static void enable_audio_dsc_wa(struct intel_encoder *encoder, intel_de_write(i915, AUD_CONFIG_BE, val); } -#undef ROUNDING_FACTOR - static void hsw_audio_codec_enable(struct intel_encoder *encoder, const struct intel_crtc_state *crtc_state, const struct drm_connector_state *conn_state) { - struct drm_i915_private *dev_priv = to_i915(encoder->base.dev); + struct drm_i915_private *i915 = to_i915(encoder->base.dev); + struct intel_crtc *crtc = to_intel_crtc(crtc_state->uapi.crtc); struct drm_connector *connector = conn_state->connector; enum transcoder cpu_transcoder = crtc_state->cpu_transcoder; - const u8 *eld = connector->eld; - u32 tmp; - int len, i; + const u32 *eld = (const u32 *)connector->eld; + int eld_buffer_size, len, i; - mutex_lock(&dev_priv->display.audio.mutex); + mutex_lock(&i915->display.audio.mutex); /* Enable Audio WA for 4k DSC usecases */ if (intel_crtc_has_type(crtc_state, INTEL_OUTPUT_DP)) enable_audio_dsc_wa(encoder, crtc_state); - /* Enable audio presence detect, invalidate ELD */ - tmp = intel_de_read(dev_priv, HSW_AUD_PIN_ELD_CP_VLD); - tmp |= AUDIO_OUTPUT_ENABLE(cpu_transcoder); - tmp &= ~AUDIO_ELD_VALID(cpu_transcoder); - intel_de_write(dev_priv, HSW_AUD_PIN_ELD_CP_VLD, tmp); + /* Enable audio presence detect */ + intel_de_rmw(i915, HSW_AUD_PIN_ELD_CP_VLD, + 0, AUDIO_OUTPUT_ENABLE(cpu_transcoder)); - /* - * FIXME: We're supposed to wait for vblank here, but we have vblanks - * disabled during the mode set. The proper fix would be to push the - * rest of the setup into a vblank work item, queued here, but the - * infrastructure is not there yet. - */ + intel_crtc_wait_for_next_vblank(crtc); + + /* Invalidate ELD */ + intel_de_rmw(i915, HSW_AUD_PIN_ELD_CP_VLD, + AUDIO_ELD_VALID(cpu_transcoder), 0); + + /* Reset ELD address */ + intel_de_rmw(i915, HSW_AUD_DIP_ELD_CTRL(cpu_transcoder), + IBX_ELD_ADDRESS_MASK, 0); - /* Reset ELD write address */ - tmp = intel_de_read(dev_priv, HSW_AUD_DIP_ELD_CTRL(cpu_transcoder)); - tmp &= ~IBX_ELD_ADDRESS_MASK; - intel_de_write(dev_priv, HSW_AUD_DIP_ELD_CTRL(cpu_transcoder), tmp); + eld_buffer_size = hsw_eld_buffer_size(i915, cpu_transcoder); + len = min(drm_eld_size(connector->eld) / 4, eld_buffer_size); - /* Up to 84 bytes of hw ELD buffer */ - len = min(drm_eld_size(eld), 84); - for (i = 0; i < len / 4; i++) - intel_de_write(dev_priv, HSW_AUD_EDID_DATA(cpu_transcoder), - *((const u32 *)eld + i)); + for (i = 0; i < len; i++) + intel_de_write(i915, HSW_AUD_EDID_DATA(cpu_transcoder), eld[i]); + for (; i < eld_buffer_size; i++) + intel_de_write(i915, HSW_AUD_EDID_DATA(cpu_transcoder), 0); + + drm_WARN_ON(&i915->drm, + (intel_de_read(i915, HSW_AUD_DIP_ELD_CTRL(cpu_transcoder)) & + IBX_ELD_ADDRESS_MASK) != 0); /* ELD valid */ - tmp = intel_de_read(dev_priv, HSW_AUD_PIN_ELD_CP_VLD); - tmp |= AUDIO_ELD_VALID(cpu_transcoder); - intel_de_write(dev_priv, HSW_AUD_PIN_ELD_CP_VLD, tmp); + intel_de_rmw(i915, HSW_AUD_PIN_ELD_CP_VLD, + 0, AUDIO_ELD_VALID(cpu_transcoder)); /* Enable timestamps */ hsw_audio_config_update(encoder, crtc_state); - mutex_unlock(&dev_priv->display.audio.mutex); + mutex_unlock(&i915->display.audio.mutex); +} + +struct ilk_audio_regs { + i915_reg_t hdmiw_hdmiedid, aud_config, aud_cntl_st, aud_cntrl_st2; +}; + +static void ilk_audio_regs_init(struct drm_i915_private *i915, + enum pipe pipe, + struct ilk_audio_regs *regs) +{ + if (HAS_PCH_IBX(i915)) { + regs->hdmiw_hdmiedid = IBX_HDMIW_HDMIEDID(pipe); + regs->aud_config = IBX_AUD_CFG(pipe); + regs->aud_cntl_st = IBX_AUD_CNTL_ST(pipe); + regs->aud_cntrl_st2 = IBX_AUD_CNTL_ST2; + } else if (IS_VALLEYVIEW(i915) || IS_CHERRYVIEW(i915)) { + regs->hdmiw_hdmiedid = VLV_HDMIW_HDMIEDID(pipe); + regs->aud_config = VLV_AUD_CFG(pipe); + regs->aud_cntl_st = VLV_AUD_CNTL_ST(pipe); + regs->aud_cntrl_st2 = VLV_AUD_CNTL_ST2; + } else { + regs->hdmiw_hdmiedid = CPT_HDMIW_HDMIEDID(pipe); + regs->aud_config = CPT_AUD_CFG(pipe); + regs->aud_cntl_st = CPT_AUD_CNTL_ST(pipe); + regs->aud_cntrl_st2 = CPT_AUD_CNTRL_ST2; + } +} + +/* ELD buffer size in dwords */ +static int ilk_eld_buffer_size(struct drm_i915_private *i915, + enum pipe pipe) +{ + struct ilk_audio_regs regs; + u32 tmp; + + ilk_audio_regs_init(i915, pipe, ®s); + + tmp = intel_de_read(i915, regs.aud_cntl_st); + + return REG_FIELD_GET(IBX_ELD_BUFFER_SIZE_MASK, tmp); } static void ilk_audio_codec_disable(struct intel_encoder *encoder, const struct intel_crtc_state *old_crtc_state, const struct drm_connector_state *old_conn_state) { - struct drm_i915_private *dev_priv = to_i915(encoder->base.dev); + struct drm_i915_private *i915 = to_i915(encoder->base.dev); struct intel_crtc *crtc = to_intel_crtc(old_crtc_state->uapi.crtc); - enum pipe pipe = crtc->pipe; enum port port = encoder->port; - u32 tmp, eldv; - i915_reg_t aud_config, aud_cntrl_st2; + enum pipe pipe = crtc->pipe; + struct ilk_audio_regs regs; - if (drm_WARN_ON(&dev_priv->drm, port == PORT_A)) + if (drm_WARN_ON(&i915->drm, port == PORT_A)) return; - if (HAS_PCH_IBX(dev_priv)) { - aud_config = IBX_AUD_CFG(pipe); - aud_cntrl_st2 = IBX_AUD_CNTL_ST2; - } else if (IS_VALLEYVIEW(dev_priv) || IS_CHERRYVIEW(dev_priv)) { - aud_config = VLV_AUD_CFG(pipe); - aud_cntrl_st2 = VLV_AUD_CNTL_ST2; - } else { - aud_config = CPT_AUD_CFG(pipe); - aud_cntrl_st2 = CPT_AUD_CNTRL_ST2; - } + ilk_audio_regs_init(i915, pipe, ®s); - /* Disable timestamps */ - tmp = intel_de_read(dev_priv, aud_config); - tmp &= ~AUD_CONFIG_N_VALUE_INDEX; - tmp |= AUD_CONFIG_N_PROG_ENABLE; - tmp &= ~AUD_CONFIG_UPPER_N_MASK; - tmp &= ~AUD_CONFIG_LOWER_N_MASK; - if (intel_crtc_has_dp_encoder(old_crtc_state)) - tmp |= AUD_CONFIG_N_VALUE_INDEX; - intel_de_write(dev_priv, aud_config, tmp); + mutex_lock(&i915->display.audio.mutex); - eldv = IBX_ELD_VALID(port); + /* Disable timestamps */ + intel_de_rmw(i915, regs.aud_config, + AUD_CONFIG_N_VALUE_INDEX | + AUD_CONFIG_UPPER_N_MASK | + AUD_CONFIG_LOWER_N_MASK, + AUD_CONFIG_N_PROG_ENABLE | + (intel_crtc_has_dp_encoder(old_crtc_state) ? + AUD_CONFIG_N_VALUE_INDEX : 0)); /* Invalidate ELD */ - tmp = intel_de_read(dev_priv, aud_cntrl_st2); - tmp &= ~eldv; - intel_de_write(dev_priv, aud_cntrl_st2, tmp); + intel_de_rmw(i915, regs.aud_cntrl_st2, + IBX_ELD_VALID(port), 0); + + mutex_unlock(&i915->display.audio.mutex); + + intel_crtc_wait_for_next_vblank(crtc); + intel_crtc_wait_for_next_vblank(crtc); } static void ilk_audio_codec_enable(struct intel_encoder *encoder, const struct intel_crtc_state *crtc_state, const struct drm_connector_state *conn_state) { - struct drm_i915_private *dev_priv = to_i915(encoder->base.dev); + struct drm_i915_private *i915 = to_i915(encoder->base.dev); struct intel_crtc *crtc = to_intel_crtc(crtc_state->uapi.crtc); struct drm_connector *connector = conn_state->connector; - enum pipe pipe = crtc->pipe; + const u32 *eld = (const u32 *)connector->eld; enum port port = encoder->port; - const u8 *eld = connector->eld; - u32 tmp, eldv; - int len, i; - i915_reg_t hdmiw_hdmiedid, aud_config, aud_cntl_st, aud_cntrl_st2; + enum pipe pipe = crtc->pipe; + int eld_buffer_size, len, i; + struct ilk_audio_regs regs; - if (drm_WARN_ON(&dev_priv->drm, port == PORT_A)) + if (drm_WARN_ON(&i915->drm, port == PORT_A)) return; - /* - * FIXME: We're supposed to wait for vblank here, but we have vblanks - * disabled during the mode set. The proper fix would be to push the - * rest of the setup into a vblank work item, queued here, but the - * infrastructure is not there yet. - */ + intel_crtc_wait_for_next_vblank(crtc); - if (HAS_PCH_IBX(dev_priv)) { - hdmiw_hdmiedid = IBX_HDMIW_HDMIEDID(pipe); - aud_config = IBX_AUD_CFG(pipe); - aud_cntl_st = IBX_AUD_CNTL_ST(pipe); - aud_cntrl_st2 = IBX_AUD_CNTL_ST2; - } else if (IS_VALLEYVIEW(dev_priv) || - IS_CHERRYVIEW(dev_priv)) { - hdmiw_hdmiedid = VLV_HDMIW_HDMIEDID(pipe); - aud_config = VLV_AUD_CFG(pipe); - aud_cntl_st = VLV_AUD_CNTL_ST(pipe); - aud_cntrl_st2 = VLV_AUD_CNTL_ST2; - } else { - hdmiw_hdmiedid = CPT_HDMIW_HDMIEDID(pipe); - aud_config = CPT_AUD_CFG(pipe); - aud_cntl_st = CPT_AUD_CNTL_ST(pipe); - aud_cntrl_st2 = CPT_AUD_CNTRL_ST2; - } + ilk_audio_regs_init(i915, pipe, ®s); - eldv = IBX_ELD_VALID(port); + mutex_lock(&i915->display.audio.mutex); /* Invalidate ELD */ - tmp = intel_de_read(dev_priv, aud_cntrl_st2); - tmp &= ~eldv; - intel_de_write(dev_priv, aud_cntrl_st2, tmp); + intel_de_rmw(i915, regs.aud_cntrl_st2, + IBX_ELD_VALID(port), 0); + + /* Reset ELD address */ + intel_de_rmw(i915, regs.aud_cntl_st, + IBX_ELD_ADDRESS_MASK, 0); - /* Reset ELD write address */ - tmp = intel_de_read(dev_priv, aud_cntl_st); - tmp &= ~IBX_ELD_ADDRESS_MASK; - intel_de_write(dev_priv, aud_cntl_st, tmp); + eld_buffer_size = ilk_eld_buffer_size(i915, pipe); + len = min(drm_eld_size(connector->eld) / 4, eld_buffer_size); - /* Up to 84 bytes of hw ELD buffer */ - len = min(drm_eld_size(eld), 84); - for (i = 0; i < len / 4; i++) - intel_de_write(dev_priv, hdmiw_hdmiedid, - *((const u32 *)eld + i)); + for (i = 0; i < len; i++) + intel_de_write(i915, regs.hdmiw_hdmiedid, eld[i]); + for (; i < eld_buffer_size; i++) + intel_de_write(i915, regs.hdmiw_hdmiedid, 0); + + drm_WARN_ON(&i915->drm, + (intel_de_read(i915, regs.aud_cntl_st) & IBX_ELD_ADDRESS_MASK) != 0); /* ELD valid */ - tmp = intel_de_read(dev_priv, aud_cntrl_st2); - tmp |= eldv; - intel_de_write(dev_priv, aud_cntrl_st2, tmp); + intel_de_rmw(i915, regs.aud_cntrl_st2, + 0, IBX_ELD_VALID(port)); /* Enable timestamps */ - tmp = intel_de_read(dev_priv, aud_config); - tmp &= ~AUD_CONFIG_N_VALUE_INDEX; - tmp &= ~AUD_CONFIG_N_PROG_ENABLE; - tmp &= ~AUD_CONFIG_PIXEL_CLOCK_HDMI_MASK; - if (intel_crtc_has_dp_encoder(crtc_state)) - tmp |= AUD_CONFIG_N_VALUE_INDEX; - else - tmp |= audio_config_hdmi_pixel_clock(crtc_state); - intel_de_write(dev_priv, aud_config, tmp); + intel_de_rmw(i915, regs.aud_config, + AUD_CONFIG_N_VALUE_INDEX | + AUD_CONFIG_N_PROG_ENABLE | + AUD_CONFIG_PIXEL_CLOCK_HDMI_MASK, + (intel_crtc_has_dp_encoder(crtc_state) ? + AUD_CONFIG_N_VALUE_INDEX : + audio_config_hdmi_pixel_clock(crtc_state))); + + mutex_unlock(&i915->display.audio.mutex); } /** @@ -813,8 +811,8 @@ void intel_audio_codec_enable(struct intel_encoder *encoder, const struct intel_crtc_state *crtc_state, const struct drm_connector_state *conn_state) { - struct drm_i915_private *dev_priv = to_i915(encoder->base.dev); - struct i915_audio_component *acomp = dev_priv->display.audio.component; + struct drm_i915_private *i915 = to_i915(encoder->base.dev); + struct i915_audio_component *acomp = i915->display.audio.component; struct intel_crtc *crtc = to_intel_crtc(crtc_state->uapi.crtc); struct drm_connector *connector = conn_state->connector; const struct drm_display_mode *adjusted_mode = @@ -825,30 +823,30 @@ void intel_audio_codec_enable(struct intel_encoder *encoder, if (!crtc_state->has_audio) return; - drm_dbg_kms(&dev_priv->drm, "[CONNECTOR:%d:%s][ENCODER:%d:%s] Enable audio codec on pipe %c, %u bytes ELD\n", + drm_dbg_kms(&i915->drm, "[CONNECTOR:%d:%s][ENCODER:%d:%s] Enable audio codec on pipe %c, %u bytes ELD\n", connector->base.id, connector->name, encoder->base.base.id, encoder->base.name, pipe_name(pipe), drm_eld_size(connector->eld)); /* FIXME precompute the ELD in .compute_config() */ if (!connector->eld[0]) - drm_dbg_kms(&dev_priv->drm, + drm_dbg_kms(&i915->drm, "Bogus ELD on [CONNECTOR:%d:%s]\n", connector->base.id, connector->name); connector->eld[6] = drm_av_sync_delay(connector, adjusted_mode) / 2; - if (dev_priv->display.funcs.audio) - dev_priv->display.funcs.audio->audio_codec_enable(encoder, - crtc_state, - conn_state); + if (i915->display.funcs.audio) + i915->display.funcs.audio->audio_codec_enable(encoder, + crtc_state, + conn_state); - mutex_lock(&dev_priv->display.audio.mutex); + mutex_lock(&i915->display.audio.mutex); encoder->audio_connector = connector; /* referred in audio callbacks */ - dev_priv->display.audio.encoder_map[pipe] = encoder; - mutex_unlock(&dev_priv->display.audio.mutex); + i915->display.audio.encoder_map[pipe] = encoder; + mutex_unlock(&i915->display.audio.mutex); if (acomp && acomp->base.audio_ops && acomp->base.audio_ops->pin_eld_notify) { @@ -856,10 +854,10 @@ void intel_audio_codec_enable(struct intel_encoder *encoder, if (!intel_crtc_has_type(crtc_state, INTEL_OUTPUT_DP_MST)) pipe = -1; acomp->base.audio_ops->pin_eld_notify(acomp->base.audio_ops->audio_ptr, - (int) port, (int) pipe); + (int)port, (int)pipe); } - intel_lpe_audio_notify(dev_priv, pipe, port, connector->eld, + intel_lpe_audio_notify(i915, pipe, port, connector->eld, crtc_state->port_clock, intel_crtc_has_dp_encoder(crtc_state)); } @@ -877,8 +875,8 @@ void intel_audio_codec_disable(struct intel_encoder *encoder, const struct intel_crtc_state *old_crtc_state, const struct drm_connector_state *old_conn_state) { - struct drm_i915_private *dev_priv = to_i915(encoder->base.dev); - struct i915_audio_component *acomp = dev_priv->display.audio.component; + struct drm_i915_private *i915 = to_i915(encoder->base.dev); + struct i915_audio_component *acomp = i915->display.audio.component; struct intel_crtc *crtc = to_intel_crtc(old_crtc_state->uapi.crtc); struct drm_connector *connector = old_conn_state->connector; enum port port = encoder->port; @@ -887,19 +885,19 @@ void intel_audio_codec_disable(struct intel_encoder *encoder, if (!old_crtc_state->has_audio) return; - drm_dbg_kms(&dev_priv->drm, "[CONNECTOR:%d:%s][ENCODER:%d:%s] Disable audio codec on pipe %c\n", + drm_dbg_kms(&i915->drm, "[CONNECTOR:%d:%s][ENCODER:%d:%s] Disable audio codec on pipe %c\n", connector->base.id, connector->name, encoder->base.base.id, encoder->base.name, pipe_name(pipe)); - if (dev_priv->display.funcs.audio) - dev_priv->display.funcs.audio->audio_codec_disable(encoder, - old_crtc_state, - old_conn_state); + if (i915->display.funcs.audio) + i915->display.funcs.audio->audio_codec_disable(encoder, + old_crtc_state, + old_conn_state); - mutex_lock(&dev_priv->display.audio.mutex); + mutex_lock(&i915->display.audio.mutex); encoder->audio_connector = NULL; - dev_priv->display.audio.encoder_map[pipe] = NULL; - mutex_unlock(&dev_priv->display.audio.mutex); + i915->display.audio.encoder_map[pipe] = NULL; + mutex_unlock(&i915->display.audio.mutex); if (acomp && acomp->base.audio_ops && acomp->base.audio_ops->pin_eld_notify) { @@ -907,10 +905,10 @@ void intel_audio_codec_disable(struct intel_encoder *encoder, if (!intel_crtc_has_type(old_crtc_state, INTEL_OUTPUT_DP_MST)) pipe = -1; acomp->base.audio_ops->pin_eld_notify(acomp->base.audio_ops->audio_ptr, - (int) port, (int) pipe); + (int)port, (int)pipe); } - intel_lpe_audio_notify(dev_priv, pipe, port, NULL, 0, false); + intel_lpe_audio_notify(i915, pipe, port, NULL, 0, false); } static const struct intel_audio_funcs g4x_audio_funcs = { @@ -930,19 +928,18 @@ static const struct intel_audio_funcs hsw_audio_funcs = { /** * intel_audio_hooks_init - Set up chip specific audio hooks - * @dev_priv: device private + * @i915: device private */ -void intel_audio_hooks_init(struct drm_i915_private *dev_priv) +void intel_audio_hooks_init(struct drm_i915_private *i915) { - if (IS_G4X(dev_priv)) { - dev_priv->display.funcs.audio = &g4x_audio_funcs; - } else if (IS_VALLEYVIEW(dev_priv) || IS_CHERRYVIEW(dev_priv)) { - dev_priv->display.funcs.audio = &ilk_audio_funcs; - } else if (IS_HASWELL(dev_priv) || DISPLAY_VER(dev_priv) >= 8) { - dev_priv->display.funcs.audio = &hsw_audio_funcs; - } else if (HAS_PCH_SPLIT(dev_priv)) { - dev_priv->display.funcs.audio = &ilk_audio_funcs; - } + if (IS_G4X(i915)) + i915->display.funcs.audio = &g4x_audio_funcs; + else if (IS_VALLEYVIEW(i915) || IS_CHERRYVIEW(i915)) + i915->display.funcs.audio = &ilk_audio_funcs; + else if (IS_HASWELL(i915) || DISPLAY_VER(i915) >= 8) + i915->display.funcs.audio = &hsw_audio_funcs; + else if (HAS_PCH_SPLIT(i915)) + i915->display.funcs.audio = &ilk_audio_funcs; } struct aud_ts_cdclk_m_n { @@ -1000,7 +997,7 @@ static int glk_force_audio_cdclk_commit(struct intel_atomic_state *state, return drm_atomic_commit(&state->base); } -static void glk_force_audio_cdclk(struct drm_i915_private *dev_priv, +static void glk_force_audio_cdclk(struct drm_i915_private *i915, bool enable) { struct drm_modeset_acquire_ctx ctx; @@ -1008,13 +1005,13 @@ static void glk_force_audio_cdclk(struct drm_i915_private *dev_priv, struct intel_crtc *crtc; int ret; - crtc = intel_first_crtc(dev_priv); + crtc = intel_first_crtc(i915); if (!crtc) return; drm_modeset_acquire_init(&ctx, 0); - state = drm_atomic_state_alloc(&dev_priv->drm); - if (drm_WARN_ON(&dev_priv->drm, !state)) + state = drm_atomic_state_alloc(&i915->drm); + if (drm_WARN_ON(&i915->drm, !state)) return; state->acquire_ctx = &ctx; @@ -1028,7 +1025,7 @@ retry: goto retry; } - drm_WARN_ON(&dev_priv->drm, ret); + drm_WARN_ON(&i915->drm, ret); drm_atomic_state_put(state); @@ -1038,30 +1035,30 @@ retry: static unsigned long i915_audio_component_get_power(struct device *kdev) { - struct drm_i915_private *dev_priv = kdev_to_i915(kdev); + struct drm_i915_private *i915 = kdev_to_i915(kdev); intel_wakeref_t ret; /* Catch potential impedance mismatches before they occur! */ BUILD_BUG_ON(sizeof(intel_wakeref_t) > sizeof(unsigned long)); - ret = intel_display_power_get(dev_priv, POWER_DOMAIN_AUDIO_PLAYBACK); + ret = intel_display_power_get(i915, POWER_DOMAIN_AUDIO_PLAYBACK); - if (dev_priv->display.audio.power_refcount++ == 0) { - if (DISPLAY_VER(dev_priv) >= 9) { - intel_de_write(dev_priv, AUD_FREQ_CNTRL, - dev_priv->display.audio.freq_cntrl); - drm_dbg_kms(&dev_priv->drm, + if (i915->display.audio.power_refcount++ == 0) { + if (DISPLAY_VER(i915) >= 9) { + intel_de_write(i915, AUD_FREQ_CNTRL, + i915->display.audio.freq_cntrl); + drm_dbg_kms(&i915->drm, "restored AUD_FREQ_CNTRL to 0x%x\n", - dev_priv->display.audio.freq_cntrl); + i915->display.audio.freq_cntrl); } /* Force CDCLK to 2*BCLK as long as we need audio powered. */ - if (IS_GEMINILAKE(dev_priv)) - glk_force_audio_cdclk(dev_priv, true); + if (IS_GEMINILAKE(i915)) + glk_force_audio_cdclk(i915, true); - if (DISPLAY_VER(dev_priv) >= 10) - intel_de_write(dev_priv, AUD_PIN_BUF_CTL, - (intel_de_read(dev_priv, AUD_PIN_BUF_CTL) | AUD_PIN_BUF_ENABLE)); + if (DISPLAY_VER(i915) >= 10) + intel_de_rmw(i915, AUD_PIN_BUF_CTL, + 0, AUD_PIN_BUF_ENABLE); } return ret; @@ -1070,24 +1067,23 @@ static unsigned long i915_audio_component_get_power(struct device *kdev) static void i915_audio_component_put_power(struct device *kdev, unsigned long cookie) { - struct drm_i915_private *dev_priv = kdev_to_i915(kdev); + struct drm_i915_private *i915 = kdev_to_i915(kdev); /* Stop forcing CDCLK to 2*BCLK if no need for audio to be powered. */ - if (--dev_priv->display.audio.power_refcount == 0) - if (IS_GEMINILAKE(dev_priv)) - glk_force_audio_cdclk(dev_priv, false); + if (--i915->display.audio.power_refcount == 0) + if (IS_GEMINILAKE(i915)) + glk_force_audio_cdclk(i915, false); - intel_display_power_put(dev_priv, POWER_DOMAIN_AUDIO_PLAYBACK, cookie); + intel_display_power_put(i915, POWER_DOMAIN_AUDIO_PLAYBACK, cookie); } static void i915_audio_component_codec_wake_override(struct device *kdev, bool enable) { - struct drm_i915_private *dev_priv = kdev_to_i915(kdev); + struct drm_i915_private *i915 = kdev_to_i915(kdev); unsigned long cookie; - u32 tmp; - if (DISPLAY_VER(dev_priv) < 9) + if (DISPLAY_VER(i915) < 9) return; cookie = i915_audio_component_get_power(kdev); @@ -1096,15 +1092,13 @@ static void i915_audio_component_codec_wake_override(struct device *kdev, * Enable/disable generating the codec wake signal, overriding the * internal logic to generate the codec wake to controller. */ - tmp = intel_de_read(dev_priv, HSW_AUD_CHICKENBIT); - tmp &= ~SKL_AUD_CODEC_WAKE_SIGNAL; - intel_de_write(dev_priv, HSW_AUD_CHICKENBIT, tmp); + intel_de_rmw(i915, HSW_AUD_CHICKENBIT, + SKL_AUD_CODEC_WAKE_SIGNAL, 0); usleep_range(1000, 1500); if (enable) { - tmp = intel_de_read(dev_priv, HSW_AUD_CHICKENBIT); - tmp |= SKL_AUD_CODEC_WAKE_SIGNAL; - intel_de_write(dev_priv, HSW_AUD_CHICKENBIT, tmp); + intel_de_rmw(i915, HSW_AUD_CHICKENBIT, + 0, SKL_AUD_CODEC_WAKE_SIGNAL); usleep_range(1000, 1500); } @@ -1114,12 +1108,12 @@ static void i915_audio_component_codec_wake_override(struct device *kdev, /* Get CDCLK in kHz */ static int i915_audio_component_get_cdclk_freq(struct device *kdev) { - struct drm_i915_private *dev_priv = kdev_to_i915(kdev); + struct drm_i915_private *i915 = kdev_to_i915(kdev); - if (drm_WARN_ON_ONCE(&dev_priv->drm, !HAS_DDI(dev_priv))) + if (drm_WARN_ON_ONCE(&i915->drm, !HAS_DDI(i915))) return -ENODEV; - return dev_priv->display.cdclk.hw.cdclk; + return i915->display.cdclk.hw.cdclk; } /* @@ -1132,24 +1126,24 @@ static int i915_audio_component_get_cdclk_freq(struct device *kdev) * will get the right intel_encoder with port matched * Non-MST & (pipe < 0): get the right intel_encoder with port matched */ -static struct intel_encoder *get_saved_enc(struct drm_i915_private *dev_priv, - int port, int pipe) +static struct intel_encoder *get_saved_enc(struct drm_i915_private *i915, + int port, int pipe) { - struct intel_encoder *encoder; - /* MST */ if (pipe >= 0) { - if (drm_WARN_ON(&dev_priv->drm, - pipe >= ARRAY_SIZE(dev_priv->display.audio.encoder_map))) + struct intel_encoder *encoder; + + if (drm_WARN_ON(&i915->drm, + pipe >= ARRAY_SIZE(i915->display.audio.encoder_map))) return NULL; - encoder = dev_priv->display.audio.encoder_map[pipe]; + encoder = i915->display.audio.encoder_map[pipe]; /* * when bootup, audio driver may not know it is * MST or not. So it will poll all the port & pipe * combinations */ - if (encoder != NULL && encoder->port == port && + if (encoder && encoder->port == port && encoder->type == INTEL_OUTPUT_DP_MST) return encoder; } @@ -1158,15 +1152,13 @@ static struct intel_encoder *get_saved_enc(struct drm_i915_private *dev_priv, if (pipe > 0) return NULL; - for_each_pipe(dev_priv, pipe) { - encoder = dev_priv->display.audio.encoder_map[pipe]; - if (encoder == NULL) - continue; + for_each_pipe(i915, pipe) { + struct intel_encoder *encoder; - if (encoder->type == INTEL_OUTPUT_DP_MST) - continue; + encoder = i915->display.audio.encoder_map[pipe]; - if (port == encoder->port) + if (encoder && encoder->port == port && + encoder->type != INTEL_OUTPUT_DP_MST) return encoder; } @@ -1176,23 +1168,23 @@ static struct intel_encoder *get_saved_enc(struct drm_i915_private *dev_priv, static int i915_audio_component_sync_audio_rate(struct device *kdev, int port, int pipe, int rate) { - struct drm_i915_private *dev_priv = kdev_to_i915(kdev); - struct i915_audio_component *acomp = dev_priv->display.audio.component; + struct drm_i915_private *i915 = kdev_to_i915(kdev); + struct i915_audio_component *acomp = i915->display.audio.component; struct intel_encoder *encoder; struct intel_crtc *crtc; unsigned long cookie; int err = 0; - if (!HAS_DDI(dev_priv)) + if (!HAS_DDI(i915)) return 0; cookie = i915_audio_component_get_power(kdev); - mutex_lock(&dev_priv->display.audio.mutex); + mutex_lock(&i915->display.audio.mutex); /* 1. get the pipe */ - encoder = get_saved_enc(dev_priv, port, pipe); + encoder = get_saved_enc(i915, port, pipe); if (!encoder || !encoder->base.crtc) { - drm_dbg_kms(&dev_priv->drm, "Not valid for port %c\n", + drm_dbg_kms(&i915->drm, "Not valid for port %c\n", port_name(port)); err = -ENODEV; goto unlock; @@ -1206,7 +1198,7 @@ static int i915_audio_component_sync_audio_rate(struct device *kdev, int port, hsw_audio_config_update(encoder, crtc->config); unlock: - mutex_unlock(&dev_priv->display.audio.mutex); + mutex_unlock(&i915->display.audio.mutex); i915_audio_component_put_power(kdev, cookie); return err; } @@ -1215,18 +1207,18 @@ static int i915_audio_component_get_eld(struct device *kdev, int port, int pipe, bool *enabled, unsigned char *buf, int max_bytes) { - struct drm_i915_private *dev_priv = kdev_to_i915(kdev); + struct drm_i915_private *i915 = kdev_to_i915(kdev); struct intel_encoder *intel_encoder; const u8 *eld; int ret = -EINVAL; - mutex_lock(&dev_priv->display.audio.mutex); + mutex_lock(&i915->display.audio.mutex); - intel_encoder = get_saved_enc(dev_priv, port, pipe); + intel_encoder = get_saved_enc(i915, port, pipe); if (!intel_encoder) { - drm_dbg_kms(&dev_priv->drm, "Not valid for port %c\n", + drm_dbg_kms(&i915->drm, "Not valid for port %c\n", port_name(port)); - mutex_unlock(&dev_priv->display.audio.mutex); + mutex_unlock(&i915->display.audio.mutex); return ret; } @@ -1238,7 +1230,7 @@ static int i915_audio_component_get_eld(struct device *kdev, int port, memcpy(buf, eld, min(max_bytes, ret)); } - mutex_unlock(&dev_priv->display.audio.mutex); + mutex_unlock(&i915->display.audio.mutex); return ret; } @@ -1256,25 +1248,25 @@ static int i915_audio_component_bind(struct device *i915_kdev, struct device *hda_kdev, void *data) { struct i915_audio_component *acomp = data; - struct drm_i915_private *dev_priv = kdev_to_i915(i915_kdev); + struct drm_i915_private *i915 = kdev_to_i915(i915_kdev); int i; - if (drm_WARN_ON(&dev_priv->drm, acomp->base.ops || acomp->base.dev)) + if (drm_WARN_ON(&i915->drm, acomp->base.ops || acomp->base.dev)) return -EEXIST; - if (drm_WARN_ON(&dev_priv->drm, + if (drm_WARN_ON(&i915->drm, !device_link_add(hda_kdev, i915_kdev, DL_FLAG_STATELESS))) return -ENOMEM; - drm_modeset_lock_all(&dev_priv->drm); + drm_modeset_lock_all(&i915->drm); acomp->base.ops = &i915_audio_component_ops; acomp->base.dev = i915_kdev; BUILD_BUG_ON(MAX_PORTS != I915_MAX_PORTS); for (i = 0; i < ARRAY_SIZE(acomp->aud_sample_rate); i++) acomp->aud_sample_rate[i] = 0; - dev_priv->display.audio.component = acomp; - drm_modeset_unlock_all(&dev_priv->drm); + i915->display.audio.component = acomp; + drm_modeset_unlock_all(&i915->drm); return 0; } @@ -1283,19 +1275,19 @@ static void i915_audio_component_unbind(struct device *i915_kdev, struct device *hda_kdev, void *data) { struct i915_audio_component *acomp = data; - struct drm_i915_private *dev_priv = kdev_to_i915(i915_kdev); + struct drm_i915_private *i915 = kdev_to_i915(i915_kdev); - drm_modeset_lock_all(&dev_priv->drm); + drm_modeset_lock_all(&i915->drm); acomp->base.ops = NULL; acomp->base.dev = NULL; - dev_priv->display.audio.component = NULL; - drm_modeset_unlock_all(&dev_priv->drm); + i915->display.audio.component = NULL; + drm_modeset_unlock_all(&i915->drm); device_link_remove(hda_kdev, i915_kdev); - if (dev_priv->display.audio.power_refcount) - drm_err(&dev_priv->drm, "audio power refcount %d after unbind\n", - dev_priv->display.audio.power_refcount); + if (i915->display.audio.power_refcount) + drm_err(&i915->drm, "audio power refcount %d after unbind\n", + i915->display.audio.power_refcount); } static const struct component_ops i915_audio_component_bind_ops = { @@ -1314,7 +1306,7 @@ static const struct component_ops i915_audio_component_bind_ops = { /** * i915_audio_component_init - initialize and register the audio component - * @dev_priv: i915 device instance + * @i915: i915 device instance * * This will register with the component framework a child component which * will bind dynamically to the snd_hda_intel driver's corresponding master @@ -1328,83 +1320,83 @@ static const struct component_ops i915_audio_component_bind_ops = { * We ignore any error during registration and continue with reduced * functionality (i.e. without HDMI audio). */ -static void i915_audio_component_init(struct drm_i915_private *dev_priv) +static void i915_audio_component_init(struct drm_i915_private *i915) { u32 aud_freq, aud_freq_init; int ret; - ret = component_add_typed(dev_priv->drm.dev, + ret = component_add_typed(i915->drm.dev, &i915_audio_component_bind_ops, I915_COMPONENT_AUDIO); if (ret < 0) { - drm_err(&dev_priv->drm, + drm_err(&i915->drm, "failed to add audio component (%d)\n", ret); /* continue with reduced functionality */ return; } - if (DISPLAY_VER(dev_priv) >= 9) { - aud_freq_init = intel_de_read(dev_priv, AUD_FREQ_CNTRL); + if (DISPLAY_VER(i915) >= 9) { + aud_freq_init = intel_de_read(i915, AUD_FREQ_CNTRL); - if (DISPLAY_VER(dev_priv) >= 12) + if (DISPLAY_VER(i915) >= 12) aud_freq = AUD_FREQ_GEN12; else aud_freq = aud_freq_init; /* use BIOS provided value for TGL and RKL unless it is a known bad value */ - if ((IS_TIGERLAKE(dev_priv) || IS_ROCKETLAKE(dev_priv)) && + if ((IS_TIGERLAKE(i915) || IS_ROCKETLAKE(i915)) && aud_freq_init != AUD_FREQ_TGL_BROKEN) aud_freq = aud_freq_init; - drm_dbg_kms(&dev_priv->drm, "use AUD_FREQ_CNTRL of 0x%x (init value 0x%x)\n", + drm_dbg_kms(&i915->drm, "use AUD_FREQ_CNTRL of 0x%x (init value 0x%x)\n", aud_freq, aud_freq_init); - dev_priv->display.audio.freq_cntrl = aud_freq; + i915->display.audio.freq_cntrl = aud_freq; } /* init with current cdclk */ - intel_audio_cdclk_change_post(dev_priv); + intel_audio_cdclk_change_post(i915); - dev_priv->display.audio.component_registered = true; + i915->display.audio.component_registered = true; } /** * i915_audio_component_cleanup - deregister the audio component - * @dev_priv: i915 device instance + * @i915: i915 device instance * * Deregisters the audio component, breaking any existing binding to the * corresponding snd_hda_intel driver's master component. */ -static void i915_audio_component_cleanup(struct drm_i915_private *dev_priv) +static void i915_audio_component_cleanup(struct drm_i915_private *i915) { - if (!dev_priv->display.audio.component_registered) + if (!i915->display.audio.component_registered) return; - component_del(dev_priv->drm.dev, &i915_audio_component_bind_ops); - dev_priv->display.audio.component_registered = false; + component_del(i915->drm.dev, &i915_audio_component_bind_ops); + i915->display.audio.component_registered = false; } /** * intel_audio_init() - Initialize the audio driver either using * component framework or using lpe audio bridge - * @dev_priv: the i915 drm device private data + * @i915: the i915 drm device private data * */ -void intel_audio_init(struct drm_i915_private *dev_priv) +void intel_audio_init(struct drm_i915_private *i915) { - if (intel_lpe_audio_init(dev_priv) < 0) - i915_audio_component_init(dev_priv); + if (intel_lpe_audio_init(i915) < 0) + i915_audio_component_init(i915); } /** * intel_audio_deinit() - deinitialize the audio driver - * @dev_priv: the i915 drm device private data + * @i915: the i915 drm device private data * */ -void intel_audio_deinit(struct drm_i915_private *dev_priv) +void intel_audio_deinit(struct drm_i915_private *i915) { - if (dev_priv->display.audio.lpe.platdev != NULL) - intel_lpe_audio_teardown(dev_priv); + if (i915->display.audio.lpe.platdev != NULL) + intel_lpe_audio_teardown(i915); else - i915_audio_component_cleanup(dev_priv); + i915_audio_component_cleanup(i915); } diff --git a/drivers/gpu/drm/i915/display/intel_audio_regs.h b/drivers/gpu/drm/i915/display/intel_audio_regs.h index d1e5844e3484..616e7b1275c4 100644 --- a/drivers/gpu/drm/i915/display/intel_audio_regs.h +++ b/drivers/gpu/drm/i915/display/intel_audio_regs.h @@ -6,18 +6,13 @@ #ifndef __INTEL_AUDIO_REGS_H__ #define __INTEL_AUDIO_REGS_H__ -#include "i915_reg_defs.h" - -#define G4X_AUD_VID_DID _MMIO(DISPLAY_MMIO_BASE(dev_priv) + 0x62020) -#define INTEL_AUDIO_DEVCL 0x808629FB -#define INTEL_AUDIO_DEVBLC 0x80862801 -#define INTEL_AUDIO_DEVCTG 0x80862802 +#include "intel_display_reg_defs.h" #define G4X_AUD_CNTL_ST _MMIO(0x620B4) -#define G4X_ELDV_DEVCL_DEVBLC (1 << 13) -#define G4X_ELDV_DEVCTG (1 << 14) -#define G4X_ELD_ADDR_MASK (0xf << 5) -#define G4X_ELD_ACK (1 << 4) +#define G4X_ELD_VALID REG_BIT(14) +#define G4X_ELD_BUFFER_SIZE_MASK REG_GENMASK(13, 9) +#define G4X_ELD_ADDRESS_MASK REG_GENMASK(8, 5) +#define G4X_ELD_ACK REG_BIT(4) #define G4X_HDMIW_HDMIEDID _MMIO(0x6210C) #define _IBX_HDMIW_HDMIEDID_A 0xE2050 @@ -28,12 +23,12 @@ #define _IBX_AUD_CNTL_ST_B 0xE21B4 #define IBX_AUD_CNTL_ST(pipe) _MMIO_PIPE(pipe, _IBX_AUD_CNTL_ST_A, \ _IBX_AUD_CNTL_ST_B) -#define IBX_ELD_BUFFER_SIZE_MASK (0x1f << 10) -#define IBX_ELD_ADDRESS_MASK (0x1f << 5) -#define IBX_ELD_ACK (1 << 4) +#define IBX_ELD_BUFFER_SIZE_MASK REG_GENMASK(14, 10) +#define IBX_ELD_ADDRESS_MASK REG_GENMASK(9, 5) +#define IBX_ELD_ACK REG_BIT(4) #define IBX_AUD_CNTL_ST2 _MMIO(0xE20C0) -#define IBX_CP_READY(port) ((1 << 1) << (((port) - 1) * 4)) -#define IBX_ELD_VALID(port) ((1 << 0) << (((port) - 1) * 4)) +#define IBX_CP_READY(port) REG_BIT(((port) - 1) * 4 + 1) +#define IBX_ELD_VALID(port) REG_BIT(((port) - 1) * 4 + 0) #define _CPT_HDMIW_HDMIEDID_A 0xE5050 #define _CPT_HDMIW_HDMIEDID_B 0xE5150 @@ -60,34 +55,30 @@ #define _VLV_AUD_CONFIG_A (VLV_DISPLAY_BASE + 0x62000) #define _VLV_AUD_CONFIG_B (VLV_DISPLAY_BASE + 0x62100) #define VLV_AUD_CFG(pipe) _MMIO_PIPE(pipe, _VLV_AUD_CONFIG_A, _VLV_AUD_CONFIG_B) - -#define AUD_CONFIG_N_VALUE_INDEX (1 << 29) -#define AUD_CONFIG_N_PROG_ENABLE (1 << 28) -#define AUD_CONFIG_UPPER_N_SHIFT 20 -#define AUD_CONFIG_UPPER_N_MASK (0xff << 20) -#define AUD_CONFIG_LOWER_N_SHIFT 4 -#define AUD_CONFIG_LOWER_N_MASK (0xfff << 4) -#define AUD_CONFIG_N_MASK (AUD_CONFIG_UPPER_N_MASK | AUD_CONFIG_LOWER_N_MASK) -#define AUD_CONFIG_N(n) \ - (((((n) >> 12) & 0xff) << AUD_CONFIG_UPPER_N_SHIFT) | \ - (((n) & 0xfff) << AUD_CONFIG_LOWER_N_SHIFT)) -#define AUD_CONFIG_PIXEL_CLOCK_HDMI_SHIFT 16 -#define AUD_CONFIG_PIXEL_CLOCK_HDMI_MASK (0xf << 16) -#define AUD_CONFIG_PIXEL_CLOCK_HDMI_25175 (0 << 16) -#define AUD_CONFIG_PIXEL_CLOCK_HDMI_25200 (1 << 16) -#define AUD_CONFIG_PIXEL_CLOCK_HDMI_27000 (2 << 16) -#define AUD_CONFIG_PIXEL_CLOCK_HDMI_27027 (3 << 16) -#define AUD_CONFIG_PIXEL_CLOCK_HDMI_54000 (4 << 16) -#define AUD_CONFIG_PIXEL_CLOCK_HDMI_54054 (5 << 16) -#define AUD_CONFIG_PIXEL_CLOCK_HDMI_74176 (6 << 16) -#define AUD_CONFIG_PIXEL_CLOCK_HDMI_74250 (7 << 16) -#define AUD_CONFIG_PIXEL_CLOCK_HDMI_148352 (8 << 16) -#define AUD_CONFIG_PIXEL_CLOCK_HDMI_148500 (9 << 16) -#define AUD_CONFIG_PIXEL_CLOCK_HDMI_296703 (10 << 16) -#define AUD_CONFIG_PIXEL_CLOCK_HDMI_297000 (11 << 16) -#define AUD_CONFIG_PIXEL_CLOCK_HDMI_593407 (12 << 16) -#define AUD_CONFIG_PIXEL_CLOCK_HDMI_594000 (13 << 16) -#define AUD_CONFIG_DISABLE_NCTS (1 << 3) +#define AUD_CONFIG_N_VALUE_INDEX REG_BIT(29) +#define AUD_CONFIG_N_PROG_ENABLE REG_BIT(28) +#define AUD_CONFIG_UPPER_N_MASK REG_GENMASK(27, 20) +#define AUD_CONFIG_LOWER_N_MASK REG_GENMASK(15, 4) +#define AUD_CONFIG_N_MASK (AUD_CONFIG_UPPER_N_MASK | \ + AUD_CONFIG_LOWER_N_MASK) +#define AUD_CONFIG_N(n) (REG_FIELD_PREP(AUD_CONFIG_UPPER_N_MASK, (n) >> 12) | \ + REG_FIELD_PREP(AUD_CONFIG_LOWER_N_MASK, (n) & 0xfff)) +#define AUD_CONFIG_PIXEL_CLOCK_HDMI_MASK REG_GENMASK(19, 16) +#define AUD_CONFIG_PIXEL_CLOCK_HDMI_25175 REG_FIELD_PREP(AUD_CONFIG_PIXEL_CLOCK_HDMI_MASK, 0) +#define AUD_CONFIG_PIXEL_CLOCK_HDMI_25200 REG_FIELD_PREP(AUD_CONFIG_PIXEL_CLOCK_HDMI_MASK, 1) +#define AUD_CONFIG_PIXEL_CLOCK_HDMI_27000 REG_FIELD_PREP(AUD_CONFIG_PIXEL_CLOCK_HDMI_MASK, 2) +#define AUD_CONFIG_PIXEL_CLOCK_HDMI_27027 REG_FIELD_PREP(AUD_CONFIG_PIXEL_CLOCK_HDMI_MASK, 3) +#define AUD_CONFIG_PIXEL_CLOCK_HDMI_54000 REG_FIELD_PREP(AUD_CONFIG_PIXEL_CLOCK_HDMI_MASK, 4) +#define AUD_CONFIG_PIXEL_CLOCK_HDMI_54054 REG_FIELD_PREP(AUD_CONFIG_PIXEL_CLOCK_HDMI_MASK, 5) +#define AUD_CONFIG_PIXEL_CLOCK_HDMI_74176 REG_FIELD_PREP(AUD_CONFIG_PIXEL_CLOCK_HDMI_MASK, 6) +#define AUD_CONFIG_PIXEL_CLOCK_HDMI_74250 REG_FIELD_PREP(AUD_CONFIG_PIXEL_CLOCK_HDMI_MASK, 7) +#define AUD_CONFIG_PIXEL_CLOCK_HDMI_148352 REG_FIELD_PREP(AUD_CONFIG_PIXEL_CLOCK_HDMI_MASK, 8) +#define AUD_CONFIG_PIXEL_CLOCK_HDMI_148500 REG_FIELD_PREP(AUD_CONFIG_PIXEL_CLOCK_HDMI_MASK, 9) +#define AUD_CONFIG_PIXEL_CLOCK_HDMI_296703 REG_FIELD_PREP(AUD_CONFIG_PIXEL_CLOCK_HDMI_MASK, 10) +#define AUD_CONFIG_PIXEL_CLOCK_HDMI_297000 REG_FIELD_PREP(AUD_CONFIG_PIXEL_CLOCK_HDMI_MASK, 11) +#define AUD_CONFIG_PIXEL_CLOCK_HDMI_593407 REG_FIELD_PREP(AUD_CONFIG_PIXEL_CLOCK_HDMI_MASK, 12) +#define AUD_CONFIG_PIXEL_CLOCK_HDMI_594000 REG_FIELD_PREP(AUD_CONFIG_PIXEL_CLOCK_HDMI_MASK, 13) +#define AUD_CONFIG_DISABLE_NCTS REG_BIT(3) #define _HSW_AUD_CONFIG_A 0x65000 #define _HSW_AUD_CONFIG_B 0x65100 @@ -100,9 +91,9 @@ #define _HSW_AUD_M_CTS_ENABLE_A 0x65028 #define _HSW_AUD_M_CTS_ENABLE_B 0x65128 #define HSW_AUD_M_CTS_ENABLE(trans) _MMIO_TRANS(trans, _HSW_AUD_M_CTS_ENABLE_A, _HSW_AUD_M_CTS_ENABLE_B) -#define AUD_M_CTS_M_VALUE_INDEX (1 << 21) -#define AUD_M_CTS_M_PROG_ENABLE (1 << 20) -#define AUD_CONFIG_M_MASK 0xfffff +#define AUD_M_CTS_M_VALUE_INDEX REG_BIT(21) +#define AUD_M_CTS_M_PROG_ENABLE REG_BIT(20) +#define AUD_CONFIG_M_MASK REG_GENMASK(19, 0) #define _HSW_AUD_DIP_ELD_CTRL_ST_A 0x650b4 #define _HSW_AUD_DIP_ELD_CTRL_ST_B 0x651b4 @@ -130,11 +121,11 @@ #define AUD_DP_2DOT0_CTRL(trans) _MMIO_TRANS(trans, _AUD_TCA_DP_2DOT0_CTRL, _AUD_TCB_DP_2DOT0_CTRL) #define AUD_ENABLE_SDP_SPLIT REG_BIT(31) -#define HSW_AUD_CHICKENBIT _MMIO(0x65f10) -#define SKL_AUD_CODEC_WAKE_SIGNAL (1 << 15) +#define HSW_AUD_CHICKENBIT _MMIO(0x65f10) +#define SKL_AUD_CODEC_WAKE_SIGNAL REG_BIT(15) #define AUD_FREQ_CNTRL _MMIO(0x65900) -#define AUD_PIN_BUF_CTL _MMIO(0x48414) +#define AUD_PIN_BUF_CTL _MMIO(0x48414) #define AUD_PIN_BUF_ENABLE REG_BIT(31) #define AUD_TS_CDCLK_M _MMIO(0x65ea0) diff --git a/drivers/gpu/drm/i915/display/intel_backlight.c b/drivers/gpu/drm/i915/display/intel_backlight.c index beba39a38c87..71af88a70461 100644 --- a/drivers/gpu/drm/i915/display/intel_backlight.c +++ b/drivers/gpu/drm/i915/display/intel_backlight.c @@ -10,6 +10,7 @@ #include <acpi/video.h> +#include "i915_reg.h" #include "intel_backlight.h" #include "intel_backlight_regs.h" #include "intel_connector.h" diff --git a/drivers/gpu/drm/i915/display/intel_backlight_regs.h b/drivers/gpu/drm/i915/display/intel_backlight_regs.h index 50c1210f6d5d..344eb8096bd2 100644 --- a/drivers/gpu/drm/i915/display/intel_backlight_regs.h +++ b/drivers/gpu/drm/i915/display/intel_backlight_regs.h @@ -6,7 +6,7 @@ #ifndef __INTEL_BACKLIGHT_REGS_H__ #define __INTEL_BACKLIGHT_REGS_H__ -#include "i915_reg_defs.h" +#include "intel_display_reg_defs.h" #define _VLV_BLC_PWM_CTL2_A (DISPLAY_MMIO_BASE(dev_priv) + 0x61250) #define _VLV_BLC_PWM_CTL2_B (DISPLAY_MMIO_BASE(dev_priv) + 0x61350) diff --git a/drivers/gpu/drm/i915/display/intel_bios.c b/drivers/gpu/drm/i915/display/intel_bios.c index 28bdb936cd1f..572a4e3769f3 100644 --- a/drivers/gpu/drm/i915/display/intel_bios.c +++ b/drivers/gpu/drm/i915/display/intel_bios.c @@ -414,7 +414,7 @@ static void *generate_lfp_data_ptrs(struct drm_i915_private *i915, ptrs->lvds_entries++; if (size != 0 || ptrs->lvds_entries != 3) { - kfree(ptrs); + kfree(ptrs_block); return NULL; } @@ -2188,7 +2188,7 @@ static u8 map_ddc_pin(struct drm_i915_private *i915, u8 vbt_pin) const u8 *ddc_pin_map; int n_entries; - if (IS_ALDERLAKE_P(i915)) { + if (HAS_PCH_MTP(i915) || IS_ALDERLAKE_P(i915)) { ddc_pin_map = adlp_ddc_pin_map; n_entries = ARRAY_SIZE(adlp_ddc_pin_map); } else if (IS_ALDERLAKE_S(i915)) { @@ -2676,6 +2676,14 @@ static void print_ddi_port(const struct intel_bios_encoder_data *devdata, drm_dbg_kms(&i915->drm, "Port %c VBT DP max link rate: %d\n", port_name(port), dp_max_link_rate); + + /* + * FIXME need to implement support for VBT + * vswing/preemph tables should this ever trigger. + */ + drm_WARN(&i915->drm, child->use_vbt_vswing, + "Port %c asks to use VBT vswing/preemph tables\n", + port_name(port)); } static void parse_ddi_port(struct intel_bios_encoder_data *devdata) diff --git a/drivers/gpu/drm/i915/display/intel_bw.c b/drivers/gpu/drm/i915/display/intel_bw.c index 4ace026b29bd..1c236f02b380 100644 --- a/drivers/gpu/drm/i915/display/intel_bw.c +++ b/drivers/gpu/drm/i915/display/intel_bw.c @@ -439,7 +439,8 @@ static int tgl_get_bw_info(struct drm_i915_private *dev_priv, const struct intel return ret; } - if (dram_info->type == INTEL_DRAM_LPDDR4 || dram_info->type == INTEL_DRAM_LPDDR5) + if (DISPLAY_VER(dev_priv) < 14 && + (dram_info->type == INTEL_DRAM_LPDDR4 || dram_info->type == INTEL_DRAM_LPDDR5)) num_channels *= 2; qi.deinterleave = qi.deinterleave ? : DIV_ROUND_UP(num_channels, is_y_tile ? 4 : 2); diff --git a/drivers/gpu/drm/i915/display/intel_cdclk.c b/drivers/gpu/drm/i915/display/intel_cdclk.c index ed05070b7307..b74e36d76013 100644 --- a/drivers/gpu/drm/i915/display/intel_cdclk.c +++ b/drivers/gpu/drm/i915/display/intel_cdclk.c @@ -24,6 +24,7 @@ #include <linux/time.h> #include "hsw_ips.h" +#include "i915_reg.h" #include "intel_atomic.h" #include "intel_atomic_plane.h" #include "intel_audio.h" @@ -1220,11 +1221,6 @@ static void skl_cdclk_uninit_hw(struct drm_i915_private *dev_priv) skl_set_cdclk(dev_priv, &cdclk_config, INVALID_PIPE); } -static bool has_cdclk_squasher(struct drm_i915_private *i915) -{ - return IS_DG2(i915); -} - struct intel_cdclk_vals { u32 cdclk; u16 refclk; @@ -1520,7 +1516,7 @@ static void bxt_get_cdclk(struct drm_i915_private *dev_priv, return; } - if (has_cdclk_squasher(dev_priv)) + if (HAS_CDCLK_SQUASH(dev_priv)) squash_ctl = intel_de_read(dev_priv, CDCLK_SQUASH_CTL); if (squash_ctl & CDCLK_SQUASH_ENABLE) { @@ -1689,6 +1685,38 @@ static u32 cdclk_squash_waveform(struct drm_i915_private *dev_priv, return 0xffff; } +static void icl_cdclk_pll_update(struct drm_i915_private *i915, int vco) +{ + if (i915->display.cdclk.hw.vco != 0 && + i915->display.cdclk.hw.vco != vco) + icl_cdclk_pll_disable(i915); + + if (i915->display.cdclk.hw.vco != vco) + icl_cdclk_pll_enable(i915, vco); +} + +static void bxt_cdclk_pll_update(struct drm_i915_private *i915, int vco) +{ + if (i915->display.cdclk.hw.vco != 0 && + i915->display.cdclk.hw.vco != vco) + bxt_de_pll_disable(i915); + + if (i915->display.cdclk.hw.vco != vco) + bxt_de_pll_enable(i915, vco); +} + +static void dg2_cdclk_squash_program(struct drm_i915_private *i915, + u16 waveform) +{ + u32 squash_ctl = 0; + + if (waveform) + squash_ctl = CDCLK_SQUASH_ENABLE | + CDCLK_SQUASH_WINDOW_SIZE(0xf) | waveform; + + intel_de_write(i915, CDCLK_SQUASH_CTL, squash_ctl); +} + static void bxt_set_cdclk(struct drm_i915_private *dev_priv, const struct intel_cdclk_config *cdclk_config, enum pipe pipe) @@ -1724,21 +1752,10 @@ static void bxt_set_cdclk(struct drm_i915_private *dev_priv, if (HAS_CDCLK_CRAWL(dev_priv) && dev_priv->display.cdclk.hw.vco > 0 && vco > 0) { if (dev_priv->display.cdclk.hw.vco != vco) adlp_cdclk_pll_crawl(dev_priv, vco); - } else if (DISPLAY_VER(dev_priv) >= 11) { - if (dev_priv->display.cdclk.hw.vco != 0 && - dev_priv->display.cdclk.hw.vco != vco) - icl_cdclk_pll_disable(dev_priv); - - if (dev_priv->display.cdclk.hw.vco != vco) - icl_cdclk_pll_enable(dev_priv, vco); - } else { - if (dev_priv->display.cdclk.hw.vco != 0 && - dev_priv->display.cdclk.hw.vco != vco) - bxt_de_pll_disable(dev_priv); - - if (dev_priv->display.cdclk.hw.vco != vco) - bxt_de_pll_enable(dev_priv, vco); - } + } else if (DISPLAY_VER(dev_priv) >= 11) + icl_cdclk_pll_update(dev_priv, vco); + else + bxt_cdclk_pll_update(dev_priv, vco); waveform = cdclk_squash_waveform(dev_priv, cdclk); @@ -1747,15 +1764,8 @@ static void bxt_set_cdclk(struct drm_i915_private *dev_priv, else clock = cdclk; - if (has_cdclk_squasher(dev_priv)) { - u32 squash_ctl = 0; - - if (waveform) - squash_ctl = CDCLK_SQUASH_ENABLE | - CDCLK_SQUASH_WINDOW_SIZE(0xf) | waveform; - - intel_de_write(dev_priv, CDCLK_SQUASH_CTL, squash_ctl); - } + if (HAS_CDCLK_SQUASH(dev_priv)) + dg2_cdclk_squash_program(dev_priv, waveform); val = bxt_cdclk_cd2x_div_sel(dev_priv, clock, vco) | bxt_cdclk_cd2x_pipe(dev_priv, pipe) | @@ -1845,7 +1855,7 @@ static void bxt_sanitize_cdclk(struct drm_i915_private *dev_priv) expected = skl_cdclk_decimal(cdclk); /* Figure out what CD2X divider we should be using for this cdclk */ - if (has_cdclk_squasher(dev_priv)) + if (HAS_CDCLK_SQUASH(dev_priv)) clock = dev_priv->display.cdclk.hw.vco / 2; else clock = dev_priv->display.cdclk.hw.cdclk; @@ -1976,7 +1986,7 @@ static bool intel_cdclk_can_squash(struct drm_i915_private *dev_priv, * the moment all platforms with squasher use a fixed cd2x * divider. */ - if (!has_cdclk_squasher(dev_priv)) + if (!HAS_CDCLK_SQUASH(dev_priv)) return false; return a->cdclk != b->cdclk && @@ -2028,7 +2038,7 @@ static bool intel_cdclk_can_cd2x_update(struct drm_i915_private *dev_priv, * the moment all platforms with squasher use a fixed cd2x * divider. */ - if (has_cdclk_squasher(dev_priv)) + if (HAS_CDCLK_SQUASH(dev_priv)) return false; return a->cdclk != b->cdclk && @@ -2464,10 +2474,6 @@ static int bdw_modeset_calc_cdclk(struct intel_cdclk_state *cdclk_state) if (min_cdclk < 0) return min_cdclk; - /* - * FIXME should also account for plane ratio - * once 64bpp pixel formats are supported. - */ cdclk = bdw_calc_cdclk(min_cdclk); cdclk_state->logical.cdclk = cdclk; @@ -2534,10 +2540,6 @@ static int skl_modeset_calc_cdclk(struct intel_cdclk_state *cdclk_state) vco = skl_dpll0_vco(cdclk_state); - /* - * FIXME should also account for plane ratio - * once 64bpp pixel formats are supported. - */ cdclk = skl_calc_cdclk(min_cdclk, vco); cdclk_state->logical.vco = vco; @@ -2754,7 +2756,7 @@ int intel_modeset_calc_cdclk(struct intel_atomic_state *state) if (IS_ERR(crtc_state)) return PTR_ERR(crtc_state); - if (drm_atomic_crtc_needs_modeset(&crtc_state->uapi)) + if (intel_crtc_needs_modeset(crtc_state)) pipe = INVALID_PIPE; } @@ -2762,12 +2764,12 @@ int intel_modeset_calc_cdclk(struct intel_atomic_state *state) &old_cdclk_state->actual, &new_cdclk_state->actual)) { drm_dbg_kms(&dev_priv->drm, - "Can change cdclk via squasher\n"); + "Can change cdclk via squashing\n"); } else if (intel_cdclk_can_crawl(dev_priv, &old_cdclk_state->actual, &new_cdclk_state->actual)) { drm_dbg_kms(&dev_priv->drm, - "Can change cdclk via crawl\n"); + "Can change cdclk via crawling\n"); } else if (pipe != INVALID_PIPE) { new_cdclk_state->pipe = pipe; @@ -2777,7 +2779,7 @@ int intel_modeset_calc_cdclk(struct intel_atomic_state *state) } else if (intel_cdclk_needs_modeset(&old_cdclk_state->actual, &new_cdclk_state->actual)) { /* All pipes must be switched off while we change the cdclk. */ - ret = intel_modeset_all_pipes(state); + ret = intel_modeset_all_pipes(state, "CDCLK change"); if (ret) return ret; diff --git a/drivers/gpu/drm/i915/display/intel_color.c b/drivers/gpu/drm/i915/display/intel_color.c index 6bda4274eae9..250e83f1f5ac 100644 --- a/drivers/gpu/drm/i915/display/intel_color.c +++ b/drivers/gpu/drm/i915/display/intel_color.c @@ -22,12 +22,11 @@ * */ +#include "i915_reg.h" #include "intel_color.h" #include "intel_de.h" #include "intel_display_types.h" -#include "intel_dpll.h" #include "intel_dsb.h" -#include "vlv_dsi_pll.h" struct intel_color_funcs { int (*color_check)(struct intel_crtc_state *crtc_state); @@ -186,31 +185,31 @@ static void ilk_update_pipe_csc(struct intel_crtc *crtc, const u16 coeff[9], const u16 postoff[3]) { - struct drm_i915_private *dev_priv = to_i915(crtc->base.dev); + struct drm_i915_private *i915 = to_i915(crtc->base.dev); enum pipe pipe = crtc->pipe; - intel_de_write_fw(dev_priv, PIPE_CSC_PREOFF_HI(pipe), preoff[0]); - intel_de_write_fw(dev_priv, PIPE_CSC_PREOFF_ME(pipe), preoff[1]); - intel_de_write_fw(dev_priv, PIPE_CSC_PREOFF_LO(pipe), preoff[2]); + intel_de_write_fw(i915, PIPE_CSC_PREOFF_HI(pipe), preoff[0]); + intel_de_write_fw(i915, PIPE_CSC_PREOFF_ME(pipe), preoff[1]); + intel_de_write_fw(i915, PIPE_CSC_PREOFF_LO(pipe), preoff[2]); - intel_de_write_fw(dev_priv, PIPE_CSC_COEFF_RY_GY(pipe), + intel_de_write_fw(i915, PIPE_CSC_COEFF_RY_GY(pipe), coeff[0] << 16 | coeff[1]); - intel_de_write_fw(dev_priv, PIPE_CSC_COEFF_BY(pipe), coeff[2] << 16); + intel_de_write_fw(i915, PIPE_CSC_COEFF_BY(pipe), coeff[2] << 16); - intel_de_write_fw(dev_priv, PIPE_CSC_COEFF_RU_GU(pipe), + intel_de_write_fw(i915, PIPE_CSC_COEFF_RU_GU(pipe), coeff[3] << 16 | coeff[4]); - intel_de_write_fw(dev_priv, PIPE_CSC_COEFF_BU(pipe), coeff[5] << 16); + intel_de_write_fw(i915, PIPE_CSC_COEFF_BU(pipe), coeff[5] << 16); - intel_de_write_fw(dev_priv, PIPE_CSC_COEFF_RV_GV(pipe), + intel_de_write_fw(i915, PIPE_CSC_COEFF_RV_GV(pipe), coeff[6] << 16 | coeff[7]); - intel_de_write_fw(dev_priv, PIPE_CSC_COEFF_BV(pipe), coeff[8] << 16); + intel_de_write_fw(i915, PIPE_CSC_COEFF_BV(pipe), coeff[8] << 16); - if (DISPLAY_VER(dev_priv) >= 7) { - intel_de_write_fw(dev_priv, PIPE_CSC_POSTOFF_HI(pipe), + if (DISPLAY_VER(i915) >= 7) { + intel_de_write_fw(i915, PIPE_CSC_POSTOFF_HI(pipe), postoff[0]); - intel_de_write_fw(dev_priv, PIPE_CSC_POSTOFF_ME(pipe), + intel_de_write_fw(i915, PIPE_CSC_POSTOFF_ME(pipe), postoff[1]); - intel_de_write_fw(dev_priv, PIPE_CSC_POSTOFF_LO(pipe), + intel_de_write_fw(i915, PIPE_CSC_POSTOFF_LO(pipe), postoff[2]); } } @@ -220,55 +219,55 @@ static void icl_update_output_csc(struct intel_crtc *crtc, const u16 coeff[9], const u16 postoff[3]) { - struct drm_i915_private *dev_priv = to_i915(crtc->base.dev); + struct drm_i915_private *i915 = to_i915(crtc->base.dev); enum pipe pipe = crtc->pipe; - intel_de_write_fw(dev_priv, PIPE_CSC_OUTPUT_PREOFF_HI(pipe), preoff[0]); - intel_de_write_fw(dev_priv, PIPE_CSC_OUTPUT_PREOFF_ME(pipe), preoff[1]); - intel_de_write_fw(dev_priv, PIPE_CSC_OUTPUT_PREOFF_LO(pipe), preoff[2]); + intel_de_write_fw(i915, PIPE_CSC_OUTPUT_PREOFF_HI(pipe), preoff[0]); + intel_de_write_fw(i915, PIPE_CSC_OUTPUT_PREOFF_ME(pipe), preoff[1]); + intel_de_write_fw(i915, PIPE_CSC_OUTPUT_PREOFF_LO(pipe), preoff[2]); - intel_de_write_fw(dev_priv, PIPE_CSC_OUTPUT_COEFF_RY_GY(pipe), + intel_de_write_fw(i915, PIPE_CSC_OUTPUT_COEFF_RY_GY(pipe), coeff[0] << 16 | coeff[1]); - intel_de_write_fw(dev_priv, PIPE_CSC_OUTPUT_COEFF_BY(pipe), + intel_de_write_fw(i915, PIPE_CSC_OUTPUT_COEFF_BY(pipe), coeff[2] << 16); - intel_de_write_fw(dev_priv, PIPE_CSC_OUTPUT_COEFF_RU_GU(pipe), + intel_de_write_fw(i915, PIPE_CSC_OUTPUT_COEFF_RU_GU(pipe), coeff[3] << 16 | coeff[4]); - intel_de_write_fw(dev_priv, PIPE_CSC_OUTPUT_COEFF_BU(pipe), + intel_de_write_fw(i915, PIPE_CSC_OUTPUT_COEFF_BU(pipe), coeff[5] << 16); - intel_de_write_fw(dev_priv, PIPE_CSC_OUTPUT_COEFF_RV_GV(pipe), + intel_de_write_fw(i915, PIPE_CSC_OUTPUT_COEFF_RV_GV(pipe), coeff[6] << 16 | coeff[7]); - intel_de_write_fw(dev_priv, PIPE_CSC_OUTPUT_COEFF_BV(pipe), + intel_de_write_fw(i915, PIPE_CSC_OUTPUT_COEFF_BV(pipe), coeff[8] << 16); - intel_de_write_fw(dev_priv, PIPE_CSC_OUTPUT_POSTOFF_HI(pipe), postoff[0]); - intel_de_write_fw(dev_priv, PIPE_CSC_OUTPUT_POSTOFF_ME(pipe), postoff[1]); - intel_de_write_fw(dev_priv, PIPE_CSC_OUTPUT_POSTOFF_LO(pipe), postoff[2]); + intel_de_write_fw(i915, PIPE_CSC_OUTPUT_POSTOFF_HI(pipe), postoff[0]); + intel_de_write_fw(i915, PIPE_CSC_OUTPUT_POSTOFF_ME(pipe), postoff[1]); + intel_de_write_fw(i915, PIPE_CSC_OUTPUT_POSTOFF_LO(pipe), postoff[2]); } static bool ilk_csc_limited_range(const struct intel_crtc_state *crtc_state) { - struct drm_i915_private *dev_priv = to_i915(crtc_state->uapi.crtc->dev); + struct drm_i915_private *i915 = to_i915(crtc_state->uapi.crtc->dev); /* * FIXME if there's a gamma LUT after the CSC, we should * do the range compression using the gamma LUT instead. */ return crtc_state->limited_color_range && - (IS_HASWELL(dev_priv) || IS_BROADWELL(dev_priv) || - IS_DISPLAY_VER(dev_priv, 9, 10)); + (IS_HASWELL(i915) || IS_BROADWELL(i915) || + IS_DISPLAY_VER(i915, 9, 10)); } static void ilk_csc_convert_ctm(const struct intel_crtc_state *crtc_state, - u16 coeffs[9]) + u16 coeffs[9], bool limited_color_range) { const struct drm_color_ctm *ctm = crtc_state->hw.ctm->data; const u64 *input; u64 temp[9]; int i; - if (ilk_csc_limited_range(crtc_state)) + if (limited_color_range) input = ctm_mult_by_limited(temp, ctm->matrix); else input = ctm->matrix; @@ -315,13 +314,13 @@ static void ilk_csc_convert_ctm(const struct intel_crtc_state *crtc_state, static void ilk_load_csc_matrix(const struct intel_crtc_state *crtc_state) { struct intel_crtc *crtc = to_intel_crtc(crtc_state->uapi.crtc); - struct drm_i915_private *dev_priv = to_i915(crtc->base.dev); + struct drm_i915_private *i915 = to_i915(crtc->base.dev); bool limited_color_range = ilk_csc_limited_range(crtc_state); if (crtc_state->hw.ctm) { u16 coeff[9]; - ilk_csc_convert_ctm(crtc_state, coeff); + ilk_csc_convert_ctm(crtc_state, coeff, limited_color_range); ilk_update_pipe_csc(crtc, ilk_csc_off_zero, coeff, limited_color_range ? ilk_csc_postoff_limited_range : @@ -341,7 +340,7 @@ static void ilk_load_csc_matrix(const struct intel_crtc_state *crtc_state) * LUT is needed but CSC is not we need to load an * identity matrix. */ - drm_WARN_ON(&dev_priv->drm, !IS_GEMINILAKE(dev_priv)); + drm_WARN_ON(&i915->drm, !IS_GEMINILAKE(i915)); ilk_update_pipe_csc(crtc, ilk_csc_off_zero, ilk_csc_coeff_identity, @@ -356,7 +355,7 @@ static void icl_load_csc_matrix(const struct intel_crtc_state *crtc_state) if (crtc_state->hw.ctm) { u16 coeff[9]; - ilk_csc_convert_ctm(crtc_state, coeff); + ilk_csc_convert_ctm(crtc_state, coeff, false); ilk_update_pipe_csc(crtc, ilk_csc_off_zero, coeff, ilk_csc_off_zero); } @@ -375,7 +374,7 @@ static void icl_load_csc_matrix(const struct intel_crtc_state *crtc_state) static void chv_load_cgm_csc(struct intel_crtc *crtc, const struct drm_property_blob *blob) { - struct drm_i915_private *dev_priv = to_i915(crtc->base.dev); + struct drm_i915_private *i915 = to_i915(crtc->base.dev); const struct drm_color_ctm *ctm = blob->data; enum pipe pipe = crtc->pipe; u16 coeffs[9]; @@ -399,15 +398,15 @@ static void chv_load_cgm_csc(struct intel_crtc *crtc, coeffs[i] |= (abs_coeff >> 20) & 0xfff; } - intel_de_write_fw(dev_priv, CGM_PIPE_CSC_COEFF01(pipe), + intel_de_write_fw(i915, CGM_PIPE_CSC_COEFF01(pipe), coeffs[1] << 16 | coeffs[0]); - intel_de_write_fw(dev_priv, CGM_PIPE_CSC_COEFF23(pipe), + intel_de_write_fw(i915, CGM_PIPE_CSC_COEFF23(pipe), coeffs[3] << 16 | coeffs[2]); - intel_de_write_fw(dev_priv, CGM_PIPE_CSC_COEFF45(pipe), + intel_de_write_fw(i915, CGM_PIPE_CSC_COEFF45(pipe), coeffs[5] << 16 | coeffs[4]); - intel_de_write_fw(dev_priv, CGM_PIPE_CSC_COEFF67(pipe), + intel_de_write_fw(i915, CGM_PIPE_CSC_COEFF67(pipe), coeffs[7] << 16 | coeffs[6]); - intel_de_write_fw(dev_priv, CGM_PIPE_CSC_COEFF8(pipe), + intel_de_write_fw(i915, CGM_PIPE_CSC_COEFF8(pipe), coeffs[8]); } @@ -426,32 +425,32 @@ static u32 intel_color_lut_pack(u32 val, int bit_precision) static u32 i9xx_lut_8(const struct drm_color_lut *color) { - return drm_color_lut_extract(color->red, 8) << 16 | - drm_color_lut_extract(color->green, 8) << 8 | - drm_color_lut_extract(color->blue, 8); + return REG_FIELD_PREP(PALETTE_RED_MASK, drm_color_lut_extract(color->red, 8)) | + REG_FIELD_PREP(PALETTE_GREEN_MASK, drm_color_lut_extract(color->green, 8)) | + REG_FIELD_PREP(PALETTE_BLUE_MASK, drm_color_lut_extract(color->blue, 8)); } static void i9xx_lut_8_pack(struct drm_color_lut *entry, u32 val) { - entry->red = intel_color_lut_pack(REG_FIELD_GET(LGC_PALETTE_RED_MASK, val), 8); - entry->green = intel_color_lut_pack(REG_FIELD_GET(LGC_PALETTE_GREEN_MASK, val), 8); - entry->blue = intel_color_lut_pack(REG_FIELD_GET(LGC_PALETTE_BLUE_MASK, val), 8); + entry->red = intel_color_lut_pack(REG_FIELD_GET(PALETTE_RED_MASK, val), 8); + entry->green = intel_color_lut_pack(REG_FIELD_GET(PALETTE_GREEN_MASK, val), 8); + entry->blue = intel_color_lut_pack(REG_FIELD_GET(PALETTE_BLUE_MASK, val), 8); } /* i965+ "10.6" bit interpolated format "even DW" (low 8 bits) */ static u32 i965_lut_10p6_ldw(const struct drm_color_lut *color) { - return (color->red & 0xff) << 16 | - (color->green & 0xff) << 8 | - (color->blue & 0xff); + return REG_FIELD_PREP(PALETTE_RED_MASK, color->red & 0xff) | + REG_FIELD_PREP(PALETTE_GREEN_MASK, color->green & 0xff) | + REG_FIELD_PREP(PALETTE_BLUE_MASK, color->blue & 0xff); } /* i965+ "10.6" interpolated format "odd DW" (high 8 bits) */ static u32 i965_lut_10p6_udw(const struct drm_color_lut *color) { - return (color->red >> 8) << 16 | - (color->green >> 8) << 8 | - (color->blue >> 8); + return REG_FIELD_PREP(PALETTE_RED_MASK, color->red >> 8) | + REG_FIELD_PREP(PALETTE_GREEN_MASK, color->green >> 8) | + REG_FIELD_PREP(PALETTE_BLUE_MASK, color->blue >> 8); } static void i965_lut_10p6_pack(struct drm_color_lut *entry, u32 ldw, u32 udw) @@ -472,26 +471,42 @@ static u16 i965_lut_11p6_max_pack(u32 val) static u32 ilk_lut_10(const struct drm_color_lut *color) { - return drm_color_lut_extract(color->red, 10) << 20 | - drm_color_lut_extract(color->green, 10) << 10 | - drm_color_lut_extract(color->blue, 10); + return REG_FIELD_PREP(PREC_PALETTE_10_RED_MASK, drm_color_lut_extract(color->red, 10)) | + REG_FIELD_PREP(PREC_PALETTE_10_GREEN_MASK, drm_color_lut_extract(color->green, 10)) | + REG_FIELD_PREP(PREC_PALETTE_10_BLUE_MASK, drm_color_lut_extract(color->blue, 10)); } static void ilk_lut_10_pack(struct drm_color_lut *entry, u32 val) { - entry->red = intel_color_lut_pack(REG_FIELD_GET(PREC_PALETTE_RED_MASK, val), 10); - entry->green = intel_color_lut_pack(REG_FIELD_GET(PREC_PALETTE_GREEN_MASK, val), 10); - entry->blue = intel_color_lut_pack(REG_FIELD_GET(PREC_PALETTE_BLUE_MASK, val), 10); + entry->red = intel_color_lut_pack(REG_FIELD_GET(PREC_PALETTE_10_RED_MASK, val), 10); + entry->green = intel_color_lut_pack(REG_FIELD_GET(PREC_PALETTE_10_GREEN_MASK, val), 10); + entry->blue = intel_color_lut_pack(REG_FIELD_GET(PREC_PALETTE_10_BLUE_MASK, val), 10); +} + +/* ilk+ "12.4" interpolated format (low 6 bits) */ +static u32 ilk_lut_12p4_ldw(const struct drm_color_lut *color) +{ + return REG_FIELD_PREP(PREC_PALETTE_12P4_RED_LDW_MASK, color->red & 0x3f) | + REG_FIELD_PREP(PREC_PALETTE_12P4_GREEN_LDW_MASK, color->green & 0x3f) | + REG_FIELD_PREP(PREC_PALETTE_12P4_BLUE_LDW_MASK, color->blue & 0x3f); +} + +/* ilk+ "12.4" interpolated format (high 10 bits) */ +static u32 ilk_lut_12p4_udw(const struct drm_color_lut *color) +{ + return REG_FIELD_PREP(PREC_PALETTE_12P4_RED_UDW_MASK, color->red >> 6) | + REG_FIELD_PREP(PREC_PALETTE_12P4_GREEN_UDW_MASK, color->green >> 6) | + REG_FIELD_PREP(PREC_PALETTE_12P4_BLUE_UDW_MASK, color->blue >> 6); } -static void icl_lut_multi_seg_pack(struct drm_color_lut *entry, u32 ldw, u32 udw) +static void ilk_lut_12p4_pack(struct drm_color_lut *entry, u32 ldw, u32 udw) { - entry->red = REG_FIELD_GET(PAL_PREC_MULTI_SEG_RED_UDW_MASK, udw) << 6 | - REG_FIELD_GET(PAL_PREC_MULTI_SEG_RED_LDW_MASK, ldw); - entry->green = REG_FIELD_GET(PAL_PREC_MULTI_SEG_GREEN_UDW_MASK, udw) << 6 | - REG_FIELD_GET(PAL_PREC_MULTI_SEG_GREEN_LDW_MASK, ldw); - entry->blue = REG_FIELD_GET(PAL_PREC_MULTI_SEG_BLUE_UDW_MASK, udw) << 6 | - REG_FIELD_GET(PAL_PREC_MULTI_SEG_BLUE_LDW_MASK, ldw); + entry->red = REG_FIELD_GET(PREC_PALETTE_12P4_RED_UDW_MASK, udw) << 6 | + REG_FIELD_GET(PREC_PALETTE_12P4_RED_LDW_MASK, ldw); + entry->green = REG_FIELD_GET(PREC_PALETTE_12P4_GREEN_UDW_MASK, udw) << 6 | + REG_FIELD_GET(PREC_PALETTE_12P4_GREEN_LDW_MASK, ldw); + entry->blue = REG_FIELD_GET(PREC_PALETTE_12P4_BLUE_UDW_MASK, udw) << 6 | + REG_FIELD_GET(PREC_PALETTE_12P4_BLUE_LDW_MASK, ldw); } static void icl_color_commit_noarm(const struct intel_crtc_state *crtc_state) @@ -513,31 +528,31 @@ static void i9xx_color_commit_arm(const struct intel_crtc_state *crtc_state) static void ilk_color_commit_arm(const struct intel_crtc_state *crtc_state) { struct intel_crtc *crtc = to_intel_crtc(crtc_state->uapi.crtc); - struct drm_i915_private *dev_priv = to_i915(crtc->base.dev); + struct drm_i915_private *i915 = to_i915(crtc->base.dev); /* update PIPECONF GAMMA_MODE */ ilk_set_pipeconf(crtc_state); - intel_de_write_fw(dev_priv, PIPE_CSC_MODE(crtc->pipe), + intel_de_write_fw(i915, PIPE_CSC_MODE(crtc->pipe), crtc_state->csc_mode); } static void hsw_color_commit_arm(const struct intel_crtc_state *crtc_state) { struct intel_crtc *crtc = to_intel_crtc(crtc_state->uapi.crtc); - struct drm_i915_private *dev_priv = to_i915(crtc->base.dev); + struct drm_i915_private *i915 = to_i915(crtc->base.dev); - intel_de_write(dev_priv, GAMMA_MODE(crtc->pipe), + intel_de_write(i915, GAMMA_MODE(crtc->pipe), crtc_state->gamma_mode); - intel_de_write_fw(dev_priv, PIPE_CSC_MODE(crtc->pipe), + intel_de_write_fw(i915, PIPE_CSC_MODE(crtc->pipe), crtc_state->csc_mode); } static void skl_color_commit_arm(const struct intel_crtc_state *crtc_state) { struct intel_crtc *crtc = to_intel_crtc(crtc_state->uapi.crtc); - struct drm_i915_private *dev_priv = to_i915(crtc->base.dev); + struct drm_i915_private *i915 = to_i915(crtc->base.dev); enum pipe pipe = crtc->pipe; u32 val = 0; @@ -550,15 +565,65 @@ static void skl_color_commit_arm(const struct intel_crtc_state *crtc_state) val |= SKL_BOTTOM_COLOR_GAMMA_ENABLE; if (crtc_state->csc_enable) val |= SKL_BOTTOM_COLOR_CSC_ENABLE; - intel_de_write(dev_priv, SKL_BOTTOM_COLOR(pipe), val); + intel_de_write(i915, SKL_BOTTOM_COLOR(pipe), val); - intel_de_write(dev_priv, GAMMA_MODE(crtc->pipe), + intel_de_write(i915, GAMMA_MODE(crtc->pipe), crtc_state->gamma_mode); - intel_de_write_fw(dev_priv, PIPE_CSC_MODE(crtc->pipe), + intel_de_write_fw(i915, PIPE_CSC_MODE(crtc->pipe), crtc_state->csc_mode); } +static struct drm_property_blob * +create_linear_lut(struct drm_i915_private *i915, int lut_size) +{ + struct drm_property_blob *blob; + struct drm_color_lut *lut; + int i; + + blob = drm_property_create_blob(&i915->drm, + sizeof(lut[0]) * lut_size, + NULL); + if (IS_ERR(blob)) + return blob; + + lut = blob->data; + + for (i = 0; i < lut_size; i++) { + u16 val = 0xffff * i / (lut_size - 1); + + lut[i].red = val; + lut[i].green = val; + lut[i].blue = val; + } + + return blob; +} + +static struct drm_property_blob * +create_resized_lut(struct drm_i915_private *i915, + const struct drm_property_blob *blob_in, int lut_out_size) +{ + int i, lut_in_size = drm_color_lut_size(blob_in); + struct drm_property_blob *blob_out; + const struct drm_color_lut *lut_in; + struct drm_color_lut *lut_out; + + blob_out = drm_property_create_blob(&i915->drm, + sizeof(lut_out[0]) * lut_out_size, + NULL); + if (IS_ERR(blob_out)) + return blob_out; + + lut_in = blob_in->data; + lut_out = blob_out->data; + + for (i = 0; i < lut_out_size; i++) + lut_out[i] = lut_in[i * (lut_in_size - 1) / (lut_out_size - 1)]; + + return blob_out; +} + static void i9xx_load_lut_8(struct intel_crtc *crtc, const struct drm_property_blob *blob) { @@ -580,12 +645,9 @@ static void i9xx_load_lut_8(struct intel_crtc *crtc, static void i9xx_load_luts(const struct intel_crtc_state *crtc_state) { struct intel_crtc *crtc = to_intel_crtc(crtc_state->uapi.crtc); - struct drm_i915_private *dev_priv = to_i915(crtc->base.dev); - const struct drm_property_blob *gamma_lut = crtc_state->hw.gamma_lut; + const struct drm_property_blob *post_csc_lut = crtc_state->post_csc_lut; - assert_pll_enabled(dev_priv, crtc->pipe); - - i9xx_load_lut_8(crtc, gamma_lut); + i9xx_load_lut_8(crtc, post_csc_lut); } static void i965_load_lut_10p6(struct intel_crtc *crtc, @@ -611,24 +673,18 @@ static void i965_load_lut_10p6(struct intel_crtc *crtc, static void i965_load_luts(const struct intel_crtc_state *crtc_state) { struct intel_crtc *crtc = to_intel_crtc(crtc_state->uapi.crtc); - struct drm_i915_private *dev_priv = to_i915(crtc->base.dev); - const struct drm_property_blob *gamma_lut = crtc_state->hw.gamma_lut; - - if (intel_crtc_has_type(crtc_state, INTEL_OUTPUT_DSI)) - assert_dsi_pll_enabled(dev_priv); - else - assert_pll_enabled(dev_priv, crtc->pipe); + const struct drm_property_blob *post_csc_lut = crtc_state->post_csc_lut; if (crtc_state->gamma_mode == GAMMA_MODE_MODE_8BIT) - i9xx_load_lut_8(crtc, gamma_lut); + i9xx_load_lut_8(crtc, post_csc_lut); else - i965_load_lut_10p6(crtc, gamma_lut); + i965_load_lut_10p6(crtc, post_csc_lut); } static void ilk_load_lut_8(struct intel_crtc *crtc, const struct drm_property_blob *blob) { - struct drm_i915_private *dev_priv = to_i915(crtc->base.dev); + struct drm_i915_private *i915 = to_i915(crtc->base.dev); const struct drm_color_lut *lut; enum pipe pipe = crtc->pipe; int i; @@ -639,34 +695,36 @@ static void ilk_load_lut_8(struct intel_crtc *crtc, lut = blob->data; for (i = 0; i < 256; i++) - intel_de_write_fw(dev_priv, LGC_PALETTE(pipe, i), + intel_de_write_fw(i915, LGC_PALETTE(pipe, i), i9xx_lut_8(&lut[i])); } static void ilk_load_lut_10(struct intel_crtc *crtc, const struct drm_property_blob *blob) { - struct drm_i915_private *dev_priv = to_i915(crtc->base.dev); + struct drm_i915_private *i915 = to_i915(crtc->base.dev); const struct drm_color_lut *lut = blob->data; int i, lut_size = drm_color_lut_size(blob); enum pipe pipe = crtc->pipe; for (i = 0; i < lut_size; i++) - intel_de_write_fw(dev_priv, PREC_PALETTE(pipe, i), + intel_de_write_fw(i915, PREC_PALETTE(pipe, i), ilk_lut_10(&lut[i])); } static void ilk_load_luts(const struct intel_crtc_state *crtc_state) { struct intel_crtc *crtc = to_intel_crtc(crtc_state->uapi.crtc); - const struct drm_property_blob *gamma_lut = crtc_state->hw.gamma_lut; + const struct drm_property_blob *post_csc_lut = crtc_state->post_csc_lut; + const struct drm_property_blob *pre_csc_lut = crtc_state->pre_csc_lut; + const struct drm_property_blob *blob = post_csc_lut ?: pre_csc_lut; switch (crtc_state->gamma_mode) { case GAMMA_MODE_MODE_8BIT: - ilk_load_lut_8(crtc, gamma_lut); + ilk_load_lut_8(crtc, blob); break; case GAMMA_MODE_MODE_10BIT: - ilk_load_lut_10(crtc, gamma_lut); + ilk_load_lut_10(crtc, blob); break; default: MISSING_CASE(crtc_state->gamma_mode); @@ -691,27 +749,22 @@ static void ivb_load_lut_10(struct intel_crtc *crtc, const struct drm_property_blob *blob, u32 prec_index) { - struct drm_i915_private *dev_priv = to_i915(crtc->base.dev); - int hw_lut_size = ivb_lut_10_size(prec_index); + struct drm_i915_private *i915 = to_i915(crtc->base.dev); const struct drm_color_lut *lut = blob->data; int i, lut_size = drm_color_lut_size(blob); enum pipe pipe = crtc->pipe; - for (i = 0; i < hw_lut_size; i++) { - /* We discard half the user entries in split gamma mode */ - const struct drm_color_lut *entry = - &lut[i * (lut_size - 1) / (hw_lut_size - 1)]; - - intel_de_write_fw(dev_priv, PREC_PAL_INDEX(pipe), prec_index++); - intel_de_write_fw(dev_priv, PREC_PAL_DATA(pipe), - ilk_lut_10(entry)); + for (i = 0; i < lut_size; i++) { + intel_de_write_fw(i915, PREC_PAL_INDEX(pipe), prec_index++); + intel_de_write_fw(i915, PREC_PAL_DATA(pipe), + ilk_lut_10(&lut[i])); } /* * Reset the index, otherwise it prevents the legacy palette to be * written properly. */ - intel_de_write_fw(dev_priv, PREC_PAL_INDEX(pipe), 0); + intel_de_write_fw(i915, PREC_PAL_INDEX(pipe), 0); } /* On BDW+ the index auto increment mode actually works */ @@ -719,73 +772,63 @@ static void bdw_load_lut_10(struct intel_crtc *crtc, const struct drm_property_blob *blob, u32 prec_index) { - struct drm_i915_private *dev_priv = to_i915(crtc->base.dev); - int hw_lut_size = ivb_lut_10_size(prec_index); + struct drm_i915_private *i915 = to_i915(crtc->base.dev); const struct drm_color_lut *lut = blob->data; int i, lut_size = drm_color_lut_size(blob); enum pipe pipe = crtc->pipe; - intel_de_write_fw(dev_priv, PREC_PAL_INDEX(pipe), + intel_de_write_fw(i915, PREC_PAL_INDEX(pipe), prec_index | PAL_PREC_AUTO_INCREMENT); - for (i = 0; i < hw_lut_size; i++) { - /* We discard half the user entries in split gamma mode */ - const struct drm_color_lut *entry = - &lut[i * (lut_size - 1) / (hw_lut_size - 1)]; - - intel_de_write_fw(dev_priv, PREC_PAL_DATA(pipe), - ilk_lut_10(entry)); - } + for (i = 0; i < lut_size; i++) + intel_de_write_fw(i915, PREC_PAL_DATA(pipe), + ilk_lut_10(&lut[i])); /* * Reset the index, otherwise it prevents the legacy palette to be * written properly. */ - intel_de_write_fw(dev_priv, PREC_PAL_INDEX(pipe), 0); + intel_de_write_fw(i915, PREC_PAL_INDEX(pipe), 0); } static void ivb_load_lut_ext_max(const struct intel_crtc_state *crtc_state) { struct intel_crtc *crtc = to_intel_crtc(crtc_state->uapi.crtc); - struct drm_i915_private *dev_priv = to_i915(crtc->base.dev); enum pipe pipe = crtc->pipe; /* Program the max register to clamp values > 1.0. */ intel_dsb_reg_write(crtc_state, PREC_PAL_EXT_GC_MAX(pipe, 0), 1 << 16); intel_dsb_reg_write(crtc_state, PREC_PAL_EXT_GC_MAX(pipe, 1), 1 << 16); intel_dsb_reg_write(crtc_state, PREC_PAL_EXT_GC_MAX(pipe, 2), 1 << 16); +} - /* - * Program the gc max 2 register to clamp values > 1.0. - * ToDo: Extend the ABI to be able to program values - * from 3.0 to 7.0 - */ - if (DISPLAY_VER(dev_priv) >= 10) { - intel_dsb_reg_write(crtc_state, PREC_PAL_EXT2_GC_MAX(pipe, 0), - 1 << 16); - intel_dsb_reg_write(crtc_state, PREC_PAL_EXT2_GC_MAX(pipe, 1), - 1 << 16); - intel_dsb_reg_write(crtc_state, PREC_PAL_EXT2_GC_MAX(pipe, 2), - 1 << 16); - } +static void glk_load_lut_ext2_max(const struct intel_crtc_state *crtc_state) +{ + struct intel_crtc *crtc = to_intel_crtc(crtc_state->uapi.crtc); + enum pipe pipe = crtc->pipe; + + /* Program the max register to clamp values > 1.0. */ + intel_dsb_reg_write(crtc_state, PREC_PAL_EXT2_GC_MAX(pipe, 0), 1 << 16); + intel_dsb_reg_write(crtc_state, PREC_PAL_EXT2_GC_MAX(pipe, 1), 1 << 16); + intel_dsb_reg_write(crtc_state, PREC_PAL_EXT2_GC_MAX(pipe, 2), 1 << 16); } static void ivb_load_luts(const struct intel_crtc_state *crtc_state) { struct intel_crtc *crtc = to_intel_crtc(crtc_state->uapi.crtc); - const struct drm_property_blob *gamma_lut = crtc_state->hw.gamma_lut; - const struct drm_property_blob *degamma_lut = crtc_state->hw.degamma_lut; - const struct drm_property_blob *blob = gamma_lut ?: degamma_lut; + const struct drm_property_blob *post_csc_lut = crtc_state->post_csc_lut; + const struct drm_property_blob *pre_csc_lut = crtc_state->pre_csc_lut; + const struct drm_property_blob *blob = post_csc_lut ?: pre_csc_lut; switch (crtc_state->gamma_mode) { case GAMMA_MODE_MODE_8BIT: ilk_load_lut_8(crtc, blob); break; case GAMMA_MODE_MODE_SPLIT: - ivb_load_lut_10(crtc, degamma_lut, PAL_PREC_SPLIT_MODE | + ivb_load_lut_10(crtc, pre_csc_lut, PAL_PREC_SPLIT_MODE | PAL_PREC_INDEX_VALUE(0)); ivb_load_lut_ext_max(crtc_state); - ivb_load_lut_10(crtc, gamma_lut, PAL_PREC_SPLIT_MODE | + ivb_load_lut_10(crtc, post_csc_lut, PAL_PREC_SPLIT_MODE | PAL_PREC_INDEX_VALUE(512)); break; case GAMMA_MODE_MODE_10BIT: @@ -802,19 +845,19 @@ static void ivb_load_luts(const struct intel_crtc_state *crtc_state) static void bdw_load_luts(const struct intel_crtc_state *crtc_state) { struct intel_crtc *crtc = to_intel_crtc(crtc_state->uapi.crtc); - const struct drm_property_blob *gamma_lut = crtc_state->hw.gamma_lut; - const struct drm_property_blob *degamma_lut = crtc_state->hw.degamma_lut; - const struct drm_property_blob *blob = gamma_lut ?: degamma_lut; + const struct drm_property_blob *post_csc_lut = crtc_state->post_csc_lut; + const struct drm_property_blob *pre_csc_lut = crtc_state->pre_csc_lut; + const struct drm_property_blob *blob = post_csc_lut ?: pre_csc_lut; switch (crtc_state->gamma_mode) { case GAMMA_MODE_MODE_8BIT: ilk_load_lut_8(crtc, blob); break; case GAMMA_MODE_MODE_SPLIT: - bdw_load_lut_10(crtc, degamma_lut, PAL_PREC_SPLIT_MODE | + bdw_load_lut_10(crtc, pre_csc_lut, PAL_PREC_SPLIT_MODE | PAL_PREC_INDEX_VALUE(0)); ivb_load_lut_ext_max(crtc_state); - bdw_load_lut_10(crtc, gamma_lut, PAL_PREC_SPLIT_MODE | + bdw_load_lut_10(crtc, post_csc_lut, PAL_PREC_SPLIT_MODE | PAL_PREC_INDEX_VALUE(512)); break; case GAMMA_MODE_MODE_10BIT: @@ -837,21 +880,22 @@ static int glk_degamma_lut_size(struct drm_i915_private *i915) return 35; } -static void glk_load_degamma_lut(const struct intel_crtc_state *crtc_state) +static void glk_load_degamma_lut(const struct intel_crtc_state *crtc_state, + const struct drm_property_blob *blob) { struct intel_crtc *crtc = to_intel_crtc(crtc_state->uapi.crtc); - struct drm_i915_private *dev_priv = to_i915(crtc->base.dev); + struct drm_i915_private *i915 = to_i915(crtc->base.dev); + const struct drm_color_lut *lut = blob->data; + int i, lut_size = drm_color_lut_size(blob); enum pipe pipe = crtc->pipe; - int i, lut_size = INTEL_INFO(dev_priv)->display.color.degamma_lut_size; - const struct drm_color_lut *lut = crtc_state->hw.degamma_lut->data; /* * When setting the auto-increment bit, the hardware seems to * ignore the index bits, so we need to reset it to index 0 * separately. */ - intel_de_write_fw(dev_priv, PRE_CSC_GAMC_INDEX(pipe), 0); - intel_de_write_fw(dev_priv, PRE_CSC_GAMC_INDEX(pipe), + intel_de_write_fw(i915, PRE_CSC_GAMC_INDEX(pipe), 0); + intel_de_write_fw(i915, PRE_CSC_GAMC_INDEX(pipe), PRE_CSC_GAMC_AUTO_INCREMENT); for (i = 0; i < lut_size; i++) { @@ -868,71 +912,34 @@ static void glk_load_degamma_lut(const struct intel_crtc_state *crtc_state) * ToDo: Extend to max 7.0. Enable 32 bit input value * as compared to just 16 to achieve this. */ - intel_de_write_fw(dev_priv, PRE_CSC_GAMC_DATA(pipe), + intel_de_write_fw(i915, PRE_CSC_GAMC_DATA(pipe), lut[i].green); } /* Clamp values > 1.0. */ - while (i++ < glk_degamma_lut_size(dev_priv)) - intel_de_write_fw(dev_priv, PRE_CSC_GAMC_DATA(pipe), 1 << 16); + while (i++ < glk_degamma_lut_size(i915)) + intel_de_write_fw(i915, PRE_CSC_GAMC_DATA(pipe), 1 << 16); - intel_de_write_fw(dev_priv, PRE_CSC_GAMC_INDEX(pipe), 0); -} - -static void glk_load_degamma_lut_linear(const struct intel_crtc_state *crtc_state) -{ - struct intel_crtc *crtc = to_intel_crtc(crtc_state->uapi.crtc); - struct drm_i915_private *dev_priv = to_i915(crtc->base.dev); - enum pipe pipe = crtc->pipe; - int i, lut_size = INTEL_INFO(dev_priv)->display.color.degamma_lut_size; - - /* - * When setting the auto-increment bit, the hardware seems to - * ignore the index bits, so we need to reset it to index 0 - * separately. - */ - intel_de_write_fw(dev_priv, PRE_CSC_GAMC_INDEX(pipe), 0); - intel_de_write_fw(dev_priv, PRE_CSC_GAMC_INDEX(pipe), - PRE_CSC_GAMC_AUTO_INCREMENT); - - for (i = 0; i < lut_size; i++) { - u32 v = (i << 16) / (lut_size - 1); - - intel_de_write_fw(dev_priv, PRE_CSC_GAMC_DATA(pipe), v); - } - - /* Clamp values > 1.0. */ - while (i++ < 35) - intel_de_write_fw(dev_priv, PRE_CSC_GAMC_DATA(pipe), 1 << 16); - - intel_de_write_fw(dev_priv, PRE_CSC_GAMC_INDEX(pipe), 0); + intel_de_write_fw(i915, PRE_CSC_GAMC_INDEX(pipe), 0); } static void glk_load_luts(const struct intel_crtc_state *crtc_state) { - const struct drm_property_blob *gamma_lut = crtc_state->hw.gamma_lut; + const struct drm_property_blob *pre_csc_lut = crtc_state->pre_csc_lut; + const struct drm_property_blob *post_csc_lut = crtc_state->post_csc_lut; struct intel_crtc *crtc = to_intel_crtc(crtc_state->uapi.crtc); - /* - * On GLK+ both pipe CSC and degamma LUT are controlled - * by csc_enable. Hence for the cases where the CSC is - * needed but degamma LUT is not we need to load a - * linear degamma LUT. In fact we'll just always load - * the degama LUT so that we don't have to reload - * it every time the pipe CSC is being enabled. - */ - if (crtc_state->hw.degamma_lut) - glk_load_degamma_lut(crtc_state); - else - glk_load_degamma_lut_linear(crtc_state); + if (pre_csc_lut) + glk_load_degamma_lut(crtc_state, pre_csc_lut); switch (crtc_state->gamma_mode) { case GAMMA_MODE_MODE_8BIT: - ilk_load_lut_8(crtc, gamma_lut); + ilk_load_lut_8(crtc, post_csc_lut); break; case GAMMA_MODE_MODE_10BIT: - bdw_load_lut_10(crtc, gamma_lut, PAL_PREC_INDEX_VALUE(0)); + bdw_load_lut_10(crtc, post_csc_lut, PAL_PREC_INDEX_VALUE(0)); ivb_load_lut_ext_max(crtc_state); + glk_load_lut_ext2_max(crtc_state); break; default: MISSING_CASE(crtc_state->gamma_mode); @@ -940,23 +947,9 @@ static void glk_load_luts(const struct intel_crtc_state *crtc_state) } } -/* ilk+ "12.4" interpolated format (high 10 bits) */ -static u32 ilk_lut_12p4_udw(const struct drm_color_lut *color) -{ - return (color->red >> 6) << 20 | (color->green >> 6) << 10 | - (color->blue >> 6); -} - -/* ilk+ "12.4" interpolated format (low 6 bits) */ -static u32 ilk_lut_12p4_ldw(const struct drm_color_lut *color) -{ - return (color->red & 0x3f) << 24 | (color->green & 0x3f) << 14 | - (color->blue & 0x3f) << 4; -} - static void -icl_load_gcmax(const struct intel_crtc_state *crtc_state, - const struct drm_color_lut *color) +ivb_load_lut_max(const struct intel_crtc_state *crtc_state, + const struct drm_color_lut *color) { struct intel_crtc *crtc = to_intel_crtc(crtc_state->uapi.crtc); enum pipe pipe = crtc->pipe; @@ -971,7 +964,7 @@ static void icl_program_gamma_superfine_segment(const struct intel_crtc_state *crtc_state) { struct intel_crtc *crtc = to_intel_crtc(crtc_state->uapi.crtc); - const struct drm_property_blob *blob = crtc_state->hw.gamma_lut; + const struct drm_property_blob *blob = crtc_state->post_csc_lut; const struct drm_color_lut *lut = blob->data; enum pipe pipe = crtc->pipe; int i; @@ -1000,7 +993,7 @@ static void icl_program_gamma_multi_segment(const struct intel_crtc_state *crtc_state) { struct intel_crtc *crtc = to_intel_crtc(crtc_state->uapi.crtc); - const struct drm_property_blob *blob = crtc_state->hw.gamma_lut; + const struct drm_property_blob *blob = crtc_state->post_csc_lut; const struct drm_color_lut *lut = blob->data; const struct drm_color_lut *entry; enum pipe pipe = crtc->pipe; @@ -1048,29 +1041,32 @@ icl_program_gamma_multi_segment(const struct intel_crtc_state *crtc_state) /* The last entry in the LUT is to be programmed in GCMAX */ entry = &lut[256 * 8 * 128]; - icl_load_gcmax(crtc_state, entry); - ivb_load_lut_ext_max(crtc_state); + ivb_load_lut_max(crtc_state, entry); } static void icl_load_luts(const struct intel_crtc_state *crtc_state) { - const struct drm_property_blob *gamma_lut = crtc_state->hw.gamma_lut; + const struct drm_property_blob *pre_csc_lut = crtc_state->pre_csc_lut; + const struct drm_property_blob *post_csc_lut = crtc_state->post_csc_lut; struct intel_crtc *crtc = to_intel_crtc(crtc_state->uapi.crtc); - if (crtc_state->hw.degamma_lut) - glk_load_degamma_lut(crtc_state); + if (pre_csc_lut) + glk_load_degamma_lut(crtc_state, pre_csc_lut); switch (crtc_state->gamma_mode & GAMMA_MODE_MODE_MASK) { case GAMMA_MODE_MODE_8BIT: - ilk_load_lut_8(crtc, gamma_lut); + ilk_load_lut_8(crtc, post_csc_lut); break; case GAMMA_MODE_MODE_12BIT_MULTI_SEGMENTED: icl_program_gamma_superfine_segment(crtc_state); icl_program_gamma_multi_segment(crtc_state); + ivb_load_lut_ext_max(crtc_state); + glk_load_lut_ext2_max(crtc_state); break; case GAMMA_MODE_MODE_10BIT: - bdw_load_lut_10(crtc, gamma_lut, PAL_PREC_INDEX_VALUE(0)); + bdw_load_lut_10(crtc, post_csc_lut, PAL_PREC_INDEX_VALUE(0)); ivb_load_lut_ext_max(crtc_state); + glk_load_lut_ext2_max(crtc_state); break; default: MISSING_CASE(crtc_state->gamma_mode); @@ -1082,61 +1078,61 @@ static void icl_load_luts(const struct intel_crtc_state *crtc_state) static u32 chv_cgm_degamma_ldw(const struct drm_color_lut *color) { - return drm_color_lut_extract(color->green, 14) << 16 | - drm_color_lut_extract(color->blue, 14); + return REG_FIELD_PREP(CGM_PIPE_DEGAMMA_GREEN_LDW_MASK, drm_color_lut_extract(color->green, 14)) | + REG_FIELD_PREP(CGM_PIPE_DEGAMMA_BLUE_LDW_MASK, drm_color_lut_extract(color->blue, 14)); } static u32 chv_cgm_degamma_udw(const struct drm_color_lut *color) { - return drm_color_lut_extract(color->red, 14); + return REG_FIELD_PREP(CGM_PIPE_DEGAMMA_RED_UDW_MASK, drm_color_lut_extract(color->red, 14)); } static void chv_load_cgm_degamma(struct intel_crtc *crtc, const struct drm_property_blob *blob) { - struct drm_i915_private *dev_priv = to_i915(crtc->base.dev); + struct drm_i915_private *i915 = to_i915(crtc->base.dev); const struct drm_color_lut *lut = blob->data; int i, lut_size = drm_color_lut_size(blob); enum pipe pipe = crtc->pipe; for (i = 0; i < lut_size; i++) { - intel_de_write_fw(dev_priv, CGM_PIPE_DEGAMMA(pipe, i, 0), + intel_de_write_fw(i915, CGM_PIPE_DEGAMMA(pipe, i, 0), chv_cgm_degamma_ldw(&lut[i])); - intel_de_write_fw(dev_priv, CGM_PIPE_DEGAMMA(pipe, i, 1), + intel_de_write_fw(i915, CGM_PIPE_DEGAMMA(pipe, i, 1), chv_cgm_degamma_udw(&lut[i])); } } static u32 chv_cgm_gamma_ldw(const struct drm_color_lut *color) { - return drm_color_lut_extract(color->green, 10) << 16 | - drm_color_lut_extract(color->blue, 10); + return REG_FIELD_PREP(CGM_PIPE_GAMMA_GREEN_LDW_MASK, drm_color_lut_extract(color->green, 10)) | + REG_FIELD_PREP(CGM_PIPE_GAMMA_BLUE_LDW_MASK, drm_color_lut_extract(color->blue, 10)); } static u32 chv_cgm_gamma_udw(const struct drm_color_lut *color) { - return drm_color_lut_extract(color->red, 10); + return REG_FIELD_PREP(CGM_PIPE_GAMMA_RED_UDW_MASK, drm_color_lut_extract(color->red, 10)); } static void chv_cgm_gamma_pack(struct drm_color_lut *entry, u32 ldw, u32 udw) { - entry->green = intel_color_lut_pack(REG_FIELD_GET(CGM_PIPE_GAMMA_GREEN_MASK, ldw), 10); - entry->blue = intel_color_lut_pack(REG_FIELD_GET(CGM_PIPE_GAMMA_BLUE_MASK, ldw), 10); - entry->red = intel_color_lut_pack(REG_FIELD_GET(CGM_PIPE_GAMMA_RED_MASK, udw), 10); + entry->green = intel_color_lut_pack(REG_FIELD_GET(CGM_PIPE_GAMMA_GREEN_LDW_MASK, ldw), 10); + entry->blue = intel_color_lut_pack(REG_FIELD_GET(CGM_PIPE_GAMMA_BLUE_LDW_MASK, ldw), 10); + entry->red = intel_color_lut_pack(REG_FIELD_GET(CGM_PIPE_GAMMA_RED_UDW_MASK, udw), 10); } static void chv_load_cgm_gamma(struct intel_crtc *crtc, const struct drm_property_blob *blob) { - struct drm_i915_private *dev_priv = to_i915(crtc->base.dev); + struct drm_i915_private *i915 = to_i915(crtc->base.dev); const struct drm_color_lut *lut = blob->data; int i, lut_size = drm_color_lut_size(blob); enum pipe pipe = crtc->pipe; for (i = 0; i < lut_size; i++) { - intel_de_write_fw(dev_priv, CGM_PIPE_GAMMA(pipe, i, 0), + intel_de_write_fw(i915, CGM_PIPE_GAMMA(pipe, i, 0), chv_cgm_gamma_ldw(&lut[i])); - intel_de_write_fw(dev_priv, CGM_PIPE_GAMMA(pipe, i, 1), + intel_de_write_fw(i915, CGM_PIPE_GAMMA(pipe, i, 1), chv_cgm_gamma_udw(&lut[i])); } } @@ -1144,46 +1140,46 @@ static void chv_load_cgm_gamma(struct intel_crtc *crtc, static void chv_load_luts(const struct intel_crtc_state *crtc_state) { struct intel_crtc *crtc = to_intel_crtc(crtc_state->uapi.crtc); - struct drm_i915_private *dev_priv = to_i915(crtc->base.dev); - const struct drm_property_blob *degamma_lut = crtc_state->hw.degamma_lut; - const struct drm_property_blob *gamma_lut = crtc_state->hw.gamma_lut; + struct drm_i915_private *i915 = to_i915(crtc->base.dev); + const struct drm_property_blob *pre_csc_lut = crtc_state->pre_csc_lut; + const struct drm_property_blob *post_csc_lut = crtc_state->post_csc_lut; const struct drm_property_blob *ctm = crtc_state->hw.ctm; if (crtc_state->cgm_mode & CGM_PIPE_MODE_CSC) chv_load_cgm_csc(crtc, ctm); if (crtc_state->cgm_mode & CGM_PIPE_MODE_DEGAMMA) - chv_load_cgm_degamma(crtc, degamma_lut); + chv_load_cgm_degamma(crtc, pre_csc_lut); if (crtc_state->cgm_mode & CGM_PIPE_MODE_GAMMA) - chv_load_cgm_gamma(crtc, gamma_lut); + chv_load_cgm_gamma(crtc, post_csc_lut); else i965_load_luts(crtc_state); - intel_de_write_fw(dev_priv, CGM_PIPE_MODE(crtc->pipe), + intel_de_write_fw(i915, CGM_PIPE_MODE(crtc->pipe), crtc_state->cgm_mode); } void intel_color_load_luts(const struct intel_crtc_state *crtc_state) { - struct drm_i915_private *dev_priv = to_i915(crtc_state->uapi.crtc->dev); + struct drm_i915_private *i915 = to_i915(crtc_state->uapi.crtc->dev); - dev_priv->display.funcs.color->load_luts(crtc_state); + i915->display.funcs.color->load_luts(crtc_state); } void intel_color_commit_noarm(const struct intel_crtc_state *crtc_state) { - struct drm_i915_private *dev_priv = to_i915(crtc_state->uapi.crtc->dev); + struct drm_i915_private *i915 = to_i915(crtc_state->uapi.crtc->dev); - if (dev_priv->display.funcs.color->color_commit_noarm) - dev_priv->display.funcs.color->color_commit_noarm(crtc_state); + if (i915->display.funcs.color->color_commit_noarm) + i915->display.funcs.color->color_commit_noarm(crtc_state); } void intel_color_commit_arm(const struct intel_crtc_state *crtc_state) { - struct drm_i915_private *dev_priv = to_i915(crtc_state->uapi.crtc->dev); + struct drm_i915_private *i915 = to_i915(crtc_state->uapi.crtc->dev); - dev_priv->display.funcs.color->color_commit_arm(crtc_state); + i915->display.funcs.color->color_commit_arm(crtc_state); } static bool intel_can_preload_luts(const struct intel_crtc_state *new_crtc_state) @@ -1194,8 +1190,8 @@ static bool intel_can_preload_luts(const struct intel_crtc_state *new_crtc_state const struct intel_crtc_state *old_crtc_state = intel_atomic_get_old_crtc_state(state, crtc); - return !old_crtc_state->hw.gamma_lut && - !old_crtc_state->hw.degamma_lut; + return !old_crtc_state->post_csc_lut && + !old_crtc_state->pre_csc_lut; } static bool chv_can_preload_luts(const struct intel_crtc_state *new_crtc_state) @@ -1214,46 +1210,28 @@ static bool chv_can_preload_luts(const struct intel_crtc_state *new_crtc_state) if (old_crtc_state->cgm_mode || new_crtc_state->cgm_mode) return false; - return !old_crtc_state->hw.gamma_lut; -} - -static bool glk_can_preload_luts(const struct intel_crtc_state *new_crtc_state) -{ - struct intel_crtc *crtc = to_intel_crtc(new_crtc_state->uapi.crtc); - struct intel_atomic_state *state = - to_intel_atomic_state(new_crtc_state->uapi.state); - const struct intel_crtc_state *old_crtc_state = - intel_atomic_get_old_crtc_state(state, crtc); - - /* - * The hardware degamma is active whenever the pipe - * CSC is active. Thus even if the old state has no - * software degamma we need to avoid clobbering the - * linear hardware degamma mid scanout. - */ - return !old_crtc_state->csc_enable && - !old_crtc_state->hw.gamma_lut; + return !old_crtc_state->post_csc_lut; } int intel_color_check(struct intel_crtc_state *crtc_state) { - struct drm_i915_private *dev_priv = to_i915(crtc_state->uapi.crtc->dev); + struct drm_i915_private *i915 = to_i915(crtc_state->uapi.crtc->dev); - return dev_priv->display.funcs.color->color_check(crtc_state); + return i915->display.funcs.color->color_check(crtc_state); } void intel_color_get_config(struct intel_crtc_state *crtc_state) { - struct drm_i915_private *dev_priv = to_i915(crtc_state->uapi.crtc->dev); + struct drm_i915_private *i915 = to_i915(crtc_state->uapi.crtc->dev); - if (dev_priv->display.funcs.color->read_luts) - dev_priv->display.funcs.color->read_luts(crtc_state); + if (i915->display.funcs.color->read_luts) + i915->display.funcs.color->read_luts(crtc_state); } static bool need_plane_update(struct intel_plane *plane, const struct intel_crtc_state *crtc_state) { - struct drm_i915_private *dev_priv = to_i915(plane->base.dev); + struct drm_i915_private *i915 = to_i915(plane->base.dev); /* * On pre-SKL the pipe gamma enable and pipe csc enable for @@ -1261,7 +1239,7 @@ static bool need_plane_update(struct intel_plane *plane, * We have to reconfigure that even if the plane is inactive. */ return crtc_state->active_planes & BIT(plane->id) || - (DISPLAY_VER(dev_priv) < 9 && + (DISPLAY_VER(i915) < 9 && plane->id == PLANE_PRIMARY); } @@ -1269,7 +1247,7 @@ static int intel_color_add_affected_planes(struct intel_crtc_state *new_crtc_state) { struct intel_crtc *crtc = to_intel_crtc(new_crtc_state->uapi.crtc); - struct drm_i915_private *dev_priv = to_i915(crtc->base.dev); + struct drm_i915_private *i915 = to_i915(crtc->base.dev); struct intel_atomic_state *state = to_intel_atomic_state(new_crtc_state->uapi.state); const struct intel_crtc_state *old_crtc_state = @@ -1277,14 +1255,14 @@ intel_color_add_affected_planes(struct intel_crtc_state *new_crtc_state) struct intel_plane *plane; if (!new_crtc_state->hw.active || - drm_atomic_crtc_needs_modeset(&new_crtc_state->uapi)) + intel_crtc_needs_modeset(new_crtc_state)) return 0; if (new_crtc_state->gamma_enable == old_crtc_state->gamma_enable && new_crtc_state->csc_enable == old_crtc_state->csc_enable) return 0; - for_each_intel_plane_on_crtc(&dev_priv->drm, crtc, plane) { + for_each_intel_plane_on_crtc(&i915->drm, crtc, plane) { struct intel_plane_state *plane_state; if (!need_plane_update(plane, new_crtc_state)) @@ -1295,6 +1273,10 @@ intel_color_add_affected_planes(struct intel_crtc_state *new_crtc_state) return PTR_ERR(plane_state); new_crtc_state->update_planes |= BIT(plane->id); + + /* plane control register changes blocked by CxSR */ + if (HAS_GMCH(i915)) + new_crtc_state->disable_cxsr = true; } return 0; @@ -1319,7 +1301,7 @@ static int check_lut_size(const struct drm_property_blob *lut, int expected) static int check_luts(const struct intel_crtc_state *crtc_state) { - struct drm_i915_private *dev_priv = to_i915(crtc_state->uapi.crtc->dev); + struct drm_i915_private *i915 = to_i915(crtc_state->uapi.crtc->dev); const struct drm_property_blob *gamma_lut = crtc_state->hw.gamma_lut; const struct drm_property_blob *degamma_lut = crtc_state->hw.degamma_lut; int gamma_length, degamma_length; @@ -1331,15 +1313,15 @@ static int check_luts(const struct intel_crtc_state *crtc_state) /* C8 relies on its palette being stored in the legacy LUT */ if (crtc_state->c8_planes) { - drm_dbg_kms(&dev_priv->drm, + drm_dbg_kms(&i915->drm, "C8 pixelformat requires the legacy LUT\n"); return -EINVAL; } - degamma_length = INTEL_INFO(dev_priv)->display.color.degamma_lut_size; - gamma_length = INTEL_INFO(dev_priv)->display.color.gamma_lut_size; - degamma_tests = INTEL_INFO(dev_priv)->display.color.degamma_lut_tests; - gamma_tests = INTEL_INFO(dev_priv)->display.color.gamma_lut_tests; + degamma_length = INTEL_INFO(i915)->display.color.degamma_lut_size; + gamma_length = INTEL_INFO(i915)->display.color.gamma_lut_size; + degamma_tests = INTEL_INFO(i915)->display.color.degamma_lut_tests; + gamma_tests = INTEL_INFO(i915)->display.color.gamma_lut_tests; if (check_lut_size(degamma_lut, degamma_length) || check_lut_size(gamma_lut, gamma_length)) @@ -1361,6 +1343,40 @@ static u32 i9xx_gamma_mode(struct intel_crtc_state *crtc_state) return GAMMA_MODE_MODE_10BIT; /* i965+ only */ } +void intel_color_assert_luts(const struct intel_crtc_state *crtc_state) +{ + struct drm_i915_private *i915 = to_i915(crtc_state->uapi.crtc->dev); + + /* make sure {pre,post}_csc_lut were correctly assigned */ + if (DISPLAY_VER(i915) >= 11 || HAS_GMCH(i915)) { + drm_WARN_ON(&i915->drm, + crtc_state->pre_csc_lut != crtc_state->hw.degamma_lut); + drm_WARN_ON(&i915->drm, + crtc_state->post_csc_lut != crtc_state->hw.gamma_lut); + } else if (DISPLAY_VER(i915) == 10) { + drm_WARN_ON(&i915->drm, + crtc_state->pre_csc_lut != crtc_state->hw.degamma_lut && + crtc_state->pre_csc_lut != i915->display.color.glk_linear_degamma_lut); + drm_WARN_ON(&i915->drm, + crtc_state->post_csc_lut != crtc_state->hw.gamma_lut); + } else if (crtc_state->gamma_mode != GAMMA_MODE_MODE_SPLIT) { + drm_WARN_ON(&i915->drm, + crtc_state->pre_csc_lut != crtc_state->hw.degamma_lut && + crtc_state->pre_csc_lut != crtc_state->hw.gamma_lut); + drm_WARN_ON(&i915->drm, + crtc_state->post_csc_lut != crtc_state->hw.degamma_lut && + crtc_state->post_csc_lut != crtc_state->hw.gamma_lut); + } +} + +static void intel_assign_luts(struct intel_crtc_state *crtc_state) +{ + drm_property_replace_blob(&crtc_state->pre_csc_lut, + crtc_state->hw.degamma_lut); + drm_property_replace_blob(&crtc_state->post_csc_lut, + crtc_state->hw.gamma_lut); +} + static int i9xx_color_check(struct intel_crtc_state *crtc_state) { int ret; @@ -1379,6 +1395,8 @@ static int i9xx_color_check(struct intel_crtc_state *crtc_state) if (ret) return ret; + intel_assign_luts(crtc_state); + crtc_state->preload_luts = intel_can_preload_luts(crtc_state); return 0; @@ -1433,11 +1451,27 @@ static int chv_color_check(struct intel_crtc_state *crtc_state) if (ret) return ret; + intel_assign_luts(crtc_state); + crtc_state->preload_luts = chv_can_preload_luts(crtc_state); return 0; } +static bool ilk_gamma_enable(const struct intel_crtc_state *crtc_state) +{ + return (crtc_state->hw.gamma_lut || + crtc_state->hw.degamma_lut) && + !crtc_state->c8_planes; +} + +static bool ilk_csc_enable(const struct intel_crtc_state *crtc_state) +{ + return crtc_state->output_format != INTEL_OUTPUT_FORMAT_RGB || + ilk_csc_limited_range(crtc_state) || + crtc_state->hw.ctm; +} + static u32 ilk_gamma_mode(const struct intel_crtc_state *crtc_state) { if (!crtc_state->gamma_enable || @@ -1458,28 +1492,54 @@ static u32 ilk_csc_mode(const struct intel_crtc_state *crtc_state) if (crtc_state->output_format != INTEL_OUTPUT_FORMAT_RGB) return CSC_BLACK_SCREEN_OFFSET; + if (crtc_state->hw.degamma_lut) + return CSC_MODE_YUV_TO_RGB; + return CSC_MODE_YUV_TO_RGB | CSC_POSITION_BEFORE_GAMMA; } +static void ilk_assign_luts(struct intel_crtc_state *crtc_state) +{ + if (crtc_state->hw.degamma_lut || + crtc_state->csc_mode & CSC_POSITION_BEFORE_GAMMA) { + drm_property_replace_blob(&crtc_state->pre_csc_lut, + crtc_state->hw.degamma_lut); + drm_property_replace_blob(&crtc_state->post_csc_lut, + crtc_state->hw.gamma_lut); + } else { + drm_property_replace_blob(&crtc_state->pre_csc_lut, + crtc_state->hw.gamma_lut); + drm_property_replace_blob(&crtc_state->post_csc_lut, + NULL); + } +} + static int ilk_color_check(struct intel_crtc_state *crtc_state) { + struct drm_i915_private *i915 = to_i915(crtc_state->uapi.crtc->dev); int ret; ret = check_luts(crtc_state); if (ret) return ret; - crtc_state->gamma_enable = - crtc_state->hw.gamma_lut && - !crtc_state->c8_planes; + if (crtc_state->hw.degamma_lut && crtc_state->hw.gamma_lut) { + drm_dbg_kms(&i915->drm, + "Degamma and gamma together are not possible\n"); + return -EINVAL; + } - /* - * We don't expose the ctm on ilk/snb currently, also RGB - * limited range output is handled by the hw automagically. - */ - crtc_state->csc_enable = - crtc_state->output_format != INTEL_OUTPUT_FORMAT_RGB; + if (crtc_state->output_format != INTEL_OUTPUT_FORMAT_RGB && + crtc_state->hw.ctm) { + drm_dbg_kms(&i915->drm, + "YCbCr and CTM together are not possible\n"); + return -EINVAL; + } + + crtc_state->gamma_enable = ilk_gamma_enable(crtc_state); + + crtc_state->csc_enable = ilk_csc_enable(crtc_state); crtc_state->gamma_mode = ilk_gamma_mode(crtc_state); @@ -1489,6 +1549,8 @@ static int ilk_color_check(struct intel_crtc_state *crtc_state) if (ret) return ret; + ilk_assign_luts(crtc_state); + crtc_state->preload_luts = intel_can_preload_luts(crtc_state); return 0; @@ -1496,14 +1558,10 @@ static int ilk_color_check(struct intel_crtc_state *crtc_state) static u32 ivb_gamma_mode(const struct intel_crtc_state *crtc_state) { - if (!crtc_state->gamma_enable || - crtc_state_is_legacy_gamma(crtc_state)) - return GAMMA_MODE_MODE_8BIT; - else if (crtc_state->hw.gamma_lut && - crtc_state->hw.degamma_lut) + if (crtc_state->hw.degamma_lut && crtc_state->hw.gamma_lut) return GAMMA_MODE_MODE_SPLIT; - else - return GAMMA_MODE_MODE_10BIT; + + return ilk_gamma_mode(crtc_state); } static u32 ivb_csc_mode(const struct intel_crtc_state *crtc_state) @@ -1522,10 +1580,41 @@ static u32 ivb_csc_mode(const struct intel_crtc_state *crtc_state) return CSC_POSITION_BEFORE_GAMMA; } +static int ivb_assign_luts(struct intel_crtc_state *crtc_state) +{ + struct drm_i915_private *i915 = to_i915(crtc_state->uapi.crtc->dev); + struct drm_property_blob *degamma_lut, *gamma_lut; + + if (crtc_state->gamma_mode != GAMMA_MODE_MODE_SPLIT) { + ilk_assign_luts(crtc_state); + return 0; + } + + drm_WARN_ON(&i915->drm, drm_color_lut_size(crtc_state->hw.degamma_lut) != 1024); + drm_WARN_ON(&i915->drm, drm_color_lut_size(crtc_state->hw.gamma_lut) != 1024); + + degamma_lut = create_resized_lut(i915, crtc_state->hw.degamma_lut, 512); + if (IS_ERR(degamma_lut)) + return PTR_ERR(degamma_lut); + + gamma_lut = create_resized_lut(i915, crtc_state->hw.gamma_lut, 512); + if (IS_ERR(gamma_lut)) { + drm_property_blob_put(degamma_lut); + return PTR_ERR(gamma_lut); + } + + drm_property_replace_blob(&crtc_state->pre_csc_lut, degamma_lut); + drm_property_replace_blob(&crtc_state->post_csc_lut, gamma_lut); + + drm_property_blob_put(degamma_lut); + drm_property_blob_put(gamma_lut); + + return 0; +} + static int ivb_color_check(struct intel_crtc_state *crtc_state) { - struct drm_i915_private *dev_priv = to_i915(crtc_state->uapi.crtc->dev); - bool limited_color_range = ilk_csc_limited_range(crtc_state); + struct drm_i915_private *i915 = to_i915(crtc_state->uapi.crtc->dev); int ret; ret = check_luts(crtc_state); @@ -1534,19 +1623,21 @@ static int ivb_color_check(struct intel_crtc_state *crtc_state) if (crtc_state->output_format != INTEL_OUTPUT_FORMAT_RGB && crtc_state->hw.ctm) { - drm_dbg_kms(&dev_priv->drm, - "YCBCR and CTM together are not possible\n"); + drm_dbg_kms(&i915->drm, + "YCbCr and CTM together are not possible\n"); return -EINVAL; } - crtc_state->gamma_enable = - (crtc_state->hw.gamma_lut || - crtc_state->hw.degamma_lut) && - !crtc_state->c8_planes; + if (crtc_state->output_format != INTEL_OUTPUT_FORMAT_RGB && + crtc_state->hw.degamma_lut && crtc_state->hw.gamma_lut) { + drm_dbg_kms(&i915->drm, + "YCbCr and degamma+gamma together are not possible\n"); + return -EINVAL; + } - crtc_state->csc_enable = - crtc_state->output_format != INTEL_OUTPUT_FORMAT_RGB || - crtc_state->hw.ctm || limited_color_range; + crtc_state->gamma_enable = ilk_gamma_enable(crtc_state); + + crtc_state->csc_enable = ilk_csc_enable(crtc_state); crtc_state->gamma_mode = ivb_gamma_mode(crtc_state); @@ -1556,6 +1647,10 @@ static int ivb_color_check(struct intel_crtc_state *crtc_state) if (ret) return ret; + ret = ivb_assign_luts(crtc_state); + if (ret) + return ret; + crtc_state->preload_luts = intel_can_preload_luts(crtc_state); return 0; @@ -1570,9 +1665,26 @@ static u32 glk_gamma_mode(const struct intel_crtc_state *crtc_state) return GAMMA_MODE_MODE_10BIT; } +static void glk_assign_luts(struct intel_crtc_state *crtc_state) +{ + struct drm_i915_private *i915 = to_i915(crtc_state->uapi.crtc->dev); + + intel_assign_luts(crtc_state); + + /* + * On GLK+ both pipe CSC and degamma LUT are controlled + * by csc_enable. Hence for the cases where the CSC is + * needed but degamma LUT is not we need to load a + * linear degamma LUT. + */ + if (crtc_state->csc_enable && !crtc_state->pre_csc_lut) + drm_property_replace_blob(&crtc_state->pre_csc_lut, + i915->display.color.glk_linear_degamma_lut); +} + static int glk_color_check(struct intel_crtc_state *crtc_state) { - struct drm_i915_private *dev_priv = to_i915(crtc_state->uapi.crtc->dev); + struct drm_i915_private *i915 = to_i915(crtc_state->uapi.crtc->dev); int ret; ret = check_luts(crtc_state); @@ -1581,8 +1693,15 @@ static int glk_color_check(struct intel_crtc_state *crtc_state) if (crtc_state->output_format != INTEL_OUTPUT_FORMAT_RGB && crtc_state->hw.ctm) { - drm_dbg_kms(&dev_priv->drm, - "YCBCR and CTM together are not possible\n"); + drm_dbg_kms(&i915->drm, + "YCbCr and CTM together are not possible\n"); + return -EINVAL; + } + + if (crtc_state->output_format != INTEL_OUTPUT_FORMAT_RGB && + crtc_state->hw.degamma_lut && crtc_state->hw.gamma_lut) { + drm_dbg_kms(&i915->drm, + "YCbCr and degamma+gamma together are not possible\n"); return -EINVAL; } @@ -1604,7 +1723,9 @@ static int glk_color_check(struct intel_crtc_state *crtc_state) if (ret) return ret; - crtc_state->preload_luts = glk_can_preload_luts(crtc_state); + glk_assign_luts(crtc_state); + + crtc_state->preload_luts = intel_can_preload_luts(crtc_state); return 0; } @@ -1664,6 +1785,8 @@ static int icl_color_check(struct intel_crtc_state *crtc_state) crtc_state->csc_mode = icl_csc_mode(crtc_state); + intel_assign_luts(crtc_state); + crtc_state->preload_luts = intel_can_preload_luts(crtc_state); return 0; @@ -1749,19 +1872,19 @@ static int icl_gamma_precision(const struct intel_crtc_state *crtc_state) int intel_color_get_gamma_bit_precision(const struct intel_crtc_state *crtc_state) { struct intel_crtc *crtc = to_intel_crtc(crtc_state->uapi.crtc); - struct drm_i915_private *dev_priv = to_i915(crtc->base.dev); + struct drm_i915_private *i915 = to_i915(crtc->base.dev); - if (HAS_GMCH(dev_priv)) { - if (IS_CHERRYVIEW(dev_priv)) + if (HAS_GMCH(i915)) { + if (IS_CHERRYVIEW(i915)) return chv_gamma_precision(crtc_state); else return i9xx_gamma_precision(crtc_state); } else { - if (DISPLAY_VER(dev_priv) >= 11) + if (DISPLAY_VER(i915) >= 11) return icl_gamma_precision(crtc_state); - else if (DISPLAY_VER(dev_priv) == 10) + else if (DISPLAY_VER(i915) == 10) return glk_gamma_precision(crtc_state); - else if (IS_IRONLAKE(dev_priv)) + else if (IS_IRONLAKE(i915)) return ilk_gamma_precision(crtc_state); } @@ -1846,7 +1969,7 @@ static struct drm_property_blob *i9xx_read_lut_8(struct intel_crtc *crtc) int i; blob = drm_property_create_blob(&dev_priv->drm, - sizeof(struct drm_color_lut) * LEGACY_LUT_LENGTH, + sizeof(lut[0]) * LEGACY_LUT_LENGTH, NULL); if (IS_ERR(blob)) return NULL; @@ -1869,7 +1992,7 @@ static void i9xx_read_luts(struct intel_crtc_state *crtc_state) if (!crtc_state->gamma_enable) return; - crtc_state->hw.gamma_lut = i9xx_read_lut_8(crtc); + crtc_state->post_csc_lut = i9xx_read_lut_8(crtc); } static struct drm_property_blob *i965_read_lut_10p6(struct intel_crtc *crtc) @@ -1881,7 +2004,7 @@ static struct drm_property_blob *i965_read_lut_10p6(struct intel_crtc *crtc) struct drm_color_lut *lut; blob = drm_property_create_blob(&dev_priv->drm, - sizeof(struct drm_color_lut) * lut_size, + sizeof(lut[0]) * lut_size, NULL); if (IS_ERR(blob)) return NULL; @@ -1910,21 +2033,21 @@ static void i965_read_luts(struct intel_crtc_state *crtc_state) return; if (crtc_state->gamma_mode == GAMMA_MODE_MODE_8BIT) - crtc_state->hw.gamma_lut = i9xx_read_lut_8(crtc); + crtc_state->post_csc_lut = i9xx_read_lut_8(crtc); else - crtc_state->hw.gamma_lut = i965_read_lut_10p6(crtc); + crtc_state->post_csc_lut = i965_read_lut_10p6(crtc); } static struct drm_property_blob *chv_read_cgm_gamma(struct intel_crtc *crtc) { - struct drm_i915_private *dev_priv = to_i915(crtc->base.dev); - int i, lut_size = INTEL_INFO(dev_priv)->display.color.gamma_lut_size; + struct drm_i915_private *i915 = to_i915(crtc->base.dev); + int i, lut_size = INTEL_INFO(i915)->display.color.gamma_lut_size; enum pipe pipe = crtc->pipe; struct drm_property_blob *blob; struct drm_color_lut *lut; - blob = drm_property_create_blob(&dev_priv->drm, - sizeof(struct drm_color_lut) * lut_size, + blob = drm_property_create_blob(&i915->drm, + sizeof(lut[0]) * lut_size, NULL); if (IS_ERR(blob)) return NULL; @@ -1932,8 +2055,8 @@ static struct drm_property_blob *chv_read_cgm_gamma(struct intel_crtc *crtc) lut = blob->data; for (i = 0; i < lut_size; i++) { - u32 ldw = intel_de_read_fw(dev_priv, CGM_PIPE_GAMMA(pipe, i, 0)); - u32 udw = intel_de_read_fw(dev_priv, CGM_PIPE_GAMMA(pipe, i, 1)); + u32 ldw = intel_de_read_fw(i915, CGM_PIPE_GAMMA(pipe, i, 0)); + u32 udw = intel_de_read_fw(i915, CGM_PIPE_GAMMA(pipe, i, 1)); chv_cgm_gamma_pack(&lut[i], ldw, udw); } @@ -1946,21 +2069,21 @@ static void chv_read_luts(struct intel_crtc_state *crtc_state) struct intel_crtc *crtc = to_intel_crtc(crtc_state->uapi.crtc); if (crtc_state->cgm_mode & CGM_PIPE_MODE_GAMMA) - crtc_state->hw.gamma_lut = chv_read_cgm_gamma(crtc); + crtc_state->post_csc_lut = chv_read_cgm_gamma(crtc); else i965_read_luts(crtc_state); } static struct drm_property_blob *ilk_read_lut_8(struct intel_crtc *crtc) { - struct drm_i915_private *dev_priv = to_i915(crtc->base.dev); + struct drm_i915_private *i915 = to_i915(crtc->base.dev); enum pipe pipe = crtc->pipe; struct drm_property_blob *blob; struct drm_color_lut *lut; int i; - blob = drm_property_create_blob(&dev_priv->drm, - sizeof(struct drm_color_lut) * LEGACY_LUT_LENGTH, + blob = drm_property_create_blob(&i915->drm, + sizeof(lut[0]) * LEGACY_LUT_LENGTH, NULL); if (IS_ERR(blob)) return NULL; @@ -1968,7 +2091,7 @@ static struct drm_property_blob *ilk_read_lut_8(struct intel_crtc *crtc) lut = blob->data; for (i = 0; i < LEGACY_LUT_LENGTH; i++) { - u32 val = intel_de_read_fw(dev_priv, LGC_PALETTE(pipe, i)); + u32 val = intel_de_read_fw(i915, LGC_PALETTE(pipe, i)); i9xx_lut_8_pack(&lut[i], val); } @@ -1978,14 +2101,14 @@ static struct drm_property_blob *ilk_read_lut_8(struct intel_crtc *crtc) static struct drm_property_blob *ilk_read_lut_10(struct intel_crtc *crtc) { - struct drm_i915_private *dev_priv = to_i915(crtc->base.dev); - int i, lut_size = INTEL_INFO(dev_priv)->display.color.gamma_lut_size; + struct drm_i915_private *i915 = to_i915(crtc->base.dev); + int i, lut_size = INTEL_INFO(i915)->display.color.gamma_lut_size; enum pipe pipe = crtc->pipe; struct drm_property_blob *blob; struct drm_color_lut *lut; - blob = drm_property_create_blob(&dev_priv->drm, - sizeof(struct drm_color_lut) * lut_size, + blob = drm_property_create_blob(&i915->drm, + sizeof(lut[0]) * lut_size, NULL); if (IS_ERR(blob)) return NULL; @@ -1993,7 +2116,7 @@ static struct drm_property_blob *ilk_read_lut_10(struct intel_crtc *crtc) lut = blob->data; for (i = 0; i < lut_size; i++) { - u32 val = intel_de_read_fw(dev_priv, PREC_PALETTE(pipe, i)); + u32 val = intel_de_read_fw(i915, PREC_PALETTE(pipe, i)); ilk_lut_10_pack(&lut[i], val); } @@ -2013,10 +2136,10 @@ static void ilk_read_luts(struct intel_crtc_state *crtc_state) switch (crtc_state->gamma_mode) { case GAMMA_MODE_MODE_8BIT: - crtc_state->hw.gamma_lut = ilk_read_lut_8(crtc); + crtc_state->post_csc_lut = ilk_read_lut_8(crtc); break; case GAMMA_MODE_MODE_10BIT: - crtc_state->hw.gamma_lut = ilk_read_lut_10(crtc); + crtc_state->post_csc_lut = ilk_read_lut_10(crtc); break; default: MISSING_CASE(crtc_state->gamma_mode); @@ -2028,33 +2151,33 @@ static void ilk_read_luts(struct intel_crtc_state *crtc_state) static struct drm_property_blob *bdw_read_lut_10(struct intel_crtc *crtc, u32 prec_index) { - struct drm_i915_private *dev_priv = to_i915(crtc->base.dev); + struct drm_i915_private *i915 = to_i915(crtc->base.dev); int i, hw_lut_size = ivb_lut_10_size(prec_index); - int lut_size = INTEL_INFO(dev_priv)->display.color.gamma_lut_size; + int lut_size = INTEL_INFO(i915)->display.color.gamma_lut_size; enum pipe pipe = crtc->pipe; struct drm_property_blob *blob; struct drm_color_lut *lut; - drm_WARN_ON(&dev_priv->drm, lut_size != hw_lut_size); + drm_WARN_ON(&i915->drm, lut_size != hw_lut_size); - blob = drm_property_create_blob(&dev_priv->drm, - sizeof(struct drm_color_lut) * lut_size, + blob = drm_property_create_blob(&i915->drm, + sizeof(lut[0]) * lut_size, NULL); if (IS_ERR(blob)) return NULL; lut = blob->data; - intel_de_write_fw(dev_priv, PREC_PAL_INDEX(pipe), + intel_de_write_fw(i915, PREC_PAL_INDEX(pipe), prec_index | PAL_PREC_AUTO_INCREMENT); for (i = 0; i < lut_size; i++) { - u32 val = intel_de_read_fw(dev_priv, PREC_PAL_DATA(pipe)); + u32 val = intel_de_read_fw(i915, PREC_PAL_DATA(pipe)); ilk_lut_10_pack(&lut[i], val); } - intel_de_write_fw(dev_priv, PREC_PAL_INDEX(pipe), 0); + intel_de_write_fw(i915, PREC_PAL_INDEX(pipe), 0); return blob; } @@ -2068,10 +2191,10 @@ static void glk_read_luts(struct intel_crtc_state *crtc_state) switch (crtc_state->gamma_mode) { case GAMMA_MODE_MODE_8BIT: - crtc_state->hw.gamma_lut = ilk_read_lut_8(crtc); + crtc_state->post_csc_lut = ilk_read_lut_8(crtc); break; case GAMMA_MODE_MODE_10BIT: - crtc_state->hw.gamma_lut = bdw_read_lut_10(crtc, PAL_PREC_INDEX_VALUE(0)); + crtc_state->post_csc_lut = bdw_read_lut_10(crtc, PAL_PREC_INDEX_VALUE(0)); break; default: MISSING_CASE(crtc_state->gamma_mode); @@ -2082,31 +2205,31 @@ static void glk_read_luts(struct intel_crtc_state *crtc_state) static struct drm_property_blob * icl_read_lut_multi_segment(struct intel_crtc *crtc) { - struct drm_i915_private *dev_priv = to_i915(crtc->base.dev); - int i, lut_size = INTEL_INFO(dev_priv)->display.color.gamma_lut_size; + struct drm_i915_private *i915 = to_i915(crtc->base.dev); + int i, lut_size = INTEL_INFO(i915)->display.color.gamma_lut_size; enum pipe pipe = crtc->pipe; struct drm_property_blob *blob; struct drm_color_lut *lut; - blob = drm_property_create_blob(&dev_priv->drm, - sizeof(struct drm_color_lut) * lut_size, + blob = drm_property_create_blob(&i915->drm, + sizeof(lut[0]) * lut_size, NULL); if (IS_ERR(blob)) return NULL; lut = blob->data; - intel_de_write_fw(dev_priv, PREC_PAL_MULTI_SEG_INDEX(pipe), + intel_de_write_fw(i915, PREC_PAL_MULTI_SEG_INDEX(pipe), PAL_PREC_AUTO_INCREMENT); for (i = 0; i < 9; i++) { - u32 ldw = intel_de_read_fw(dev_priv, PREC_PAL_MULTI_SEG_DATA(pipe)); - u32 udw = intel_de_read_fw(dev_priv, PREC_PAL_MULTI_SEG_DATA(pipe)); + u32 ldw = intel_de_read_fw(i915, PREC_PAL_MULTI_SEG_DATA(pipe)); + u32 udw = intel_de_read_fw(i915, PREC_PAL_MULTI_SEG_DATA(pipe)); - icl_lut_multi_seg_pack(&lut[i], ldw, udw); + ilk_lut_12p4_pack(&lut[i], ldw, udw); } - intel_de_write_fw(dev_priv, PREC_PAL_MULTI_SEG_INDEX(pipe), 0); + intel_de_write_fw(i915, PREC_PAL_MULTI_SEG_INDEX(pipe), 0); /* * FIXME readouts from PAL_PREC_DATA register aren't giving @@ -2126,13 +2249,13 @@ static void icl_read_luts(struct intel_crtc_state *crtc_state) switch (crtc_state->gamma_mode & GAMMA_MODE_MODE_MASK) { case GAMMA_MODE_MODE_8BIT: - crtc_state->hw.gamma_lut = ilk_read_lut_8(crtc); + crtc_state->post_csc_lut = ilk_read_lut_8(crtc); break; case GAMMA_MODE_MODE_10BIT: - crtc_state->hw.gamma_lut = bdw_read_lut_10(crtc, PAL_PREC_INDEX_VALUE(0)); + crtc_state->post_csc_lut = bdw_read_lut_10(crtc, PAL_PREC_INDEX_VALUE(0)); break; case GAMMA_MODE_MODE_12BIT_MULTI_SEGMENTED: - crtc_state->hw.gamma_lut = icl_read_lut_multi_segment(crtc); + crtc_state->post_csc_lut = icl_read_lut_multi_segment(crtc); break; default: MISSING_CASE(crtc_state->gamma_mode); @@ -2217,41 +2340,58 @@ static const struct intel_color_funcs ilk_color_funcs = { .read_luts = ilk_read_luts, }; -void intel_color_init(struct intel_crtc *crtc) +void intel_color_crtc_init(struct intel_crtc *crtc) { - struct drm_i915_private *dev_priv = to_i915(crtc->base.dev); - bool has_ctm = INTEL_INFO(dev_priv)->display.color.degamma_lut_size != 0; + struct drm_i915_private *i915 = to_i915(crtc->base.dev); + bool has_ctm = INTEL_INFO(i915)->display.color.degamma_lut_size != 0; drm_mode_crtc_set_gamma_size(&crtc->base, 256); - if (HAS_GMCH(dev_priv)) { - if (IS_CHERRYVIEW(dev_priv)) { - dev_priv->display.funcs.color = &chv_color_funcs; - } else if (DISPLAY_VER(dev_priv) >= 4) { - dev_priv->display.funcs.color = &i965_color_funcs; - } else { - dev_priv->display.funcs.color = &i9xx_color_funcs; - } - } else { - if (DISPLAY_VER(dev_priv) >= 11) - dev_priv->display.funcs.color = &icl_color_funcs; - else if (DISPLAY_VER(dev_priv) == 10) - dev_priv->display.funcs.color = &glk_color_funcs; - else if (DISPLAY_VER(dev_priv) == 9) - dev_priv->display.funcs.color = &skl_color_funcs; - else if (DISPLAY_VER(dev_priv) == 8) - dev_priv->display.funcs.color = &bdw_color_funcs; - else if (DISPLAY_VER(dev_priv) == 7) { - if (IS_HASWELL(dev_priv)) - dev_priv->display.funcs.color = &hsw_color_funcs; - else - dev_priv->display.funcs.color = &ivb_color_funcs; - } else - dev_priv->display.funcs.color = &ilk_color_funcs; - } - drm_crtc_enable_color_mgmt(&crtc->base, - INTEL_INFO(dev_priv)->display.color.degamma_lut_size, + INTEL_INFO(i915)->display.color.degamma_lut_size, has_ctm, - INTEL_INFO(dev_priv)->display.color.gamma_lut_size); + INTEL_INFO(i915)->display.color.gamma_lut_size); +} + +int intel_color_init(struct drm_i915_private *i915) +{ + struct drm_property_blob *blob; + + if (DISPLAY_VER(i915) != 10) + return 0; + + blob = create_linear_lut(i915, INTEL_INFO(i915)->display.color.degamma_lut_size); + if (IS_ERR(blob)) + return PTR_ERR(blob); + + i915->display.color.glk_linear_degamma_lut = blob; + + return 0; +} + +void intel_color_init_hooks(struct drm_i915_private *i915) +{ + if (HAS_GMCH(i915)) { + if (IS_CHERRYVIEW(i915)) + i915->display.funcs.color = &chv_color_funcs; + else if (DISPLAY_VER(i915) >= 4) + i915->display.funcs.color = &i965_color_funcs; + else + i915->display.funcs.color = &i9xx_color_funcs; + } else { + if (DISPLAY_VER(i915) >= 11) + i915->display.funcs.color = &icl_color_funcs; + else if (DISPLAY_VER(i915) == 10) + i915->display.funcs.color = &glk_color_funcs; + else if (DISPLAY_VER(i915) == 9) + i915->display.funcs.color = &skl_color_funcs; + else if (DISPLAY_VER(i915) == 8) + i915->display.funcs.color = &bdw_color_funcs; + else if (IS_HASWELL(i915)) + i915->display.funcs.color = &hsw_color_funcs; + else if (DISPLAY_VER(i915) == 7) + i915->display.funcs.color = &ivb_color_funcs; + else + i915->display.funcs.color = &ilk_color_funcs; + } } diff --git a/drivers/gpu/drm/i915/display/intel_color.h b/drivers/gpu/drm/i915/display/intel_color.h index fd873425e082..2a5ada67774d 100644 --- a/drivers/gpu/drm/i915/display/intel_color.h +++ b/drivers/gpu/drm/i915/display/intel_color.h @@ -10,9 +10,12 @@ struct intel_crtc_state; struct intel_crtc; +struct drm_i915_private; struct drm_property_blob; -void intel_color_init(struct intel_crtc *crtc); +void intel_color_init_hooks(struct drm_i915_private *i915); +int intel_color_init(struct drm_i915_private *i915); +void intel_color_crtc_init(struct intel_crtc *crtc); int intel_color_check(struct intel_crtc_state *crtc_state); void intel_color_commit_noarm(const struct intel_crtc_state *crtc_state); void intel_color_commit_arm(const struct intel_crtc_state *crtc_state); @@ -22,5 +25,6 @@ int intel_color_get_gamma_bit_precision(const struct intel_crtc_state *crtc_stat bool intel_color_lut_equal(struct drm_property_blob *blob1, struct drm_property_blob *blob2, u32 gamma_mode, u32 bit_precision); +void intel_color_assert_luts(const struct intel_crtc_state *crtc_state); #endif /* __INTEL_COLOR_H__ */ diff --git a/drivers/gpu/drm/i915/display/intel_combo_phy.c b/drivers/gpu/drm/i915/display/intel_combo_phy.c index 64890f39c3cc..8b870b2dd4f9 100644 --- a/drivers/gpu/drm/i915/display/intel_combo_phy.c +++ b/drivers/gpu/drm/i915/display/intel_combo_phy.c @@ -3,6 +3,7 @@ * Copyright © 2018 Intel Corporation */ +#include "i915_reg.h" #include "intel_combo_phy.h" #include "intel_combo_phy_regs.h" #include "intel_de.h" @@ -53,7 +54,6 @@ static const struct icl_procmon { static const struct icl_procmon * icl_get_procmon_ref_values(struct drm_i915_private *dev_priv, enum phy phy) { - const struct icl_procmon *procmon; u32 val; val = intel_de_read(dev_priv, ICL_PORT_COMP_DW3(phy)); @@ -62,23 +62,16 @@ icl_get_procmon_ref_values(struct drm_i915_private *dev_priv, enum phy phy) MISSING_CASE(val); fallthrough; case VOLTAGE_INFO_0_85V | PROCESS_INFO_DOT_0: - procmon = &icl_procmon_values[PROCMON_0_85V_DOT_0]; - break; + return &icl_procmon_values[PROCMON_0_85V_DOT_0]; case VOLTAGE_INFO_0_95V | PROCESS_INFO_DOT_0: - procmon = &icl_procmon_values[PROCMON_0_95V_DOT_0]; - break; + return &icl_procmon_values[PROCMON_0_95V_DOT_0]; case VOLTAGE_INFO_0_95V | PROCESS_INFO_DOT_1: - procmon = &icl_procmon_values[PROCMON_0_95V_DOT_1]; - break; + return &icl_procmon_values[PROCMON_0_95V_DOT_1]; case VOLTAGE_INFO_1_05V | PROCESS_INFO_DOT_0: - procmon = &icl_procmon_values[PROCMON_1_05V_DOT_0]; - break; + return &icl_procmon_values[PROCMON_1_05V_DOT_0]; case VOLTAGE_INFO_1_05V | PROCESS_INFO_DOT_1: - procmon = &icl_procmon_values[PROCMON_1_05V_DOT_1]; - break; + return &icl_procmon_values[PROCMON_1_05V_DOT_1]; } - - return procmon; } static void icl_set_procmon_ref_values(struct drm_i915_private *dev_priv, diff --git a/drivers/gpu/drm/i915/display/intel_connector.c b/drivers/gpu/drm/i915/display/intel_connector.c index 6d5cbeb8df4d..6205ddd3ded0 100644 --- a/drivers/gpu/drm/i915/display/intel_connector.c +++ b/drivers/gpu/drm/i915/display/intel_connector.c @@ -293,3 +293,21 @@ intel_attach_dp_colorspace_property(struct drm_connector *connector) if (!drm_mode_create_dp_colorspace_property(connector)) drm_connector_attach_colorspace_property(connector); } + +void +intel_attach_scaling_mode_property(struct drm_connector *connector) +{ + struct drm_i915_private *i915 = to_i915(connector->dev); + u32 scaling_modes; + + scaling_modes = BIT(DRM_MODE_SCALE_ASPECT) | + BIT(DRM_MODE_SCALE_FULLSCREEN); + + /* On GMCH platforms borders are only possible on the LVDS port */ + if (!HAS_GMCH(i915) || connector->connector_type == DRM_MODE_CONNECTOR_LVDS) + scaling_modes |= BIT(DRM_MODE_SCALE_CENTER); + + drm_connector_attach_scaling_mode_property(connector, scaling_modes); + + connector->state->scaling_mode = DRM_MODE_SCALE_ASPECT; +} diff --git a/drivers/gpu/drm/i915/display/intel_connector.h b/drivers/gpu/drm/i915/display/intel_connector.h index 661a37a3c6d8..9d2bc261b204 100644 --- a/drivers/gpu/drm/i915/display/intel_connector.h +++ b/drivers/gpu/drm/i915/display/intel_connector.h @@ -6,7 +6,7 @@ #ifndef __INTEL_CONNECTOR_H__ #define __INTEL_CONNECTOR_H__ -#include "intel_display.h" +#include <linux/types.h> struct drm_connector; struct edid; @@ -32,5 +32,6 @@ void intel_attach_broadcast_rgb_property(struct drm_connector *connector); void intel_attach_aspect_ratio_property(struct drm_connector *connector); void intel_attach_hdmi_colorspace_property(struct drm_connector *connector); void intel_attach_dp_colorspace_property(struct drm_connector *connector); +void intel_attach_scaling_mode_property(struct drm_connector *connector); #endif /* __INTEL_CONNECTOR_H__ */ diff --git a/drivers/gpu/drm/i915/display/intel_crt.c b/drivers/gpu/drm/i915/display/intel_crt.c index 4a8ff2f97608..797ad9489f7e 100644 --- a/drivers/gpu/drm/i915/display/intel_crt.c +++ b/drivers/gpu/drm/i915/display/intel_crt.c @@ -34,6 +34,8 @@ #include <drm/drm_probe_helper.h> #include "i915_drv.h" +#include "i915_irq.h" +#include "i915_reg.h" #include "intel_connector.h" #include "intel_crt.h" #include "intel_crtc.h" @@ -1044,17 +1046,14 @@ void intel_crt_init(struct drm_i915_private *dev_priv) intel_connector_attach_encoder(intel_connector, &crt->base); crt->base.type = INTEL_OUTPUT_ANALOG; - crt->base.cloneable = (1 << INTEL_OUTPUT_DVO) | (1 << INTEL_OUTPUT_HDMI); + crt->base.cloneable = BIT(INTEL_OUTPUT_DVO) | BIT(INTEL_OUTPUT_HDMI); if (IS_I830(dev_priv)) crt->base.pipe_mask = BIT(PIPE_A); else crt->base.pipe_mask = ~0; - if (DISPLAY_VER(dev_priv) == 2) - connector->interlace_allowed = 0; - else - connector->interlace_allowed = 1; - connector->doublescan_allowed = 0; + if (DISPLAY_VER(dev_priv) != 2) + connector->interlace_allowed = true; crt->adpa_reg = adpa_reg; diff --git a/drivers/gpu/drm/i915/display/intel_crtc.c b/drivers/gpu/drm/i915/display/intel_crtc.c index 6792a9056f46..037fc140b585 100644 --- a/drivers/gpu/drm/i915/display/intel_crtc.c +++ b/drivers/gpu/drm/i915/display/intel_crtc.c @@ -365,9 +365,8 @@ int intel_crtc_init(struct drm_i915_private *dev_priv, enum pipe pipe) BIT(DRM_SCALING_FILTER_DEFAULT) | BIT(DRM_SCALING_FILTER_NEAREST_NEIGHBOR)); - intel_color_init(crtc); - - intel_crtc_drrs_init(crtc); + intel_color_crtc_init(crtc); + intel_drrs_crtc_init(crtc); intel_crtc_crc_init(crtc); cpu_latency_qos_add_request(&crtc->vblank_pm_qos, PM_QOS_DEFAULT_VALUE); @@ -387,8 +386,7 @@ static bool intel_crtc_needs_vblank_work(const struct intel_crtc_state *crtc_sta return crtc_state->hw.active && !intel_crtc_needs_modeset(crtc_state) && !crtc_state->preload_luts && - (crtc_state->uapi.color_mgmt_changed || - crtc_state->update_pipe); + intel_crtc_needs_color_update(crtc_state); } static void intel_crtc_vblank_work(struct kthread_work *base) diff --git a/drivers/gpu/drm/i915/display/intel_crtc_state_dump.c b/drivers/gpu/drm/i915/display/intel_crtc_state_dump.c index e9212f69c360..e3273fe8ddac 100644 --- a/drivers/gpu/drm/i915/display/intel_crtc_state_dump.c +++ b/drivers/gpu/drm/i915/display/intel_crtc_state_dump.c @@ -298,11 +298,13 @@ void intel_crtc_state_dump(const struct intel_crtc_state *pipe_config, pipe_config->csc_mode, pipe_config->gamma_mode, pipe_config->gamma_enable, pipe_config->csc_enable); - drm_dbg_kms(&i915->drm, "degamma lut: %d entries, gamma lut: %d entries\n", - pipe_config->hw.degamma_lut ? - drm_color_lut_size(pipe_config->hw.degamma_lut) : 0, - pipe_config->hw.gamma_lut ? - drm_color_lut_size(pipe_config->hw.gamma_lut) : 0); + drm_dbg_kms(&i915->drm, "pre csc lut: %s%d entries, post csc lut: %d entries\n", + pipe_config->pre_csc_lut && pipe_config->pre_csc_lut == + i915->display.color.glk_linear_degamma_lut ? "(linear) " : "", + pipe_config->pre_csc_lut ? + drm_color_lut_size(pipe_config->pre_csc_lut) : 0, + pipe_config->post_csc_lut ? + drm_color_lut_size(pipe_config->post_csc_lut) : 0); dump_planes: if (!state) diff --git a/drivers/gpu/drm/i915/display/intel_cursor.c b/drivers/gpu/drm/i915/display/intel_cursor.c index 87899e89b3a7..d190fa0d393b 100644 --- a/drivers/gpu/drm/i915/display/intel_cursor.c +++ b/drivers/gpu/drm/i915/display/intel_cursor.c @@ -10,12 +10,13 @@ #include <drm/drm_damage_helper.h> #include <drm/drm_fourcc.h> +#include "i915_reg.h" #include "intel_atomic.h" #include "intel_atomic_plane.h" #include "intel_cursor.h" #include "intel_de.h" -#include "intel_display_types.h" #include "intel_display.h" +#include "intel_display_types.h" #include "intel_fb.h" #include "intel_fb_pin.h" #include "intel_frontbuffer.h" @@ -631,8 +632,10 @@ intel_legacy_cursor_update(struct drm_plane *_plane, * * FIXME bigjoiner fastpath would be good */ - if (!crtc_state->hw.active || intel_crtc_needs_modeset(crtc_state) || - crtc_state->update_pipe || crtc_state->bigjoiner_pipes) + if (!crtc_state->hw.active || + intel_crtc_needs_modeset(crtc_state) || + intel_crtc_needs_fastset(crtc_state) || + crtc_state->bigjoiner_pipes) goto slow; /* diff --git a/drivers/gpu/drm/i915/display/intel_ddi.c b/drivers/gpu/drm/i915/display/intel_ddi.c index 69ecf2a3d6c6..0f1ec2a98cc8 100644 --- a/drivers/gpu/drm/i915/display/intel_ddi.c +++ b/drivers/gpu/drm/i915/display/intel_ddi.c @@ -31,6 +31,7 @@ #include <drm/drm_privacy_screen_consumer.h> #include "i915_drv.h" +#include "i915_reg.h" #include "intel_audio.h" #include "intel_audio_regs.h" #include "intel_backlight.h" @@ -44,6 +45,7 @@ #include "intel_display_power.h" #include "intel_display_types.h" #include "intel_dkl_phy.h" +#include "intel_dkl_phy_regs.h" #include "intel_dp.h" #include "intel_dp_link_training.h" #include "intel_dp_mst.h" @@ -55,14 +57,15 @@ #include "intel_hdcp.h" #include "intel_hdmi.h" #include "intel_hotplug.h" +#include "intel_hti.h" #include "intel_lspcon.h" +#include "intel_mg_phy_regs.h" #include "intel_pps.h" #include "intel_psr.h" #include "intel_quirks.h" #include "intel_snps_phy.h" #include "intel_sprite.h" #include "intel_tc.h" -#include "intel_tc_phy_regs.h" #include "intel_vdsc.h" #include "intel_vrr.h" #include "skl_scaler.h" @@ -845,22 +848,65 @@ bool intel_ddi_get_hw_state(struct intel_encoder *encoder, } static enum intel_display_power_domain -intel_ddi_main_link_aux_domain(struct intel_digital_port *dig_port) +intel_ddi_main_link_aux_domain(struct intel_digital_port *dig_port, + const struct intel_crtc_state *crtc_state) { - /* ICL+ HW requires corresponding AUX IOs to be powered up for PSR with + struct drm_i915_private *i915 = to_i915(dig_port->base.base.dev); + enum phy phy = intel_port_to_phy(i915, dig_port->base.port); + + /* + * ICL+ HW requires corresponding AUX IOs to be powered up for PSR with * DC states enabled at the same time, while for driver initiated AUX * transfers we need the same AUX IOs to be powered but with DC states - * disabled. Accordingly use the AUX power domain here which leaves DC - * states enabled. - * However, for non-A AUX ports the corresponding non-EDP transcoders - * would have already enabled power well 2 and DC_OFF. This means we can - * acquire a wider POWER_DOMAIN_AUX_{B,C,D,F} reference instead of a - * specific AUX_IO reference without powering up any extra wells. - * Note that PSR is enabled only on Port A even though this function - * returns the correct domain for other ports too. + * disabled. Accordingly use the AUX_IO_<port> power domain here which + * leaves DC states enabled. + * + * Before MTL TypeC PHYs (in all TypeC modes and both DP/HDMI) also require + * AUX IO to be enabled, but all these require DC_OFF to be enabled as + * well, so we can acquire a wider AUX_<port> power domain reference + * instead of a specific AUX_IO_<port> reference without powering up any + * extra wells. */ - return dig_port->aux_ch == AUX_CH_A ? POWER_DOMAIN_AUX_IO_A : - intel_aux_power_domain(dig_port); + if (intel_encoder_can_psr(&dig_port->base)) + return intel_display_power_aux_io_domain(i915, dig_port->aux_ch); + else if (DISPLAY_VER(i915) < 14 && + (intel_crtc_has_dp_encoder(crtc_state) || + intel_phy_is_tc(i915, phy))) + return intel_aux_power_domain(dig_port); + else + return POWER_DOMAIN_INVALID; +} + +static void +main_link_aux_power_domain_get(struct intel_digital_port *dig_port, + const struct intel_crtc_state *crtc_state) +{ + struct drm_i915_private *i915 = to_i915(dig_port->base.base.dev); + enum intel_display_power_domain domain = + intel_ddi_main_link_aux_domain(dig_port, crtc_state); + + drm_WARN_ON(&i915->drm, dig_port->aux_wakeref); + + if (domain == POWER_DOMAIN_INVALID) + return; + + dig_port->aux_wakeref = intel_display_power_get(i915, domain); +} + +static void +main_link_aux_power_domain_put(struct intel_digital_port *dig_port, + const struct intel_crtc_state *crtc_state) +{ + struct drm_i915_private *i915 = to_i915(dig_port->base.base.dev); + enum intel_display_power_domain domain = + intel_ddi_main_link_aux_domain(dig_port, crtc_state); + intel_wakeref_t wf; + + wf = fetch_and_zero(&dig_port->aux_wakeref); + if (!wf) + return; + + intel_display_power_put(i915, domain, wf); } static void intel_ddi_get_power_domains(struct intel_encoder *encoder, @@ -868,7 +914,6 @@ static void intel_ddi_get_power_domains(struct intel_encoder *encoder, { struct drm_i915_private *dev_priv = to_i915(encoder->base.dev); struct intel_digital_port *dig_port; - enum phy phy = intel_port_to_phy(dev_priv, encoder->port); /* * TODO: Add support for MST encoders. Atm, the following should never @@ -887,17 +932,7 @@ static void intel_ddi_get_power_domains(struct intel_encoder *encoder, dig_port->ddi_io_power_domain); } - /* - * AUX power is only needed for (e)DP mode, and for HDMI mode on TC - * ports. - */ - if (intel_crtc_has_dp_encoder(crtc_state) || - intel_phy_is_tc(dev_priv, phy)) { - drm_WARN_ON(&dev_priv->drm, dig_port->aux_wakeref); - dig_port->aux_wakeref = - intel_display_power_get(dev_priv, - intel_ddi_main_link_aux_domain(dig_port)); - } + main_link_aux_power_domain_get(dig_port, crtc_state); } void intel_ddi_enable_pipe_clock(struct intel_encoder *encoder, @@ -1263,11 +1298,11 @@ static void tgl_dkl_phy_set_signal_levels(struct intel_encoder *encoder, for (ln = 0; ln < 2; ln++) { int level; - intel_dkl_phy_write(dev_priv, DKL_TX_PMD_LANE_SUS(tc_port), ln, 0); + intel_dkl_phy_write(dev_priv, DKL_TX_PMD_LANE_SUS(tc_port, ln), 0); level = intel_ddi_level(encoder, crtc_state, 2*ln+0); - intel_dkl_phy_rmw(dev_priv, DKL_TX_DPCNTL0(tc_port), ln, + intel_dkl_phy_rmw(dev_priv, DKL_TX_DPCNTL0(tc_port, ln), DKL_TX_PRESHOOT_COEFF_MASK | DKL_TX_DE_EMPAHSIS_COEFF_MASK | DKL_TX_VSWING_CONTROL_MASK, @@ -1277,7 +1312,7 @@ static void tgl_dkl_phy_set_signal_levels(struct intel_encoder *encoder, level = intel_ddi_level(encoder, crtc_state, 2*ln+1); - intel_dkl_phy_rmw(dev_priv, DKL_TX_DPCNTL1(tc_port), ln, + intel_dkl_phy_rmw(dev_priv, DKL_TX_DPCNTL1(tc_port, ln), DKL_TX_PRESHOOT_COEFF_MASK | DKL_TX_DE_EMPAHSIS_COEFF_MASK | DKL_TX_VSWING_CONTROL_MASK, @@ -1285,7 +1320,7 @@ static void tgl_dkl_phy_set_signal_levels(struct intel_encoder *encoder, DKL_TX_DE_EMPHASIS_COEFF(trans->entries[level].dkl.de_emphasis) | DKL_TX_VSWING_CONTROL(trans->entries[level].dkl.vswing)); - intel_dkl_phy_rmw(dev_priv, DKL_TX_DPCNTL2(tc_port), ln, + intel_dkl_phy_rmw(dev_priv, DKL_TX_DPCNTL2(tc_port, ln), DKL_TX_DP20BITMODE, 0); if (IS_ALDERLAKE_P(dev_priv)) { @@ -1304,7 +1339,7 @@ static void tgl_dkl_phy_set_signal_levels(struct intel_encoder *encoder, val |= DKL_TX_DPCNTL2_CFG_LOADGENSELECT_TX2(0); } - intel_dkl_phy_rmw(dev_priv, DKL_TX_DPCNTL2(tc_port), ln, + intel_dkl_phy_rmw(dev_priv, DKL_TX_DPCNTL2(tc_port, ln), DKL_TX_DPCNTL2_CFG_LOADGENSELECT_TX1_MASK | DKL_TX_DPCNTL2_CFG_LOADGENSELECT_TX2_MASK, val); @@ -2017,8 +2052,8 @@ icl_program_mg_dp_mode(struct intel_digital_port *dig_port, return; if (DISPLAY_VER(dev_priv) >= 12) { - ln0 = intel_dkl_phy_read(dev_priv, DKL_DP_MODE(tc_port), 0); - ln1 = intel_dkl_phy_read(dev_priv, DKL_DP_MODE(tc_port), 1); + ln0 = intel_dkl_phy_read(dev_priv, DKL_DP_MODE(tc_port, 0)); + ln1 = intel_dkl_phy_read(dev_priv, DKL_DP_MODE(tc_port, 1)); } else { ln0 = intel_de_read(dev_priv, MG_DP_MODE(0, tc_port)); ln1 = intel_de_read(dev_priv, MG_DP_MODE(1, tc_port)); @@ -2079,8 +2114,8 @@ icl_program_mg_dp_mode(struct intel_digital_port *dig_port, } if (DISPLAY_VER(dev_priv) >= 12) { - intel_dkl_phy_write(dev_priv, DKL_DP_MODE(tc_port), 0, ln0); - intel_dkl_phy_write(dev_priv, DKL_DP_MODE(tc_port), 1, ln1); + intel_dkl_phy_write(dev_priv, DKL_DP_MODE(tc_port, 0), ln0); + intel_dkl_phy_write(dev_priv, DKL_DP_MODE(tc_port, 1), ln1); } else { intel_de_write(dev_priv, MG_DP_MODE(0, tc_port), ln0); intel_de_write(dev_priv, MG_DP_MODE(1, tc_port), ln1); @@ -2736,10 +2771,7 @@ static void intel_ddi_post_disable(struct intel_atomic_state *state, intel_ddi_post_disable_dp(state, encoder, old_crtc_state, old_conn_state); - if (intel_crtc_has_dp_encoder(old_crtc_state) || is_tc_port) - intel_display_power_put(dev_priv, - intel_ddi_main_link_aux_domain(dig_port), - fetch_and_zero(&dig_port->aux_wakeref)); + main_link_aux_power_domain_put(dig_port, old_crtc_state); if (is_tc_port) intel_tc_port_put_link(dig_port); @@ -3060,12 +3092,7 @@ intel_ddi_pre_pll_enable(struct intel_atomic_state *state, if (is_tc_port) intel_tc_port_get_link(dig_port, crtc_state->lane_count); - if (intel_crtc_has_dp_encoder(crtc_state) || is_tc_port) { - drm_WARN_ON(&dev_priv->drm, dig_port->aux_wakeref); - dig_port->aux_wakeref = - intel_display_power_get(dev_priv, - intel_ddi_main_link_aux_domain(dig_port)); - } + main_link_aux_power_domain_get(dig_port, crtc_state); if (is_tc_port && !intel_tc_port_in_tbt_alt_mode(dig_port)) /* @@ -3085,7 +3112,7 @@ static void adlp_tbt_to_dp_alt_switch_wa(struct intel_encoder *encoder) int ln; for (ln = 0; ln < 2; ln++) - intel_dkl_phy_rmw(i915, DKL_PCS_DW5(tc_port), ln, DKL_PCS_DW5_CORE_SOFTRESET, 0); + intel_dkl_phy_rmw(i915, DKL_PCS_DW5(tc_port, ln), DKL_PCS_DW5_CORE_SOFTRESET, 0); } static void intel_ddi_prepare_link_retrain(struct intel_dp *intel_dp, @@ -3524,7 +3551,7 @@ static void icl_ddi_tc_get_clock(struct intel_encoder *encoder, if (drm_WARN_ON(&i915->drm, !pll)) return; - if (intel_get_shared_dpll_id(i915, pll) == DPLL_ID_ICL_TBTPLL) + if (pll->info->id == DPLL_ID_ICL_TBTPLL) port_dpll_id = ICL_PORT_DPLL_DEFAULT; else port_dpll_id = ICL_PORT_DPLL_MG_PHY; @@ -3537,7 +3564,7 @@ static void icl_ddi_tc_get_clock(struct intel_encoder *encoder, icl_set_active_port_dpll(crtc_state, port_dpll_id); - if (intel_get_shared_dpll_id(i915, crtc_state->shared_dpll) == DPLL_ID_ICL_TBTPLL) + if (crtc_state->shared_dpll->info->id == DPLL_ID_ICL_TBTPLL) crtc_state->port_clock = icl_calc_tbt_pll_link(i915, encoder->port); else crtc_state->port_clock = intel_dpll_get_freq(i915, crtc_state->shared_dpll, @@ -3579,7 +3606,7 @@ static void intel_ddi_sync_state(struct intel_encoder *encoder, enum phy phy = intel_port_to_phy(i915, encoder->port); if (intel_phy_is_tc(i915, phy)) - intel_tc_port_sanitize(enc_to_dig_port(encoder)); + intel_tc_port_sanitize_mode(enc_to_dig_port(encoder)); if (crtc_state && intel_crtc_has_dp_encoder(crtc_state)) intel_dp_sync_state(encoder, crtc_state); @@ -3789,11 +3816,17 @@ static void intel_ddi_encoder_destroy(struct drm_encoder *encoder) static void intel_ddi_encoder_reset(struct drm_encoder *encoder) { + struct drm_i915_private *i915 = to_i915(encoder->dev); struct intel_dp *intel_dp = enc_to_intel_dp(to_intel_encoder(encoder)); + struct intel_digital_port *dig_port = enc_to_dig_port(to_intel_encoder(encoder)); + enum phy phy = intel_port_to_phy(i915, dig_port->base.port); intel_dp->reset_link_params = true; intel_pps_encoder_reset(intel_dp); + + if (intel_phy_is_tc(i915, phy)) + intel_tc_port_init_mode(dig_port); } static const struct drm_encoder_funcs intel_ddi_funcs = { @@ -4106,12 +4139,6 @@ intel_ddi_max_lanes(struct intel_digital_port *dig_port) return max_lanes; } -static bool hti_uses_phy(struct drm_i915_private *i915, enum phy phy) -{ - return i915->hti_state & HDPORT_ENABLED && - i915->hti_state & HDPORT_DDI_USED(phy); -} - static enum hpd_pin xelpd_hpd_pin(struct drm_i915_private *dev_priv, enum port port) { @@ -4240,7 +4267,7 @@ void intel_ddi_init(struct drm_i915_private *dev_priv, enum port port) * driver. In that case we should skip initializing the corresponding * outputs. */ - if (hti_uses_phy(dev_priv, phy)) { + if (intel_hti_uses_phy(dev_priv, phy)) { drm_dbg_kms(&dev_priv->drm, "PORT %c / PHY %c reserved by HTI\n", port_name(port), phy_name(phy)); return; diff --git a/drivers/gpu/drm/i915/display/intel_display.c b/drivers/gpu/drm/i915/display/intel_display.c index de77054195c6..6c2686ecb62a 100644 --- a/drivers/gpu/drm/i915/display/intel_display.c +++ b/drivers/gpu/drm/i915/display/intel_display.c @@ -76,6 +76,7 @@ #include "g4x_hdmi.h" #include "hsw_ips.h" #include "i915_drv.h" +#include "i915_reg.h" #include "i915_utils.h" #include "icl_dsi.h" #include "intel_acpi.h" @@ -90,6 +91,7 @@ #include "intel_display_types.h" #include "intel_dmc.h" #include "intel_dp_link_training.h" +#include "intel_dpio_phy.h" #include "intel_dpt.h" #include "intel_dsb.h" #include "intel_fbc.h" @@ -99,6 +101,7 @@ #include "intel_frontbuffer.h" #include "intel_hdcp.h" #include "intel_hotplug.h" +#include "intel_hti.h" #include "intel_modeset_verify.h" #include "intel_modeset_setup.h" #include "intel_overlay.h" @@ -831,13 +834,27 @@ intel_plane_fence_y_offset(const struct intel_plane_state *plane_state) } static int +intel_display_commit_duplicated_state(struct intel_atomic_state *state, + struct drm_modeset_acquire_ctx *ctx) +{ + struct drm_i915_private *i915 = to_i915(state->base.dev); + int ret; + + ret = drm_atomic_helper_commit_duplicated_state(&state->base, ctx); + + drm_WARN_ON(&i915->drm, ret == -EDEADLK); + + return ret; +} + +static int __intel_display_resume(struct drm_i915_private *i915, struct drm_atomic_state *state, struct drm_modeset_acquire_ctx *ctx) { struct drm_crtc_state *crtc_state; struct drm_crtc *crtc; - int i, ret; + int i; intel_modeset_setup_hw_state(i915, ctx); intel_vga_redisable(i915); @@ -863,11 +880,7 @@ __intel_display_resume(struct drm_i915_private *i915, if (!HAS_GMCH(i915)) to_intel_atomic_state(state)->skip_intermediate_wm = true; - ret = drm_atomic_helper_commit_duplicated_state(state, ctx); - - drm_WARN_ON(&i915->drm, ret == -EDEADLK); - - return ret; + return intel_display_commit_duplicated_state(to_intel_atomic_state(state), ctx); } static bool gpu_reset_clobbers_display(struct drm_i915_private *dev_priv) @@ -878,8 +891,7 @@ static bool gpu_reset_clobbers_display(struct drm_i915_private *dev_priv) void intel_display_prepare_reset(struct drm_i915_private *dev_priv) { - struct drm_device *dev = &dev_priv->drm; - struct drm_modeset_acquire_ctx *ctx = &dev_priv->reset_ctx; + struct drm_modeset_acquire_ctx *ctx = &dev_priv->display.restore.reset_ctx; struct drm_atomic_state *state; int ret; @@ -906,10 +918,10 @@ void intel_display_prepare_reset(struct drm_i915_private *dev_priv) * Need mode_config.mutex so that we don't * trample ongoing ->detect() and whatnot. */ - mutex_lock(&dev->mode_config.mutex); + mutex_lock(&dev_priv->drm.mode_config.mutex); drm_modeset_acquire_init(ctx, 0); while (1) { - ret = drm_modeset_lock_all_ctx(dev, ctx); + ret = drm_modeset_lock_all_ctx(&dev_priv->drm, ctx); if (ret != -EDEADLK) break; @@ -919,7 +931,7 @@ void intel_display_prepare_reset(struct drm_i915_private *dev_priv) * Disabling the crtcs gracefully seems nicer. Also the * g33 docs say we should at least disable all the planes. */ - state = drm_atomic_helper_duplicate_state(dev, ctx); + state = drm_atomic_helper_duplicate_state(&dev_priv->drm, ctx); if (IS_ERR(state)) { ret = PTR_ERR(state); drm_err(&dev_priv->drm, "Duplicating state failed with %i\n", @@ -927,7 +939,7 @@ void intel_display_prepare_reset(struct drm_i915_private *dev_priv) return; } - ret = drm_atomic_helper_disable_all(dev, ctx); + ret = drm_atomic_helper_disable_all(&dev_priv->drm, ctx); if (ret) { drm_err(&dev_priv->drm, "Suspending crtc's failed with %i\n", ret); @@ -935,13 +947,13 @@ void intel_display_prepare_reset(struct drm_i915_private *dev_priv) return; } - dev_priv->modeset_restore_state = state; + dev_priv->display.restore.modeset_state = state; state->acquire_ctx = ctx; } void intel_display_finish_reset(struct drm_i915_private *i915) { - struct drm_modeset_acquire_ctx *ctx = &i915->reset_ctx; + struct drm_modeset_acquire_ctx *ctx = &i915->display.restore.reset_ctx; struct drm_atomic_state *state; int ret; @@ -952,14 +964,14 @@ void intel_display_finish_reset(struct drm_i915_private *i915) if (!test_bit(I915_RESET_MODESET, &to_gt(i915)->reset.flags)) return; - state = fetch_and_zero(&i915->modeset_restore_state); + state = fetch_and_zero(&i915->display.restore.modeset_state); if (!state) goto unlock; /* reset doesn't touch the display */ if (!gpu_reset_clobbers_display(i915)) { /* for testing only restore the display */ - ret = __intel_display_resume(i915, state, ctx); + ret = intel_display_commit_duplicated_state(to_intel_atomic_state(state), ctx); if (ret) drm_err(&i915->drm, "Restoring old state failed with %i\n", ret); @@ -1252,8 +1264,6 @@ static void intel_post_plane_update(struct intel_atomic_state *state, if (needs_cursorclk_wa(old_crtc_state) && !needs_cursorclk_wa(new_crtc_state)) icl_wa_cursorclkgating(dev_priv, pipe, false); - - intel_drrs_activate(new_crtc_state); } static void intel_crtc_enable_flip_done(struct intel_atomic_state *state, @@ -2434,7 +2444,7 @@ int intel_display_suspend(struct drm_device *dev) drm_err(&dev_priv->drm, "Suspending crtc's failed with %i\n", ret); else - dev_priv->modeset_restore_state = state; + dev_priv->display.restore.modeset_state = state; return ret; } @@ -4045,20 +4055,19 @@ static bool hsw_get_pipe_config(struct intel_crtc *crtc, struct intel_crtc_state *pipe_config) { struct drm_i915_private *dev_priv = to_i915(crtc->base.dev); - struct intel_display_power_domain_set power_domain_set = { }; bool active; u32 tmp; - if (!intel_display_power_get_in_set_if_enabled(dev_priv, &power_domain_set, + if (!intel_display_power_get_in_set_if_enabled(dev_priv, &crtc->hw_readout_power_domains, POWER_DOMAIN_PIPE(crtc->pipe))) return false; pipe_config->shared_dpll = NULL; - active = hsw_get_transcoder_state(crtc, pipe_config, &power_domain_set); + active = hsw_get_transcoder_state(crtc, pipe_config, &crtc->hw_readout_power_domains); if ((IS_GEMINILAKE(dev_priv) || IS_BROXTON(dev_priv)) && - bxt_get_dsi_transcoder_state(crtc, pipe_config, &power_domain_set)) { + bxt_get_dsi_transcoder_state(crtc, pipe_config, &crtc->hw_readout_power_domains)) { drm_WARN_ON(&dev_priv->drm, active); active = true; } @@ -4117,7 +4126,7 @@ static bool hsw_get_pipe_config(struct intel_crtc *crtc, pipe_config->ips_linetime = REG_FIELD_GET(HSW_IPS_LINETIME_MASK, tmp); - if (intel_display_power_get_in_set_if_enabled(dev_priv, &power_domain_set, + if (intel_display_power_get_in_set_if_enabled(dev_priv, &crtc->hw_readout_power_domains, POWER_DOMAIN_PIPE_PANEL_FITTER(crtc->pipe))) { if (DISPLAY_VER(dev_priv) >= 9) skl_get_pfit_config(pipe_config); @@ -4148,7 +4157,7 @@ static bool hsw_get_pipe_config(struct intel_crtc *crtc, } out: - intel_display_power_put_all_in_set(dev_priv, &power_domain_set); + intel_display_power_put_all_in_set(dev_priv, &crtc->hw_readout_power_domains); return active; } @@ -4576,8 +4585,8 @@ static bool encoders_cloneable(const struct intel_encoder *a, const struct intel_encoder *b) { /* masks could be asymmetric, so check both ways */ - return a == b || (a->cloneable & (1 << b->type) && - b->cloneable & (1 << a->type)); + return a == b || (a->cloneable & BIT(b->type) && + b->cloneable & BIT(a->type)); } static bool check_single_encoder_cloning(struct intel_atomic_state *state, @@ -4828,14 +4837,14 @@ static int intel_crtc_atomic_check(struct intel_atomic_state *state, struct drm_i915_private *dev_priv = to_i915(crtc->base.dev); struct intel_crtc_state *crtc_state = intel_atomic_get_new_crtc_state(state, crtc); - bool mode_changed = intel_crtc_needs_modeset(crtc_state); int ret; if (DISPLAY_VER(dev_priv) < 5 && !IS_G4X(dev_priv) && - mode_changed && !crtc_state->hw.active) + intel_crtc_needs_modeset(crtc_state) && + !crtc_state->hw.active) crtc_state->update_wm_post = true; - if (mode_changed) { + if (intel_crtc_needs_modeset(crtc_state)) { ret = intel_dpll_crtc_get_shared_dpll(state, crtc); if (ret) return ret; @@ -4848,8 +4857,7 @@ static int intel_crtc_atomic_check(struct intel_atomic_state *state, if (c8_planes_changed(crtc_state)) crtc_state->uapi.color_mgmt_changed = true; - if (mode_changed || crtc_state->update_pipe || - crtc_state->uapi.color_mgmt_changed) { + if (intel_crtc_needs_color_update(crtc_state)) { ret = intel_color_check(crtc_state); if (ret) return ret; @@ -4875,7 +4883,8 @@ static int intel_crtc_atomic_check(struct intel_atomic_state *state, } if (DISPLAY_VER(dev_priv) >= 9) { - if (mode_changed || crtc_state->update_pipe) { + if (intel_crtc_needs_modeset(crtc_state) || + intel_crtc_needs_fastset(crtc_state)) { ret = skl_update_scaler_crtc(crtc_state); if (ret) return ret; @@ -5641,39 +5650,6 @@ intel_pipe_config_compare(const struct intel_crtc_state *current_config, PIPE_CONF_CHECK_I(name.y2); \ } while (0) -/* This is required for BDW+ where there is only one set of registers for - * switching between high and low RR. - * This macro can be used whenever a comparison has to be made between one - * hw state and multiple sw state variables. - */ -#define PIPE_CONF_CHECK_M_N_ALT(name, alt_name) do { \ - if (!intel_compare_link_m_n(¤t_config->name, \ - &pipe_config->name) && \ - !intel_compare_link_m_n(¤t_config->alt_name, \ - &pipe_config->name)) { \ - pipe_config_mismatch(fastset, crtc, __stringify(name), \ - "(expected tu %i data %i/%i link %i/%i, " \ - "or tu %i data %i/%i link %i/%i, " \ - "found tu %i, data %i/%i link %i/%i)", \ - current_config->name.tu, \ - current_config->name.data_m, \ - current_config->name.data_n, \ - current_config->name.link_m, \ - current_config->name.link_n, \ - current_config->alt_name.tu, \ - current_config->alt_name.data_m, \ - current_config->alt_name.data_n, \ - current_config->alt_name.link_m, \ - current_config->alt_name.link_n, \ - pipe_config->name.tu, \ - pipe_config->name.data_m, \ - pipe_config->name.data_n, \ - pipe_config->name.link_m, \ - pipe_config->name.link_n); \ - ret = false; \ - } \ -} while (0) - #define PIPE_CONF_CHECK_FLAGS(name, mask) do { \ if ((current_config->name ^ pipe_config->name) & (mask)) { \ pipe_config_mismatch(fastset, crtc, __stringify(name), \ @@ -5742,7 +5718,7 @@ intel_pipe_config_compare(const struct intel_crtc_state *current_config, if (HAS_DOUBLE_BUFFERED_M_N(dev_priv)) { if (!fastset || !pipe_config->seamless_m_n) - PIPE_CONF_CHECK_M_N_ALT(dp_m_n, dp_m2_n2); + PIPE_CONF_CHECK_M_N(dp_m_n); } else { PIPE_CONF_CHECK_M_N(dp_m_n); PIPE_CONF_CHECK_M_N(dp_m2_n2); @@ -5819,7 +5795,7 @@ intel_pipe_config_compare(const struct intel_crtc_state *current_config, bp_gamma = intel_color_get_gamma_bit_precision(pipe_config); if (bp_gamma) - PIPE_CONF_CHECK_COLOR_LUT(gamma_mode, hw.gamma_lut, bp_gamma); + PIPE_CONF_CHECK_COLOR_LUT(gamma_mode, post_csc_lut, bp_gamma); if (current_config->active_planes) { PIPE_CONF_CHECK_BOOL(has_psr); @@ -5941,7 +5917,8 @@ intel_verify_planes(struct intel_atomic_state *state) plane_state->uapi.visible); } -int intel_modeset_all_pipes(struct intel_atomic_state *state) +int intel_modeset_all_pipes(struct intel_atomic_state *state, + const char *reason) { struct drm_i915_private *dev_priv = to_i915(state->base.dev); struct intel_crtc *crtc; @@ -5959,10 +5936,14 @@ int intel_modeset_all_pipes(struct intel_atomic_state *state) return PTR_ERR(crtc_state); if (!crtc_state->hw.active || - drm_atomic_crtc_needs_modeset(&crtc_state->uapi)) + intel_crtc_needs_modeset(crtc_state)) continue; + drm_dbg_kms(&dev_priv->drm, "[CRTC:%d:%s] Full modeset due to %s\n", + crtc->base.base.id, crtc->base.name, reason); + crtc_state->uapi.mode_changed = true; + crtc_state->update_pipe = false; ret = drm_atomic_add_affected_connectors(&state->base, &crtc->base); @@ -6138,7 +6119,8 @@ static void intel_crtc_check_fastset(const struct intel_crtc_state *old_crtc_sta return; new_crtc_state->uapi.mode_changed = false; - new_crtc_state->update_pipe = true; + if (!intel_crtc_needs_modeset(new_crtc_state)) + new_crtc_state->update_pipe = true; } static int intel_crtc_add_planes_to_state(struct intel_atomic_state *state, @@ -6910,12 +6892,19 @@ static int intel_atomic_check(struct drm_device *dev, for_each_oldnew_intel_crtc_in_state(state, crtc, old_crtc_state, new_crtc_state, i) { + intel_color_assert_luts(new_crtc_state); + ret = intel_async_flip_check_hw(state, crtc); if (ret) goto fail; + /* Either full modeset or fastset (or neither), never both */ + drm_WARN_ON(&dev_priv->drm, + intel_crtc_needs_modeset(new_crtc_state) && + intel_crtc_needs_fastset(new_crtc_state)); + if (!intel_crtc_needs_modeset(new_crtc_state) && - !new_crtc_state->update_pipe) + !intel_crtc_needs_fastset(new_crtc_state)) continue; intel_crtc_state_dump(new_crtc_state, state, @@ -6951,12 +6940,8 @@ static int intel_atomic_prepare_commit(struct intel_atomic_state *state) return ret; for_each_new_intel_crtc_in_state(state, crtc, crtc_state, i) { - bool mode_changed = intel_crtc_needs_modeset(crtc_state); - - if (mode_changed || crtc_state->update_pipe || - crtc_state->uapi.color_mgmt_changed) { + if (intel_crtc_needs_color_update(crtc_state)) intel_dsb_prepare(crtc_state); - } } return 0; @@ -7037,14 +7022,13 @@ static void commit_pipe_pre_planes(struct intel_atomic_state *state, * CRTC was enabled. */ if (!modeset) { - if (new_crtc_state->uapi.color_mgmt_changed || - new_crtc_state->update_pipe) + if (intel_crtc_needs_color_update(new_crtc_state)) intel_color_commit_arm(new_crtc_state); if (DISPLAY_VER(dev_priv) >= 9 || IS_BROADWELL(dev_priv)) bdw_set_pipemisc(new_crtc_state); - if (new_crtc_state->update_pipe) + if (intel_crtc_needs_fastset(new_crtc_state)) intel_pipe_fastset(old_crtc_state, new_crtc_state); } @@ -7103,25 +7087,23 @@ static void intel_update_crtc(struct intel_atomic_state *state, if (!modeset) { if (new_crtc_state->preload_luts && - (new_crtc_state->uapi.color_mgmt_changed || - new_crtc_state->update_pipe)) + intel_crtc_needs_color_update(new_crtc_state)) intel_color_load_luts(new_crtc_state); intel_pre_plane_update(state, crtc); - if (new_crtc_state->update_pipe) + if (intel_crtc_needs_fastset(new_crtc_state)) intel_encoders_update_pipe(state, crtc); if (DISPLAY_VER(i915) >= 11 && - new_crtc_state->update_pipe) + intel_crtc_needs_fastset(new_crtc_state)) icl_set_pipe_chicken(new_crtc_state); } intel_fbc_update(state, crtc); if (!modeset && - (new_crtc_state->uapi.color_mgmt_changed || - new_crtc_state->update_pipe)) + intel_crtc_needs_color_update(new_crtc_state)) intel_color_commit_noarm(new_crtc_state); intel_crtc_planes_update_noarm(state, crtc); @@ -7143,7 +7125,7 @@ static void intel_update_crtc(struct intel_atomic_state *state, * valid pipe configuration from the BIOS we need to take care * of enabling them on the CRTC's first fastset. */ - if (new_crtc_state->update_pipe && !modeset && + if (intel_crtc_needs_fastset(new_crtc_state) && !modeset && old_crtc_state->inherited) intel_crtc_arm_fifo_underrun(crtc, new_crtc_state); } @@ -7166,9 +7148,7 @@ static void intel_old_crtc_state_disables(struct intel_atomic_state *state, intel_fbc_disable(crtc); intel_disable_shared_dpll(old_crtc_state); - /* FIXME unify this for all platforms */ - if (!new_crtc_state->hw.active && - !HAS_GMCH(dev_priv)) + if (!new_crtc_state->hw.active) intel_initial_watermarks(state, crtc); } @@ -7503,9 +7483,8 @@ static void intel_atomic_commit_tail(struct intel_atomic_state *state) for_each_oldnew_intel_crtc_in_state(state, crtc, old_crtc_state, new_crtc_state, i) { if (intel_crtc_needs_modeset(new_crtc_state) || - new_crtc_state->update_pipe) { + intel_crtc_needs_fastset(new_crtc_state)) intel_modeset_get_crtc_power_domains(new_crtc_state, &put_domains[crtc->pipe]); - } } intel_commit_modeset_disables(state); @@ -7609,6 +7588,12 @@ static void intel_atomic_commit_tail(struct intel_atomic_state *state) intel_modeset_verify_crtc(crtc, state, old_crtc_state, new_crtc_state); /* + * Activate DRRS after state readout to avoid + * dp_m_n vs. dp_m2_n2 confusion on BDW+. + */ + intel_drrs_activate(new_crtc_state); + + /* * DSB cleanup is done in cleanup_work aligning with framebuffer * cleanup. So copy and reset the dsb structure to sync with * commit_done and later do dsb cleanup in cleanup_work. @@ -8348,6 +8333,7 @@ void intel_init_display_hooks(struct drm_i915_private *dev_priv) if (!HAS_DISPLAY(dev_priv)) return; + intel_color_init_hooks(dev_priv); intel_init_cdclk_hooks(dev_priv); intel_audio_hooks_init(dev_priv); @@ -8589,7 +8575,7 @@ static void intel_mode_config_init(struct drm_i915_private *i915) struct drm_mode_config *mode_config = &i915->drm.mode_config; drm_mode_config_init(&i915->drm); - INIT_LIST_HEAD(&i915->global_obj_list); + INIT_LIST_HEAD(&i915->display.global.obj_list); mode_config->min_width = 0; mode_config->min_height = 0; @@ -8678,6 +8664,10 @@ int intel_modeset_init_noirq(struct drm_i915_private *i915) if (ret) goto cleanup_vga_client_pw_domain_dmc; + ret = intel_color_init(i915); + if (ret) + goto cleanup_vga_client_pw_domain_dmc; + ret = intel_dbuf_init(i915); if (ret) goto cleanup_vga_client_pw_domain_dmc; @@ -8750,12 +8740,7 @@ int intel_modeset_init_nogem(struct drm_i915_private *i915) if (i915->display.cdclk.max_cdclk_freq == 0) intel_update_max_cdclk(i915); - /* - * If the platform has HTI, we need to find out whether it has reserved - * any display resources before we create our display outputs. - */ - if (INTEL_INFO(i915)->display.has_hti) - i915->hti_state = intel_de_read(i915, HDPORT_STATE); + intel_hti_init(i915); /* Just disable it once at startup */ intel_vga_disable(i915); @@ -8918,14 +8903,14 @@ void i830_disable_pipe(struct drm_i915_private *dev_priv, enum pipe pipe) void intel_display_resume(struct drm_device *dev) { struct drm_i915_private *i915 = to_i915(dev); - struct drm_atomic_state *state = i915->modeset_restore_state; + struct drm_atomic_state *state = i915->display.restore.modeset_state; struct drm_modeset_acquire_ctx ctx; int ret; if (!HAS_DISPLAY(i915)) return; - i915->modeset_restore_state = NULL; + i915->display.restore.modeset_state = NULL; if (state) state->acquire_ctx = &ctx; diff --git a/drivers/gpu/drm/i915/display/intel_display.h b/drivers/gpu/drm/i915/display/intel_display.h index 884e8e67b17c..714030136b7f 100644 --- a/drivers/gpu/drm/i915/display/intel_display.h +++ b/drivers/gpu/drm/i915/display/intel_display.h @@ -53,6 +53,7 @@ struct intel_digital_port; struct intel_dp; struct intel_encoder; struct intel_initial_plane_config; +struct intel_link_m_n; struct intel_load_detect_pipe; struct intel_plane; struct intel_plane_state; @@ -61,24 +62,6 @@ struct intel_remapped_info; struct intel_rotation_info; struct pci_dev; -enum i915_gpio { - GPIOA, - GPIOB, - GPIOC, - GPIOD, - GPIOE, - GPIOF, - GPIOG, - GPIOH, - __GPIOI_UNUSED, - GPIOJ, - GPIOK, - GPIOL, - GPIOM, - GPION, - GPIOO, -}; - /* * Keep the pipe enum values fixed: the code assumes that PIPE_A=0, the * rest have consecutive values and match the enum values of transcoders @@ -279,17 +262,6 @@ enum tc_port_mode { TC_PORT_LEGACY, }; -enum dpio_channel { - DPIO_CH0, - DPIO_CH1 -}; - -enum dpio_phy { - DPIO_PHY0, - DPIO_PHY1, - DPIO_PHY2, -}; - enum aux_ch { AUX_CH_A, AUX_CH_B, @@ -316,15 +288,6 @@ enum aux_ch { #define aux_ch_name(a) ((a) + 'A') -/* Used by dp and fdi links */ -struct intel_link_m_n { - u32 tu; - u32 data_m; - u32 data_n; - u32 link_m; - u32 link_n; -}; - enum phy { PHY_NONE = -1, @@ -469,10 +432,6 @@ enum hpd_pin { list_for_each_entry((intel_encoder), &(dev)->mode_config.encoder_list, base.head) \ for_each_if((intel_encoder)->base.crtc == (__crtc)) -#define for_each_connector_on_encoder(dev, __encoder, intel_connector) \ - list_for_each_entry((intel_connector), &(dev)->mode_config.connector_list, base.head) \ - for_each_if((intel_connector)->base.encoder == (__encoder)) - #define for_each_old_intel_plane_in_state(__state, plane, old_plane_state, __i) \ for ((__i) = 0; \ (__i) < (__state)->base.dev->mode_config.num_total_plane && \ @@ -683,7 +642,8 @@ void intel_modeset_driver_remove(struct drm_i915_private *i915); void intel_modeset_driver_remove_noirq(struct drm_i915_private *i915); void intel_modeset_driver_remove_nogem(struct drm_i915_private *i915); void intel_display_resume(struct drm_device *dev); -int intel_modeset_all_pipes(struct intel_atomic_state *state); +int intel_modeset_all_pipes(struct intel_atomic_state *state, + const char *reason); void intel_modeset_get_crtc_power_domains(struct intel_crtc_state *crtc_state, struct intel_power_domain_mask *old_domains); void intel_modeset_put_crtc_power_domains(struct intel_crtc *crtc, diff --git a/drivers/gpu/drm/i915/display/intel_display_core.h b/drivers/gpu/drm/i915/display/intel_display_core.h index 9b51148e8ba5..57ddce3ba02b 100644 --- a/drivers/gpu/drm/i915/display/intel_display_core.h +++ b/drivers/gpu/drm/i915/display/intel_display_core.h @@ -14,6 +14,7 @@ #include <linux/workqueue.h> #include <drm/drm_connector.h> +#include <drm/drm_modeset_lock.h> #include "intel_cdclk.h" #include "intel_display.h" @@ -28,6 +29,7 @@ struct drm_i915_private; struct drm_property; +struct drm_property_blob; struct i915_audio_component; struct i915_hdcp_comp_master; struct intel_atomic_state; @@ -309,6 +311,10 @@ struct intel_display { } cdclk; struct { + struct drm_property_blob *glk_linear_degamma_lut; + } color; + + struct { /* The current hardware dbuf configuration */ u8 enabled_slices; @@ -340,6 +346,10 @@ struct intel_display { } fdi; struct { + struct list_head obj_list; + } global; + + struct { /* * Base address of where the gmbus and gpio blocks are located * (either on PCH or on SoC for platforms without PCH). @@ -366,6 +376,16 @@ struct intel_display { } hdcp; struct { + /* + * HTI (aka HDPORT) state read during initial hw readout. Most + * platforms don't have HTI, so this will just stay 0. Those + * that do will use this later to figure out which PLLs and PHYs + * are unavailable for driver usage. + */ + u32 state; + } hti; + + struct { struct i915_power_domains domains; /* Shadow for DISPLAY_PHY_CONTROL which can't be safely read */ @@ -392,6 +412,12 @@ struct intel_display { } quirks; struct { + /* restore state for suspend/resume and display reset */ + struct drm_atomic_state *modeset_state; + struct drm_modeset_acquire_ctx reset_ctx; + } restore; + + struct { enum { I915_SAGV_UNKNOWN = 0, I915_SAGV_DISABLED, diff --git a/drivers/gpu/drm/i915/display/intel_display_debugfs.c b/drivers/gpu/drm/i915/display/intel_display_debugfs.c index 7c7253a2541c..7bcd90384a46 100644 --- a/drivers/gpu/drm/i915/display/intel_display_debugfs.c +++ b/drivers/gpu/drm/i915/display/intel_display_debugfs.c @@ -9,6 +9,8 @@ #include <drm/drm_fourcc.h> #include "i915_debugfs.h" +#include "i915_irq.h" +#include "i915_reg.h" #include "intel_de.h" #include "intel_display_debugfs.h" #include "intel_display_power.h" @@ -22,6 +24,7 @@ #include "intel_fbdev.h" #include "intel_hdcp.h" #include "intel_hdmi.h" +#include "intel_hotplug.h" #include "intel_panel.h" #include "intel_pm.h" #include "intel_psr.h" @@ -127,7 +130,6 @@ static int i915_vbt(struct seq_file *m, void *unused) static int i915_gem_framebuffer_info(struct seq_file *m, void *data) { struct drm_i915_private *dev_priv = node_to_i915(m->private); - struct drm_device *dev = &dev_priv->drm; struct intel_framebuffer *fbdev_fb = NULL; struct drm_framebuffer *drm_fb; @@ -146,8 +148,8 @@ static int i915_gem_framebuffer_info(struct seq_file *m, void *data) } #endif - mutex_lock(&dev->mode_config.fb_lock); - drm_for_each_fb(drm_fb, dev) { + mutex_lock(&dev_priv->drm.mode_config.fb_lock); + drm_for_each_fb(drm_fb, &dev_priv->drm) { struct intel_framebuffer *fb = to_intel_framebuffer(drm_fb); if (fb == fbdev_fb) continue; @@ -162,7 +164,7 @@ static int i915_gem_framebuffer_info(struct seq_file *m, void *data) i915_debugfs_describe_obj(m, intel_fb_obj(&fb->base)); seq_putc(m, '\n'); } - mutex_unlock(&dev->mode_config.fb_lock); + mutex_unlock(&dev_priv->drm.mode_config.fb_lock); return 0; } @@ -897,7 +899,6 @@ static void intel_crtc_info(struct seq_file *m, struct intel_crtc *crtc) static int i915_display_info(struct seq_file *m, void *unused) { struct drm_i915_private *dev_priv = node_to_i915(m->private); - struct drm_device *dev = &dev_priv->drm; struct intel_crtc *crtc; struct drm_connector *connector; struct drm_connector_list_iter conn_iter; @@ -905,22 +906,22 @@ static int i915_display_info(struct seq_file *m, void *unused) wakeref = intel_runtime_pm_get(&dev_priv->runtime_pm); - drm_modeset_lock_all(dev); + drm_modeset_lock_all(&dev_priv->drm); seq_printf(m, "CRTC info\n"); seq_printf(m, "---------\n"); - for_each_intel_crtc(dev, crtc) + for_each_intel_crtc(&dev_priv->drm, crtc) intel_crtc_info(m, crtc); seq_printf(m, "\n"); seq_printf(m, "Connector info\n"); seq_printf(m, "--------------\n"); - drm_connector_list_iter_begin(dev, &conn_iter); + drm_connector_list_iter_begin(&dev_priv->drm, &conn_iter); drm_for_each_connector_iter(connector, &conn_iter) intel_connector_info(m, connector); drm_connector_list_iter_end(&conn_iter); - drm_modeset_unlock_all(dev); + drm_modeset_unlock_all(&dev_priv->drm); intel_runtime_pm_put(&dev_priv->runtime_pm, wakeref); @@ -930,10 +931,9 @@ static int i915_display_info(struct seq_file *m, void *unused) static int i915_shared_dplls_info(struct seq_file *m, void *unused) { struct drm_i915_private *dev_priv = node_to_i915(m->private); - struct drm_device *dev = &dev_priv->drm; int i; - drm_modeset_lock_all(dev); + drm_modeset_lock_all(&dev_priv->drm); seq_printf(m, "PLL refclks: non-SSC: %d kHz, SSC: %d kHz\n", dev_priv->display.dpll.ref_clks.nssc, @@ -978,7 +978,7 @@ static int i915_shared_dplls_info(struct seq_file *m, void *unused) seq_printf(m, " mg_pll_tdc_coldst_bias: 0x%08x\n", pll->state.hw_state.mg_pll_tdc_coldst_bias); } - drm_modeset_unlock_all(dev); + drm_modeset_unlock_all(&dev_priv->drm); return 0; } @@ -986,14 +986,13 @@ static int i915_shared_dplls_info(struct seq_file *m, void *unused) static int i915_ddb_info(struct seq_file *m, void *unused) { struct drm_i915_private *dev_priv = node_to_i915(m->private); - struct drm_device *dev = &dev_priv->drm; struct skl_ddb_entry *entry; struct intel_crtc *crtc; if (DISPLAY_VER(dev_priv) < 9) return -ENODEV; - drm_modeset_lock_all(dev); + drm_modeset_lock_all(&dev_priv->drm); seq_printf(m, "%-15s%8s%8s%8s\n", "", "Start", "End", "Size"); @@ -1017,53 +1016,7 @@ static int i915_ddb_info(struct seq_file *m, void *unused) entry->end, skl_ddb_entry_size(entry)); } - drm_modeset_unlock_all(dev); - - return 0; -} - -static int i915_drrs_status(struct seq_file *m, void *unused) -{ - struct drm_i915_private *dev_priv = node_to_i915(m->private); - struct drm_connector_list_iter conn_iter; - struct intel_connector *connector; - struct intel_crtc *crtc; - - drm_connector_list_iter_begin(&dev_priv->drm, &conn_iter); - for_each_intel_connector_iter(connector, &conn_iter) { - seq_printf(m, "[CONNECTOR:%d:%s] DRRS type: %s\n", - connector->base.base.id, connector->base.name, - intel_drrs_type_str(intel_panel_drrs_type(connector))); - } - drm_connector_list_iter_end(&conn_iter); - - seq_puts(m, "\n"); - - for_each_intel_crtc(&dev_priv->drm, crtc) { - const struct intel_crtc_state *crtc_state = - to_intel_crtc_state(crtc->base.state); - - seq_printf(m, "[CRTC:%d:%s]:\n", - crtc->base.base.id, crtc->base.name); - - mutex_lock(&crtc->drrs.mutex); - - /* DRRS Supported */ - seq_printf(m, "\tDRRS Enabled: %s\n", - str_yes_no(crtc_state->has_drrs)); - - seq_printf(m, "\tDRRS Active: %s\n", - str_yes_no(intel_drrs_is_active(crtc))); - - seq_printf(m, "\tBusy_frontbuffer_bits: 0x%X\n", - crtc->drrs.busy_frontbuffer_bits); - - seq_printf(m, "\tDRRS refresh rate: %s\n", - crtc->drrs.refresh_rate == DRRS_REFRESH_RATE_LOW ? - "low" : "high"); - - mutex_unlock(&crtc->drrs.mutex); - } + drm_modeset_unlock_all(&dev_priv->drm); return 0; } @@ -1107,13 +1060,12 @@ static int i915_lpsp_status(struct seq_file *m, void *unused) static int i915_dp_mst_info(struct seq_file *m, void *unused) { struct drm_i915_private *dev_priv = node_to_i915(m->private); - struct drm_device *dev = &dev_priv->drm; struct intel_encoder *intel_encoder; struct intel_digital_port *dig_port; struct drm_connector *connector; struct drm_connector_list_iter conn_iter; - drm_connector_list_iter_begin(dev, &conn_iter); + drm_connector_list_iter_begin(&dev_priv->drm, &conn_iter); drm_for_each_connector_iter(connector, &conn_iter) { if (connector->connector_type != DRM_MODE_CONNECTOR_DisplayPort) continue; @@ -1200,12 +1152,11 @@ static ssize_t i915_displayport_test_active_write(struct file *file, static int i915_displayport_test_active_show(struct seq_file *m, void *data) { struct drm_i915_private *dev_priv = m->private; - struct drm_device *dev = &dev_priv->drm; struct drm_connector *connector; struct drm_connector_list_iter conn_iter; struct intel_dp *intel_dp; - drm_connector_list_iter_begin(dev, &conn_iter); + drm_connector_list_iter_begin(&dev_priv->drm, &conn_iter); drm_for_each_connector_iter(connector, &conn_iter) { struct intel_encoder *encoder; @@ -1250,12 +1201,11 @@ static const struct file_operations i915_displayport_test_active_fops = { static int i915_displayport_test_data_show(struct seq_file *m, void *data) { struct drm_i915_private *dev_priv = m->private; - struct drm_device *dev = &dev_priv->drm; struct drm_connector *connector; struct drm_connector_list_iter conn_iter; struct intel_dp *intel_dp; - drm_connector_list_iter_begin(dev, &conn_iter); + drm_connector_list_iter_begin(&dev_priv->drm, &conn_iter); drm_for_each_connector_iter(connector, &conn_iter) { struct intel_encoder *encoder; @@ -1304,12 +1254,11 @@ DEFINE_SHOW_ATTRIBUTE(i915_displayport_test_data); static int i915_displayport_test_type_show(struct seq_file *m, void *data) { struct drm_i915_private *dev_priv = m->private; - struct drm_device *dev = &dev_priv->drm; struct drm_connector *connector; struct drm_connector_list_iter conn_iter; struct intel_dp *intel_dp; - drm_connector_list_iter_begin(dev, &conn_iter); + drm_connector_list_iter_begin(&dev_priv->drm, &conn_iter); drm_for_each_connector_iter(connector, &conn_iter) { struct intel_encoder *encoder; @@ -1336,7 +1285,6 @@ DEFINE_SHOW_ATTRIBUTE(i915_displayport_test_type); static void wm_latency_show(struct seq_file *m, const u16 wm[8]) { struct drm_i915_private *dev_priv = m->private; - struct drm_device *dev = &dev_priv->drm; int level; int num_levels; @@ -1349,7 +1297,7 @@ static void wm_latency_show(struct seq_file *m, const u16 wm[8]) else num_levels = ilk_wm_max_level(dev_priv) + 1; - drm_modeset_lock_all(dev); + drm_modeset_lock_all(&dev_priv->drm); for (level = 0; level < num_levels; level++) { unsigned int latency = wm[level]; @@ -1370,7 +1318,7 @@ static void wm_latency_show(struct seq_file *m, const u16 wm[8]) level, wm[level], latency / 10, latency % 10); } - drm_modeset_unlock_all(dev); + drm_modeset_unlock_all(&dev_priv->drm); } static int pri_wm_latency_show(struct seq_file *m, void *data) @@ -1453,7 +1401,6 @@ static ssize_t wm_latency_write(struct file *file, const char __user *ubuf, { struct seq_file *m = file->private_data; struct drm_i915_private *dev_priv = m->private; - struct drm_device *dev = &dev_priv->drm; u16 new[8] = { 0 }; int num_levels; int level; @@ -1483,12 +1430,12 @@ static ssize_t wm_latency_write(struct file *file, const char __user *ubuf, if (ret != num_levels) return -EINVAL; - drm_modeset_lock_all(dev); + drm_modeset_lock_all(&dev_priv->drm); for (level = 0; level < num_levels; level++) wm[level] = new[level]; - drm_modeset_unlock_all(dev); + drm_modeset_unlock_all(&dev_priv->drm); return len; } @@ -1566,209 +1513,6 @@ static const struct file_operations i915_cur_wm_latency_fops = { .write = cur_wm_latency_write }; -static int i915_hpd_storm_ctl_show(struct seq_file *m, void *data) -{ - struct drm_i915_private *dev_priv = m->private; - struct intel_hotplug *hotplug = &dev_priv->display.hotplug; - - /* Synchronize with everything first in case there's been an HPD - * storm, but we haven't finished handling it in the kernel yet - */ - intel_synchronize_irq(dev_priv); - flush_work(&dev_priv->display.hotplug.dig_port_work); - flush_delayed_work(&dev_priv->display.hotplug.hotplug_work); - - seq_printf(m, "Threshold: %d\n", hotplug->hpd_storm_threshold); - seq_printf(m, "Detected: %s\n", - str_yes_no(delayed_work_pending(&hotplug->reenable_work))); - - return 0; -} - -static ssize_t i915_hpd_storm_ctl_write(struct file *file, - const char __user *ubuf, size_t len, - loff_t *offp) -{ - struct seq_file *m = file->private_data; - struct drm_i915_private *dev_priv = m->private; - struct intel_hotplug *hotplug = &dev_priv->display.hotplug; - unsigned int new_threshold; - int i; - char *newline; - char tmp[16]; - - if (len >= sizeof(tmp)) - return -EINVAL; - - if (copy_from_user(tmp, ubuf, len)) - return -EFAULT; - - tmp[len] = '\0'; - - /* Strip newline, if any */ - newline = strchr(tmp, '\n'); - if (newline) - *newline = '\0'; - - if (strcmp(tmp, "reset") == 0) - new_threshold = HPD_STORM_DEFAULT_THRESHOLD; - else if (kstrtouint(tmp, 10, &new_threshold) != 0) - return -EINVAL; - - if (new_threshold > 0) - drm_dbg_kms(&dev_priv->drm, - "Setting HPD storm detection threshold to %d\n", - new_threshold); - else - drm_dbg_kms(&dev_priv->drm, "Disabling HPD storm detection\n"); - - spin_lock_irq(&dev_priv->irq_lock); - hotplug->hpd_storm_threshold = new_threshold; - /* Reset the HPD storm stats so we don't accidentally trigger a storm */ - for_each_hpd_pin(i) - hotplug->stats[i].count = 0; - spin_unlock_irq(&dev_priv->irq_lock); - - /* Re-enable hpd immediately if we were in an irq storm */ - flush_delayed_work(&dev_priv->display.hotplug.reenable_work); - - return len; -} - -static int i915_hpd_storm_ctl_open(struct inode *inode, struct file *file) -{ - return single_open(file, i915_hpd_storm_ctl_show, inode->i_private); -} - -static const struct file_operations i915_hpd_storm_ctl_fops = { - .owner = THIS_MODULE, - .open = i915_hpd_storm_ctl_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, - .write = i915_hpd_storm_ctl_write -}; - -static int i915_hpd_short_storm_ctl_show(struct seq_file *m, void *data) -{ - struct drm_i915_private *dev_priv = m->private; - - seq_printf(m, "Enabled: %s\n", - str_yes_no(dev_priv->display.hotplug.hpd_short_storm_enabled)); - - return 0; -} - -static int -i915_hpd_short_storm_ctl_open(struct inode *inode, struct file *file) -{ - return single_open(file, i915_hpd_short_storm_ctl_show, - inode->i_private); -} - -static ssize_t i915_hpd_short_storm_ctl_write(struct file *file, - const char __user *ubuf, - size_t len, loff_t *offp) -{ - struct seq_file *m = file->private_data; - struct drm_i915_private *dev_priv = m->private; - struct intel_hotplug *hotplug = &dev_priv->display.hotplug; - char *newline; - char tmp[16]; - int i; - bool new_state; - - if (len >= sizeof(tmp)) - return -EINVAL; - - if (copy_from_user(tmp, ubuf, len)) - return -EFAULT; - - tmp[len] = '\0'; - - /* Strip newline, if any */ - newline = strchr(tmp, '\n'); - if (newline) - *newline = '\0'; - - /* Reset to the "default" state for this system */ - if (strcmp(tmp, "reset") == 0) - new_state = !HAS_DP_MST(dev_priv); - else if (kstrtobool(tmp, &new_state) != 0) - return -EINVAL; - - drm_dbg_kms(&dev_priv->drm, "%sabling HPD short storm detection\n", - new_state ? "En" : "Dis"); - - spin_lock_irq(&dev_priv->irq_lock); - hotplug->hpd_short_storm_enabled = new_state; - /* Reset the HPD storm stats so we don't accidentally trigger a storm */ - for_each_hpd_pin(i) - hotplug->stats[i].count = 0; - spin_unlock_irq(&dev_priv->irq_lock); - - /* Re-enable hpd immediately if we were in an irq storm */ - flush_delayed_work(&dev_priv->display.hotplug.reenable_work); - - return len; -} - -static const struct file_operations i915_hpd_short_storm_ctl_fops = { - .owner = THIS_MODULE, - .open = i915_hpd_short_storm_ctl_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, - .write = i915_hpd_short_storm_ctl_write, -}; - -static int i915_drrs_ctl_set(void *data, u64 val) -{ - struct drm_i915_private *dev_priv = data; - struct drm_device *dev = &dev_priv->drm; - struct intel_crtc *crtc; - - for_each_intel_crtc(dev, crtc) { - struct intel_crtc_state *crtc_state; - struct drm_crtc_commit *commit; - int ret; - - ret = drm_modeset_lock_single_interruptible(&crtc->base.mutex); - if (ret) - return ret; - - crtc_state = to_intel_crtc_state(crtc->base.state); - - if (!crtc_state->hw.active || - !crtc_state->has_drrs) - goto out; - - commit = crtc_state->uapi.commit; - if (commit) { - ret = wait_for_completion_interruptible(&commit->hw_done); - if (ret) - goto out; - } - - drm_dbg(&dev_priv->drm, - "Manually %sactivating DRRS\n", val ? "" : "de"); - - if (val) - intel_drrs_activate(crtc_state); - else - intel_drrs_deactivate(crtc_state); - -out: - drm_modeset_unlock(&crtc->base.mutex); - if (ret) - return ret; - } - - return 0; -} - -DEFINE_SIMPLE_ATTRIBUTE(i915_drrs_ctl_fops, NULL, i915_drrs_ctl_set, "%llu\n"); - static ssize_t i915_fifo_underrun_reset_write(struct file *filp, const char __user *ubuf, @@ -1776,7 +1520,6 @@ i915_fifo_underrun_reset_write(struct file *filp, { struct drm_i915_private *dev_priv = filp->private_data; struct intel_crtc *crtc; - struct drm_device *dev = &dev_priv->drm; int ret; bool reset; @@ -1787,7 +1530,7 @@ i915_fifo_underrun_reset_write(struct file *filp, if (!reset) return cnt; - for_each_intel_crtc(dev, crtc) { + for_each_intel_crtc(&dev_priv->drm, crtc) { struct drm_crtc_commit *commit; struct intel_crtc_state *crtc_state; @@ -1842,7 +1585,6 @@ static const struct drm_info_list intel_display_debugfs_list[] = { {"i915_shared_dplls_info", i915_shared_dplls_info, 0}, {"i915_dp_mst_info", i915_dp_mst_info, 0}, {"i915_ddb_info", i915_ddb_info, 0}, - {"i915_drrs_status", i915_drrs_status, 0}, {"i915_lpsp_status", i915_lpsp_status, 0}, }; @@ -1857,9 +1599,6 @@ static const struct { {"i915_dp_test_data", &i915_displayport_test_data_fops}, {"i915_dp_test_type", &i915_displayport_test_type_fops}, {"i915_dp_test_active", &i915_displayport_test_active_fops}, - {"i915_hpd_storm_ctl", &i915_hpd_storm_ctl_fops}, - {"i915_hpd_short_storm_ctl", &i915_hpd_short_storm_ctl_fops}, - {"i915_drrs_ctl", &i915_drrs_ctl_fops}, {"i915_edp_psr_debug", &i915_edp_psr_debug_fops}, }; @@ -1882,6 +1621,7 @@ void intel_display_debugfs_register(struct drm_i915_private *i915) intel_dmc_debugfs_register(i915); intel_fbc_debugfs_register(i915); + intel_hpd_debugfs_register(i915); skl_watermark_ipc_debugfs_register(i915); } @@ -2195,6 +1935,8 @@ void intel_connector_debugfs_add(struct intel_connector *intel_connector) if (!root) return; + intel_drrs_connector_debugfs_add(intel_connector); + if (connector->connector_type == DRM_MODE_CONNECTOR_eDP) { debugfs_create_file("i915_panel_timings", S_IRUGO, root, connector, &i915_panel_fops); @@ -2247,6 +1989,7 @@ void intel_crtc_debugfs_add(struct drm_crtc *crtc) return; crtc_updates_add(crtc); + intel_drrs_crtc_debugfs_add(to_intel_crtc(crtc)); intel_fbc_crtc_debugfs_add(to_intel_crtc(crtc)); debugfs_create_file("i915_current_bpc", 0444, crtc->debugfs_entry, crtc, diff --git a/drivers/gpu/drm/i915/display/intel_display_power.c b/drivers/gpu/drm/i915/display/intel_display_power.c index 1a63da28f330..3adba64937de 100644 --- a/drivers/gpu/drm/i915/display/intel_display_power.c +++ b/drivers/gpu/drm/i915/display/intel_display_power.c @@ -129,6 +129,18 @@ intel_display_power_domain_str(enum intel_display_power_domain domain) return "AUDIO_MMIO"; case POWER_DOMAIN_AUDIO_PLAYBACK: return "AUDIO_PLAYBACK"; + case POWER_DOMAIN_AUX_IO_A: + return "AUX_IO_A"; + case POWER_DOMAIN_AUX_IO_B: + return "AUX_IO_B"; + case POWER_DOMAIN_AUX_IO_C: + return "AUX_IO_C"; + case POWER_DOMAIN_AUX_IO_D: + return "AUX_IO_D"; + case POWER_DOMAIN_AUX_IO_E: + return "AUX_IO_E"; + case POWER_DOMAIN_AUX_IO_F: + return "AUX_IO_F"; case POWER_DOMAIN_AUX_A: return "AUX_A"; case POWER_DOMAIN_AUX_B: @@ -153,8 +165,6 @@ intel_display_power_domain_str(enum intel_display_power_domain domain) return "AUX_USBC5"; case POWER_DOMAIN_AUX_USBC6: return "AUX_USBC6"; - case POWER_DOMAIN_AUX_IO_A: - return "AUX_IO_A"; case POWER_DOMAIN_AUX_TBT1: return "AUX_TBT1"; case POWER_DOMAIN_AUX_TBT2: @@ -1148,10 +1158,9 @@ static void hsw_assert_cdclk(struct drm_i915_private *dev_priv) static void assert_can_disable_lcpll(struct drm_i915_private *dev_priv) { - struct drm_device *dev = &dev_priv->drm; struct intel_crtc *crtc; - for_each_intel_crtc(dev, crtc) + for_each_intel_crtc(&dev_priv->drm, crtc) I915_STATE_WARN(crtc->active, "CRTC for pipe %c enabled\n", pipe_name(crtc->pipe)); @@ -2290,6 +2299,7 @@ struct intel_ddi_port_domains { enum intel_display_power_domain ddi_lanes; enum intel_display_power_domain ddi_io; + enum intel_display_power_domain aux_io; enum intel_display_power_domain aux_legacy_usbc; enum intel_display_power_domain aux_tbt; }; @@ -2304,6 +2314,7 @@ i9xx_port_domains[] = { .ddi_lanes = POWER_DOMAIN_PORT_DDI_LANES_A, .ddi_io = POWER_DOMAIN_PORT_DDI_IO_A, + .aux_io = POWER_DOMAIN_AUX_IO_A, .aux_legacy_usbc = POWER_DOMAIN_AUX_A, .aux_tbt = POWER_DOMAIN_INVALID, }, @@ -2319,6 +2330,7 @@ d11_port_domains[] = { .ddi_lanes = POWER_DOMAIN_PORT_DDI_LANES_A, .ddi_io = POWER_DOMAIN_PORT_DDI_IO_A, + .aux_io = POWER_DOMAIN_AUX_IO_A, .aux_legacy_usbc = POWER_DOMAIN_AUX_A, .aux_tbt = POWER_DOMAIN_INVALID, }, { @@ -2329,6 +2341,7 @@ d11_port_domains[] = { .ddi_lanes = POWER_DOMAIN_PORT_DDI_LANES_C, .ddi_io = POWER_DOMAIN_PORT_DDI_IO_C, + .aux_io = POWER_DOMAIN_AUX_IO_C, .aux_legacy_usbc = POWER_DOMAIN_AUX_C, .aux_tbt = POWER_DOMAIN_AUX_TBT1, }, @@ -2344,6 +2357,7 @@ d12_port_domains[] = { .ddi_lanes = POWER_DOMAIN_PORT_DDI_LANES_A, .ddi_io = POWER_DOMAIN_PORT_DDI_IO_A, + .aux_io = POWER_DOMAIN_AUX_IO_A, .aux_legacy_usbc = POWER_DOMAIN_AUX_A, .aux_tbt = POWER_DOMAIN_INVALID, }, { @@ -2354,6 +2368,7 @@ d12_port_domains[] = { .ddi_lanes = POWER_DOMAIN_PORT_DDI_LANES_TC1, .ddi_io = POWER_DOMAIN_PORT_DDI_IO_TC1, + .aux_io = POWER_DOMAIN_INVALID, .aux_legacy_usbc = POWER_DOMAIN_AUX_USBC1, .aux_tbt = POWER_DOMAIN_AUX_TBT1, }, @@ -2369,6 +2384,7 @@ d13_port_domains[] = { .ddi_lanes = POWER_DOMAIN_PORT_DDI_LANES_A, .ddi_io = POWER_DOMAIN_PORT_DDI_IO_A, + .aux_io = POWER_DOMAIN_AUX_IO_A, .aux_legacy_usbc = POWER_DOMAIN_AUX_A, .aux_tbt = POWER_DOMAIN_INVALID, }, { @@ -2379,6 +2395,7 @@ d13_port_domains[] = { .ddi_lanes = POWER_DOMAIN_PORT_DDI_LANES_TC1, .ddi_io = POWER_DOMAIN_PORT_DDI_IO_TC1, + .aux_io = POWER_DOMAIN_INVALID, .aux_legacy_usbc = POWER_DOMAIN_AUX_USBC1, .aux_tbt = POWER_DOMAIN_AUX_TBT1, }, { @@ -2389,6 +2406,7 @@ d13_port_domains[] = { .ddi_lanes = POWER_DOMAIN_PORT_DDI_LANES_D, .ddi_io = POWER_DOMAIN_PORT_DDI_IO_D, + .aux_io = POWER_DOMAIN_AUX_IO_D, .aux_legacy_usbc = POWER_DOMAIN_AUX_D, .aux_tbt = POWER_DOMAIN_INVALID, }, @@ -2467,6 +2485,17 @@ intel_port_domains_for_aux_ch(struct drm_i915_private *i915, enum aux_ch aux_ch) } enum intel_display_power_domain +intel_display_power_aux_io_domain(struct drm_i915_private *i915, enum aux_ch aux_ch) +{ + const struct intel_ddi_port_domains *domains = intel_port_domains_for_aux_ch(i915, aux_ch); + + if (drm_WARN_ON(&i915->drm, !domains || domains->aux_io == POWER_DOMAIN_INVALID)) + return POWER_DOMAIN_AUX_IO_A; + + return domains->aux_io + (int)(aux_ch - domains->aux_ch_start); +} + +enum intel_display_power_domain intel_display_power_legacy_aux_domain(struct drm_i915_private *i915, enum aux_ch aux_ch) { const struct intel_ddi_port_domains *domains = intel_port_domains_for_aux_ch(i915, aux_ch); diff --git a/drivers/gpu/drm/i915/display/intel_display_power.h b/drivers/gpu/drm/i915/display/intel_display_power.h index 7136ea3f233e..2154d900b1aa 100644 --- a/drivers/gpu/drm/i915/display/intel_display_power.h +++ b/drivers/gpu/drm/i915/display/intel_display_power.h @@ -6,11 +6,12 @@ #ifndef __INTEL_DISPLAY_POWER_H__ #define __INTEL_DISPLAY_POWER_H__ -#include "intel_runtime_pm.h" +#include "intel_wakeref.h" enum aux_ch; enum dpio_channel; enum dpio_phy; +enum i915_drm_suspend_mode; enum port; struct drm_i915_private; struct i915_power_well; @@ -76,6 +77,14 @@ enum intel_display_power_domain { POWER_DOMAIN_VGA, POWER_DOMAIN_AUDIO_MMIO, POWER_DOMAIN_AUDIO_PLAYBACK, + + POWER_DOMAIN_AUX_IO_A, + POWER_DOMAIN_AUX_IO_B, + POWER_DOMAIN_AUX_IO_C, + POWER_DOMAIN_AUX_IO_D, + POWER_DOMAIN_AUX_IO_E, + POWER_DOMAIN_AUX_IO_F, + POWER_DOMAIN_AUX_A, POWER_DOMAIN_AUX_B, POWER_DOMAIN_AUX_C, @@ -90,8 +99,6 @@ enum intel_display_power_domain { POWER_DOMAIN_AUX_USBC5, POWER_DOMAIN_AUX_USBC6, - POWER_DOMAIN_AUX_IO_A, - POWER_DOMAIN_AUX_TBT1, POWER_DOMAIN_AUX_TBT2, POWER_DOMAIN_AUX_TBT3, @@ -249,6 +256,8 @@ intel_display_power_ddi_lanes_domain(struct drm_i915_private *i915, enum port po enum intel_display_power_domain intel_display_power_ddi_io_domain(struct drm_i915_private *i915, enum port port); enum intel_display_power_domain +intel_display_power_aux_io_domain(struct drm_i915_private *i915, enum aux_ch aux_ch); +enum intel_display_power_domain intel_display_power_legacy_aux_domain(struct drm_i915_private *i915, enum aux_ch aux_ch); enum intel_display_power_domain intel_display_power_tbt_aux_domain(struct drm_i915_private *i915, enum aux_ch aux_ch); diff --git a/drivers/gpu/drm/i915/display/intel_display_power_map.c b/drivers/gpu/drm/i915/display/intel_display_power_map.c index dc04afc6cc8f..f5d66ca85b19 100644 --- a/drivers/gpu/drm/i915/display/intel_display_power_map.c +++ b/drivers/gpu/drm/i915/display/intel_display_power_map.c @@ -170,6 +170,8 @@ I915_DECL_PW_DOMAINS(vlv_pwdoms_display, POWER_DOMAIN_VGA, POWER_DOMAIN_AUDIO_MMIO, POWER_DOMAIN_AUDIO_PLAYBACK, + POWER_DOMAIN_AUX_IO_B, + POWER_DOMAIN_AUX_IO_C, POWER_DOMAIN_AUX_B, POWER_DOMAIN_AUX_C, POWER_DOMAIN_GMBUS, @@ -179,6 +181,8 @@ I915_DECL_PW_DOMAINS(vlv_pwdoms_dpio_cmn_bc, POWER_DOMAIN_PORT_DDI_LANES_B, POWER_DOMAIN_PORT_DDI_LANES_C, POWER_DOMAIN_PORT_CRT, + POWER_DOMAIN_AUX_IO_B, + POWER_DOMAIN_AUX_IO_C, POWER_DOMAIN_AUX_B, POWER_DOMAIN_AUX_C, POWER_DOMAIN_INIT); @@ -186,6 +190,8 @@ I915_DECL_PW_DOMAINS(vlv_pwdoms_dpio_cmn_bc, I915_DECL_PW_DOMAINS(vlv_pwdoms_dpio_tx_bc_lanes, POWER_DOMAIN_PORT_DDI_LANES_B, POWER_DOMAIN_PORT_DDI_LANES_C, + POWER_DOMAIN_AUX_IO_B, + POWER_DOMAIN_AUX_IO_C, POWER_DOMAIN_AUX_B, POWER_DOMAIN_AUX_C, POWER_DOMAIN_INIT); @@ -243,6 +249,9 @@ I915_DECL_PW_DOMAINS(chv_pwdoms_display, POWER_DOMAIN_VGA, POWER_DOMAIN_AUDIO_MMIO, POWER_DOMAIN_AUDIO_PLAYBACK, + POWER_DOMAIN_AUX_IO_B, + POWER_DOMAIN_AUX_IO_C, + POWER_DOMAIN_AUX_IO_D, POWER_DOMAIN_AUX_B, POWER_DOMAIN_AUX_C, POWER_DOMAIN_AUX_D, @@ -252,12 +261,15 @@ I915_DECL_PW_DOMAINS(chv_pwdoms_display, I915_DECL_PW_DOMAINS(chv_pwdoms_dpio_cmn_bc, POWER_DOMAIN_PORT_DDI_LANES_B, POWER_DOMAIN_PORT_DDI_LANES_C, + POWER_DOMAIN_AUX_IO_B, + POWER_DOMAIN_AUX_IO_C, POWER_DOMAIN_AUX_B, POWER_DOMAIN_AUX_C, POWER_DOMAIN_INIT); I915_DECL_PW_DOMAINS(chv_pwdoms_dpio_cmn_d, POWER_DOMAIN_PORT_DDI_LANES_D, + POWER_DOMAIN_AUX_IO_D, POWER_DOMAIN_AUX_D, POWER_DOMAIN_INIT); @@ -305,6 +317,9 @@ static const struct i915_power_well_desc_list chv_power_wells[] = { POWER_DOMAIN_VGA, \ POWER_DOMAIN_AUDIO_MMIO, \ POWER_DOMAIN_AUDIO_PLAYBACK, \ + POWER_DOMAIN_AUX_IO_B, \ + POWER_DOMAIN_AUX_IO_C, \ + POWER_DOMAIN_AUX_IO_D, \ POWER_DOMAIN_AUX_B, \ POWER_DOMAIN_AUX_C, \ POWER_DOMAIN_AUX_D @@ -318,6 +333,7 @@ I915_DECL_PW_DOMAINS(skl_pwdoms_dc_off, POWER_DOMAIN_AUX_A, POWER_DOMAIN_MODESET, POWER_DOMAIN_GT_IRQ, + POWER_DOMAIN_DC_OFF, POWER_DOMAIN_INIT); I915_DECL_PW_DOMAINS(skl_pwdoms_ddi_io_a_e, @@ -407,6 +423,8 @@ static const struct i915_power_well_desc_list skl_power_wells[] = { POWER_DOMAIN_VGA, \ POWER_DOMAIN_AUDIO_MMIO, \ POWER_DOMAIN_AUDIO_PLAYBACK, \ + POWER_DOMAIN_AUX_IO_B, \ + POWER_DOMAIN_AUX_IO_C, \ POWER_DOMAIN_AUX_B, \ POWER_DOMAIN_AUX_C @@ -420,16 +438,20 @@ I915_DECL_PW_DOMAINS(bxt_pwdoms_dc_off, POWER_DOMAIN_GMBUS, POWER_DOMAIN_MODESET, POWER_DOMAIN_GT_IRQ, + POWER_DOMAIN_DC_OFF, POWER_DOMAIN_INIT); I915_DECL_PW_DOMAINS(bxt_pwdoms_dpio_cmn_a, POWER_DOMAIN_PORT_DDI_LANES_A, + POWER_DOMAIN_AUX_IO_A, POWER_DOMAIN_AUX_A, POWER_DOMAIN_INIT); I915_DECL_PW_DOMAINS(bxt_pwdoms_dpio_cmn_bc, POWER_DOMAIN_PORT_DDI_LANES_B, POWER_DOMAIN_PORT_DDI_LANES_C, + POWER_DOMAIN_AUX_IO_B, + POWER_DOMAIN_AUX_IO_C, POWER_DOMAIN_AUX_B, POWER_DOMAIN_AUX_C, POWER_DOMAIN_INIT); @@ -483,6 +505,8 @@ static const struct i915_power_well_desc_list bxt_power_wells[] = { POWER_DOMAIN_VGA, \ POWER_DOMAIN_AUDIO_MMIO, \ POWER_DOMAIN_AUDIO_PLAYBACK, \ + POWER_DOMAIN_AUX_IO_B, \ + POWER_DOMAIN_AUX_IO_C, \ POWER_DOMAIN_AUX_B, \ POWER_DOMAIN_AUX_C @@ -496,6 +520,7 @@ I915_DECL_PW_DOMAINS(glk_pwdoms_dc_off, POWER_DOMAIN_GMBUS, POWER_DOMAIN_MODESET, POWER_DOMAIN_GT_IRQ, + POWER_DOMAIN_DC_OFF, POWER_DOMAIN_INIT); I915_DECL_PW_DOMAINS(glk_pwdoms_ddi_io_a, POWER_DOMAIN_PORT_DDI_IO_A); @@ -504,29 +529,34 @@ I915_DECL_PW_DOMAINS(glk_pwdoms_ddi_io_c, POWER_DOMAIN_PORT_DDI_IO_C); I915_DECL_PW_DOMAINS(glk_pwdoms_dpio_cmn_a, POWER_DOMAIN_PORT_DDI_LANES_A, + POWER_DOMAIN_AUX_IO_A, POWER_DOMAIN_AUX_A, POWER_DOMAIN_INIT); I915_DECL_PW_DOMAINS(glk_pwdoms_dpio_cmn_b, POWER_DOMAIN_PORT_DDI_LANES_B, + POWER_DOMAIN_AUX_IO_B, POWER_DOMAIN_AUX_B, POWER_DOMAIN_INIT); I915_DECL_PW_DOMAINS(glk_pwdoms_dpio_cmn_c, POWER_DOMAIN_PORT_DDI_LANES_C, + POWER_DOMAIN_AUX_IO_C, POWER_DOMAIN_AUX_C, POWER_DOMAIN_INIT); I915_DECL_PW_DOMAINS(glk_pwdoms_aux_a, - POWER_DOMAIN_AUX_A, POWER_DOMAIN_AUX_IO_A, + POWER_DOMAIN_AUX_A, POWER_DOMAIN_INIT); I915_DECL_PW_DOMAINS(glk_pwdoms_aux_b, + POWER_DOMAIN_AUX_IO_B, POWER_DOMAIN_AUX_B, POWER_DOMAIN_INIT); I915_DECL_PW_DOMAINS(glk_pwdoms_aux_c, + POWER_DOMAIN_AUX_IO_C, POWER_DOMAIN_AUX_C, POWER_DOMAIN_INIT); @@ -617,6 +647,11 @@ I915_DECL_PW_DOMAINS(icl_pwdoms_pw_4, POWER_DOMAIN_VGA, \ POWER_DOMAIN_AUDIO_MMIO, \ POWER_DOMAIN_AUDIO_PLAYBACK, \ + POWER_DOMAIN_AUX_IO_B, \ + POWER_DOMAIN_AUX_IO_C, \ + POWER_DOMAIN_AUX_IO_D, \ + POWER_DOMAIN_AUX_IO_E, \ + POWER_DOMAIN_AUX_IO_F, \ POWER_DOMAIN_AUX_B, \ POWER_DOMAIN_AUX_C, \ POWER_DOMAIN_AUX_D, \ @@ -658,13 +693,23 @@ I915_DECL_PW_DOMAINS(icl_pwdoms_ddi_io_e, POWER_DOMAIN_PORT_DDI_IO_E); I915_DECL_PW_DOMAINS(icl_pwdoms_ddi_io_f, POWER_DOMAIN_PORT_DDI_IO_F); I915_DECL_PW_DOMAINS(icl_pwdoms_aux_a, - POWER_DOMAIN_AUX_A, - POWER_DOMAIN_AUX_IO_A); -I915_DECL_PW_DOMAINS(icl_pwdoms_aux_b, POWER_DOMAIN_AUX_B); -I915_DECL_PW_DOMAINS(icl_pwdoms_aux_c, POWER_DOMAIN_AUX_C); -I915_DECL_PW_DOMAINS(icl_pwdoms_aux_d, POWER_DOMAIN_AUX_D); -I915_DECL_PW_DOMAINS(icl_pwdoms_aux_e, POWER_DOMAIN_AUX_E); -I915_DECL_PW_DOMAINS(icl_pwdoms_aux_f, POWER_DOMAIN_AUX_F); + POWER_DOMAIN_AUX_IO_A, + POWER_DOMAIN_AUX_A); +I915_DECL_PW_DOMAINS(icl_pwdoms_aux_b, + POWER_DOMAIN_AUX_IO_B, + POWER_DOMAIN_AUX_B); +I915_DECL_PW_DOMAINS(icl_pwdoms_aux_c, + POWER_DOMAIN_AUX_IO_C, + POWER_DOMAIN_AUX_C); +I915_DECL_PW_DOMAINS(icl_pwdoms_aux_d, + POWER_DOMAIN_AUX_IO_D, + POWER_DOMAIN_AUX_D); +I915_DECL_PW_DOMAINS(icl_pwdoms_aux_e, + POWER_DOMAIN_AUX_IO_E, + POWER_DOMAIN_AUX_E); +I915_DECL_PW_DOMAINS(icl_pwdoms_aux_f, + POWER_DOMAIN_AUX_IO_F, + POWER_DOMAIN_AUX_F); I915_DECL_PW_DOMAINS(icl_pwdoms_aux_tbt1, POWER_DOMAIN_AUX_TBT1); I915_DECL_PW_DOMAINS(icl_pwdoms_aux_tbt2, POWER_DOMAIN_AUX_TBT2); I915_DECL_PW_DOMAINS(icl_pwdoms_aux_tbt3, POWER_DOMAIN_AUX_TBT3); @@ -816,6 +861,7 @@ I915_DECL_PW_DOMAINS(tgl_pwdoms_dc_off, POWER_DOMAIN_AUX_B, POWER_DOMAIN_AUX_C, POWER_DOMAIN_MODESET, + POWER_DOMAIN_DC_OFF, POWER_DOMAIN_INIT); I915_DECL_PW_DOMAINS(tgl_pwdoms_ddi_io_tc1, POWER_DOMAIN_PORT_DDI_IO_TC1); @@ -1012,6 +1058,7 @@ I915_DECL_PW_DOMAINS(rkl_pwdoms_dc_off, POWER_DOMAIN_AUX_A, POWER_DOMAIN_AUX_B, POWER_DOMAIN_MODESET, + POWER_DOMAIN_DC_OFF, POWER_DOMAIN_INIT); static const struct i915_power_well_desc rkl_power_wells_main[] = { @@ -1094,6 +1141,7 @@ I915_DECL_PW_DOMAINS(dg1_pwdoms_dc_off, POWER_DOMAIN_AUX_A, POWER_DOMAIN_AUX_B, POWER_DOMAIN_MODESET, + POWER_DOMAIN_DC_OFF, POWER_DOMAIN_INIT); I915_DECL_PW_DOMAINS(dg1_pwdoms_pw_2, @@ -1215,6 +1263,9 @@ I915_DECL_PW_DOMAINS(xelpd_pwdoms_pw_a, POWER_DOMAIN_PORT_DDI_LANES_TC4, \ POWER_DOMAIN_VGA, \ POWER_DOMAIN_AUDIO_PLAYBACK, \ + POWER_DOMAIN_AUX_IO_C, \ + POWER_DOMAIN_AUX_IO_D, \ + POWER_DOMAIN_AUX_IO_E, \ POWER_DOMAIN_AUX_C, \ POWER_DOMAIN_AUX_D, \ POWER_DOMAIN_AUX_E, \ @@ -1255,6 +1306,7 @@ I915_DECL_PW_DOMAINS(xelpd_pwdoms_dc_off, POWER_DOMAIN_AUX_A, POWER_DOMAIN_AUX_B, POWER_DOMAIN_MODESET, + POWER_DOMAIN_DC_OFF, POWER_DOMAIN_INIT); static const struct i915_power_well_desc xelpd_power_wells_main[] = { @@ -1376,6 +1428,7 @@ I915_DECL_PW_DOMAINS(xelpdp_pwdoms_dc_off, POWER_DOMAIN_MODESET, POWER_DOMAIN_AUX_A, POWER_DOMAIN_AUX_B, + POWER_DOMAIN_DC_OFF, POWER_DOMAIN_INIT); I915_DECL_PW_DOMAINS(xelpdp_pwdoms_aux_tc1, diff --git a/drivers/gpu/drm/i915/display/intel_display_power_well.c b/drivers/gpu/drm/i915/display/intel_display_power_well.c index 1d18eee56253..8710dd41ffd4 100644 --- a/drivers/gpu/drm/i915/display/intel_display_power_well.c +++ b/drivers/gpu/drm/i915/display/intel_display_power_well.c @@ -13,6 +13,7 @@ #include "intel_display_power_well.h" #include "intel_display_types.h" #include "intel_dkl_phy.h" +#include "intel_dkl_phy_regs.h" #include "intel_dmc.h" #include "intel_dpio_phy.h" #include "intel_dpll.h" @@ -531,7 +532,7 @@ icl_tc_phy_aux_power_well_enable(struct drm_i915_private *dev_priv, tc_port = TGL_AUX_PW_TO_TC_PORT(i915_power_well_instance(power_well)->hsw.idx); - if (wait_for(intel_dkl_phy_read(dev_priv, DKL_CMN_UC_DW_27(tc_port), 2) & + if (wait_for(intel_dkl_phy_read(dev_priv, DKL_CMN_UC_DW_27(tc_port)) & DKL_CMN_UC_DW27_UC_HEALTH, 1)) drm_warn(&dev_priv->drm, "Timeout waiting TC uC health\n"); diff --git a/drivers/gpu/drm/i915/display/intel_display_power_well.h b/drivers/gpu/drm/i915/display/intel_display_power_well.h index e13b521e322a..ba7cb977e7c7 100644 --- a/drivers/gpu/drm/i915/display/intel_display_power_well.h +++ b/drivers/gpu/drm/i915/display/intel_display_power_well.h @@ -7,8 +7,8 @@ #include <linux/types.h> -#include "intel_display.h" #include "intel_display_power.h" +#include "intel_dpio_phy.h" struct drm_i915_private; struct i915_power_well; diff --git a/drivers/gpu/drm/i915/display/intel_display_reg_defs.h b/drivers/gpu/drm/i915/display/intel_display_reg_defs.h new file mode 100644 index 000000000000..02605418ff08 --- /dev/null +++ b/drivers/gpu/drm/i915/display/intel_display_reg_defs.h @@ -0,0 +1,53 @@ +/* SPDX-License-Identifier: MIT */ +/* + * Copyright © 2022 Intel Corporation + */ + +#ifndef __INTEL_DISPLAY_REG_DEFS_H__ +#define __INTEL_DISPLAY_REG_DEFS_H__ + +#include "i915_reg_defs.h" + +#define DISPLAY_MMIO_BASE(dev_priv) (INTEL_INFO(dev_priv)->display.mmio_offset) + +#define VLV_DISPLAY_BASE 0x180000 + +/* + * Named helper wrappers around _PICK_EVEN() and _PICK(). + */ +#define _PIPE(pipe, a, b) _PICK_EVEN(pipe, a, b) +#define _PLANE(plane, a, b) _PICK_EVEN(plane, a, b) +#define _TRANS(tran, a, b) _PICK_EVEN(tran, a, b) +#define _PORT(port, a, b) _PICK_EVEN(port, a, b) +#define _PLL(pll, a, b) _PICK_EVEN(pll, a, b) +#define _PHY(phy, a, b) _PICK_EVEN(phy, a, b) + +#define _MMIO_PIPE(pipe, a, b) _MMIO(_PIPE(pipe, a, b)) +#define _MMIO_PLANE(plane, a, b) _MMIO(_PLANE(plane, a, b)) +#define _MMIO_TRANS(tran, a, b) _MMIO(_TRANS(tran, a, b)) +#define _MMIO_PORT(port, a, b) _MMIO(_PORT(port, a, b)) +#define _MMIO_PLL(pll, a, b) _MMIO(_PLL(pll, a, b)) +#define _MMIO_PHY(phy, a, b) _MMIO(_PHY(phy, a, b)) + +#define _PHY3(phy, ...) _PICK(phy, __VA_ARGS__) + +#define _MMIO_PIPE3(pipe, a, b, c) _MMIO(_PICK(pipe, a, b, c)) +#define _MMIO_PORT3(pipe, a, b, c) _MMIO(_PICK(pipe, a, b, c)) +#define _MMIO_PHY3(phy, a, b, c) _MMIO(_PHY3(phy, a, b, c)) +#define _MMIO_PLL3(pll, ...) _MMIO(_PICK(pll, __VA_ARGS__)) + +/* + * Device info offset array based helpers for groups of registers with unevenly + * spaced base offsets. + */ +#define _MMIO_PIPE2(pipe, reg) _MMIO(INTEL_INFO(dev_priv)->display.pipe_offsets[(pipe)] - \ + INTEL_INFO(dev_priv)->display.pipe_offsets[PIPE_A] + \ + DISPLAY_MMIO_BASE(dev_priv) + (reg)) +#define _MMIO_TRANS2(tran, reg) _MMIO(INTEL_INFO(dev_priv)->display.trans_offsets[(tran)] - \ + INTEL_INFO(dev_priv)->display.trans_offsets[TRANSCODER_A] + \ + DISPLAY_MMIO_BASE(dev_priv) + (reg)) +#define _MMIO_CURSOR2(pipe, reg) _MMIO(INTEL_INFO(dev_priv)->display.cursor_offsets[(pipe)] - \ + INTEL_INFO(dev_priv)->display.cursor_offsets[PIPE_A] + \ + DISPLAY_MMIO_BASE(dev_priv) + (reg)) + +#endif /* __INTEL_DISPLAY_REG_DEFS_H__ */ diff --git a/drivers/gpu/drm/i915/display/intel_display_trace.h b/drivers/gpu/drm/i915/display/intel_display_trace.h index 2dd5a4b7f5d8..725aba3fa531 100644 --- a/drivers/gpu/drm/i915/display/intel_display_trace.h +++ b/drivers/gpu/drm/i915/display/intel_display_trace.h @@ -18,11 +18,15 @@ #include "intel_crtc.h" #include "intel_display_types.h" +#define __dev_name_i915(i915) dev_name((i915)->drm.dev) +#define __dev_name_kms(obj) dev_name((obj)->base.dev->dev) + TRACE_EVENT(intel_pipe_enable, TP_PROTO(struct intel_crtc *crtc), TP_ARGS(crtc), TP_STRUCT__entry( + __string(dev, __dev_name_kms(crtc)) __array(u32, frame, 3) __array(u32, scanline, 3) __field(enum pipe, pipe) @@ -30,6 +34,7 @@ TRACE_EVENT(intel_pipe_enable, TP_fast_assign( struct drm_i915_private *dev_priv = to_i915(crtc->base.dev); struct intel_crtc *it__; + __assign_str(dev, __dev_name_kms(crtc)); for_each_intel_crtc(&dev_priv->drm, it__) { __entry->frame[it__->pipe] = intel_crtc_get_vblank_counter(it__); __entry->scanline[it__->pipe] = intel_get_crtc_scanline(it__); @@ -37,8 +42,8 @@ TRACE_EVENT(intel_pipe_enable, __entry->pipe = crtc->pipe; ), - TP_printk("pipe %c enable, pipe A: frame=%u, scanline=%u, pipe B: frame=%u, scanline=%u, pipe C: frame=%u, scanline=%u", - pipe_name(__entry->pipe), + TP_printk("dev %s, pipe %c enable, pipe A: frame=%u, scanline=%u, pipe B: frame=%u, scanline=%u, pipe C: frame=%u, scanline=%u", + __get_str(dev), pipe_name(__entry->pipe), __entry->frame[PIPE_A], __entry->scanline[PIPE_A], __entry->frame[PIPE_B], __entry->scanline[PIPE_B], __entry->frame[PIPE_C], __entry->scanline[PIPE_C]) @@ -49,6 +54,7 @@ TRACE_EVENT(intel_pipe_disable, TP_ARGS(crtc), TP_STRUCT__entry( + __string(dev, __dev_name_kms(crtc)) __array(u32, frame, 3) __array(u32, scanline, 3) __field(enum pipe, pipe) @@ -57,6 +63,7 @@ TRACE_EVENT(intel_pipe_disable, TP_fast_assign( struct drm_i915_private *dev_priv = to_i915(crtc->base.dev); struct intel_crtc *it__; + __assign_str(dev, __dev_name_kms(crtc)); for_each_intel_crtc(&dev_priv->drm, it__) { __entry->frame[it__->pipe] = intel_crtc_get_vblank_counter(it__); __entry->scanline[it__->pipe] = intel_get_crtc_scanline(it__); @@ -64,8 +71,8 @@ TRACE_EVENT(intel_pipe_disable, __entry->pipe = crtc->pipe; ), - TP_printk("pipe %c disable, pipe A: frame=%u, scanline=%u, pipe B: frame=%u, scanline=%u, pipe C: frame=%u, scanline=%u", - pipe_name(__entry->pipe), + TP_printk("dev %s, pipe %c disable, pipe A: frame=%u, scanline=%u, pipe B: frame=%u, scanline=%u, pipe C: frame=%u, scanline=%u", + __get_str(dev), pipe_name(__entry->pipe), __entry->frame[PIPE_A], __entry->scanline[PIPE_A], __entry->frame[PIPE_B], __entry->scanline[PIPE_B], __entry->frame[PIPE_C], __entry->scanline[PIPE_C]) @@ -76,6 +83,7 @@ TRACE_EVENT(intel_pipe_crc, TP_ARGS(crtc, crcs), TP_STRUCT__entry( + __string(dev, __dev_name_kms(crtc)) __field(enum pipe, pipe) __field(u32, frame) __field(u32, scanline) @@ -83,16 +91,19 @@ TRACE_EVENT(intel_pipe_crc, ), TP_fast_assign( + __assign_str(dev, __dev_name_kms(crtc)); __entry->pipe = crtc->pipe; __entry->frame = intel_crtc_get_vblank_counter(crtc); __entry->scanline = intel_get_crtc_scanline(crtc); memcpy(__entry->crcs, crcs, sizeof(__entry->crcs)); ), - TP_printk("pipe %c, frame=%u, scanline=%u crc=%08x %08x %08x %08x %08x", - pipe_name(__entry->pipe), __entry->frame, __entry->scanline, - __entry->crcs[0], __entry->crcs[1], __entry->crcs[2], - __entry->crcs[3], __entry->crcs[4]) + TP_printk("dev %s, pipe %c, frame=%u, scanline=%u crc=%08x %08x %08x %08x %08x", + __get_str(dev), pipe_name(__entry->pipe), + __entry->frame, __entry->scanline, + __entry->crcs[0], __entry->crcs[1], + __entry->crcs[2], __entry->crcs[3], + __entry->crcs[4]) ); TRACE_EVENT(intel_cpu_fifo_underrun, @@ -100,6 +111,7 @@ TRACE_EVENT(intel_cpu_fifo_underrun, TP_ARGS(dev_priv, pipe), TP_STRUCT__entry( + __string(dev, __dev_name_i915(dev_priv)) __field(enum pipe, pipe) __field(u32, frame) __field(u32, scanline) @@ -107,13 +119,14 @@ TRACE_EVENT(intel_cpu_fifo_underrun, TP_fast_assign( struct intel_crtc *crtc = intel_crtc_for_pipe(dev_priv, pipe); + __assign_str(dev, __dev_name_kms(crtc)); __entry->pipe = pipe; __entry->frame = intel_crtc_get_vblank_counter(crtc); __entry->scanline = intel_get_crtc_scanline(crtc); ), - TP_printk("pipe %c, frame=%u, scanline=%u", - pipe_name(__entry->pipe), + TP_printk("dev %s, pipe %c, frame=%u, scanline=%u", + __get_str(dev), pipe_name(__entry->pipe), __entry->frame, __entry->scanline) ); @@ -122,6 +135,7 @@ TRACE_EVENT(intel_pch_fifo_underrun, TP_ARGS(dev_priv, pch_transcoder), TP_STRUCT__entry( + __string(dev, __dev_name_i915(dev_priv)) __field(enum pipe, pipe) __field(u32, frame) __field(u32, scanline) @@ -130,13 +144,14 @@ TRACE_EVENT(intel_pch_fifo_underrun, TP_fast_assign( enum pipe pipe = pch_transcoder; struct intel_crtc *crtc = intel_crtc_for_pipe(dev_priv, pipe); + __assign_str(dev, __dev_name_i915(dev_priv)); __entry->pipe = pipe; __entry->frame = intel_crtc_get_vblank_counter(crtc); __entry->scanline = intel_get_crtc_scanline(crtc); ), - TP_printk("pch transcoder %c, frame=%u, scanline=%u", - pipe_name(__entry->pipe), + TP_printk("dev %s, pch transcoder %c, frame=%u, scanline=%u", + __get_str(dev), pipe_name(__entry->pipe), __entry->frame, __entry->scanline) ); @@ -145,6 +160,7 @@ TRACE_EVENT(intel_memory_cxsr, TP_ARGS(dev_priv, old, new), TP_STRUCT__entry( + __string(dev, __dev_name_i915(dev_priv)) __array(u32, frame, 3) __array(u32, scanline, 3) __field(bool, old) @@ -153,6 +169,7 @@ TRACE_EVENT(intel_memory_cxsr, TP_fast_assign( struct intel_crtc *crtc; + __assign_str(dev, __dev_name_i915(dev_priv)); for_each_intel_crtc(&dev_priv->drm, crtc) { __entry->frame[crtc->pipe] = intel_crtc_get_vblank_counter(crtc); __entry->scanline[crtc->pipe] = intel_get_crtc_scanline(crtc); @@ -161,8 +178,8 @@ TRACE_EVENT(intel_memory_cxsr, __entry->new = new; ), - TP_printk("%s->%s, pipe A: frame=%u, scanline=%u, pipe B: frame=%u, scanline=%u, pipe C: frame=%u, scanline=%u", - str_on_off(__entry->old), str_on_off(__entry->new), + TP_printk("dev %s, cxsr %s->%s, pipe A: frame=%u, scanline=%u, pipe B: frame=%u, scanline=%u, pipe C: frame=%u, scanline=%u", + __get_str(dev), str_on_off(__entry->old), str_on_off(__entry->new), __entry->frame[PIPE_A], __entry->scanline[PIPE_A], __entry->frame[PIPE_B], __entry->scanline[PIPE_B], __entry->frame[PIPE_C], __entry->scanline[PIPE_C]) @@ -173,6 +190,7 @@ TRACE_EVENT(g4x_wm, TP_ARGS(crtc, wm), TP_STRUCT__entry( + __string(dev, __dev_name_kms(crtc)) __field(enum pipe, pipe) __field(u32, frame) __field(u32, scanline) @@ -191,6 +209,7 @@ TRACE_EVENT(g4x_wm, ), TP_fast_assign( + __assign_str(dev, __dev_name_kms(crtc)); __entry->pipe = crtc->pipe; __entry->frame = intel_crtc_get_vblank_counter(crtc); __entry->scanline = intel_get_crtc_scanline(crtc); @@ -208,8 +227,9 @@ TRACE_EVENT(g4x_wm, __entry->fbc = wm->fbc_en; ), - TP_printk("pipe %c, frame=%u, scanline=%u, wm %d/%d/%d, sr %s/%d/%d/%d, hpll %s/%d/%d/%d, fbc %s", - pipe_name(__entry->pipe), __entry->frame, __entry->scanline, + TP_printk("dev %s, pipe %c, frame=%u, scanline=%u, wm %d/%d/%d, sr %s/%d/%d/%d, hpll %s/%d/%d/%d, fbc %s", + __get_str(dev), pipe_name(__entry->pipe), + __entry->frame, __entry->scanline, __entry->primary, __entry->sprite, __entry->cursor, str_yes_no(__entry->cxsr), __entry->sr_plane, __entry->sr_cursor, __entry->sr_fbc, str_yes_no(__entry->hpll), __entry->hpll_plane, __entry->hpll_cursor, __entry->hpll_fbc, @@ -221,6 +241,7 @@ TRACE_EVENT(vlv_wm, TP_ARGS(crtc, wm), TP_STRUCT__entry( + __string(dev, __dev_name_kms(crtc)) __field(enum pipe, pipe) __field(u32, frame) __field(u32, scanline) @@ -235,6 +256,7 @@ TRACE_EVENT(vlv_wm, ), TP_fast_assign( + __assign_str(dev, __dev_name_kms(crtc)); __entry->pipe = crtc->pipe; __entry->frame = intel_crtc_get_vblank_counter(crtc); __entry->scanline = intel_get_crtc_scanline(crtc); @@ -248,9 +270,10 @@ TRACE_EVENT(vlv_wm, __entry->sr_cursor = wm->sr.cursor; ), - TP_printk("pipe %c, frame=%u, scanline=%u, level=%d, cxsr=%d, wm %d/%d/%d/%d, sr %d/%d", - pipe_name(__entry->pipe), __entry->frame, - __entry->scanline, __entry->level, __entry->cxsr, + TP_printk("dev %s, pipe %c, frame=%u, scanline=%u, level=%d, cxsr=%d, wm %d/%d/%d/%d, sr %d/%d", + __get_str(dev), pipe_name(__entry->pipe), + __entry->frame, __entry->scanline, + __entry->level, __entry->cxsr, __entry->primary, __entry->sprite0, __entry->sprite1, __entry->cursor, __entry->sr_plane, __entry->sr_cursor) ); @@ -260,6 +283,7 @@ TRACE_EVENT(vlv_fifo_size, TP_ARGS(crtc, sprite0_start, sprite1_start, fifo_size), TP_STRUCT__entry( + __string(dev, __dev_name_kms(crtc)) __field(enum pipe, pipe) __field(u32, frame) __field(u32, scanline) @@ -269,6 +293,7 @@ TRACE_EVENT(vlv_fifo_size, ), TP_fast_assign( + __assign_str(dev, __dev_name_kms(crtc)); __entry->pipe = crtc->pipe; __entry->frame = intel_crtc_get_vblank_counter(crtc); __entry->scanline = intel_get_crtc_scanline(crtc); @@ -277,90 +302,96 @@ TRACE_EVENT(vlv_fifo_size, __entry->fifo_size = fifo_size; ), - TP_printk("pipe %c, frame=%u, scanline=%u, %d/%d/%d", - pipe_name(__entry->pipe), __entry->frame, - __entry->scanline, __entry->sprite0_start, - __entry->sprite1_start, __entry->fifo_size) + TP_printk("dev %s, pipe %c, frame=%u, scanline=%u, %d/%d/%d", + __get_str(dev), pipe_name(__entry->pipe), + __entry->frame, __entry->scanline, + __entry->sprite0_start, __entry->sprite1_start, __entry->fifo_size) ); TRACE_EVENT(intel_plane_update_noarm, - TP_PROTO(struct drm_plane *plane, struct intel_crtc *crtc), + TP_PROTO(struct intel_plane *plane, struct intel_crtc *crtc), TP_ARGS(plane, crtc), TP_STRUCT__entry( + __string(dev, __dev_name_kms(plane)) __field(enum pipe, pipe) __field(u32, frame) __field(u32, scanline) __array(int, src, 4) __array(int, dst, 4) - __string(name, plane->name) + __string(name, plane->base.name) ), TP_fast_assign( - __assign_str(name, plane->name); + __assign_str(dev, __dev_name_kms(plane)); + __assign_str(name, plane->base.name); __entry->pipe = crtc->pipe; __entry->frame = intel_crtc_get_vblank_counter(crtc); __entry->scanline = intel_get_crtc_scanline(crtc); - memcpy(__entry->src, &plane->state->src, sizeof(__entry->src)); - memcpy(__entry->dst, &plane->state->dst, sizeof(__entry->dst)); + memcpy(__entry->src, &plane->base.state->src, sizeof(__entry->src)); + memcpy(__entry->dst, &plane->base.state->dst, sizeof(__entry->dst)); ), - TP_printk("pipe %c, plane %s, frame=%u, scanline=%u, " DRM_RECT_FP_FMT " -> " DRM_RECT_FMT, - pipe_name(__entry->pipe), __get_str(name), + TP_printk("dev %s, pipe %c, plane %s, frame=%u, scanline=%u, " DRM_RECT_FP_FMT " -> " DRM_RECT_FMT, + __get_str(dev), pipe_name(__entry->pipe), __get_str(name), __entry->frame, __entry->scanline, DRM_RECT_FP_ARG((const struct drm_rect *)__entry->src), DRM_RECT_ARG((const struct drm_rect *)__entry->dst)) ); TRACE_EVENT(intel_plane_update_arm, - TP_PROTO(struct drm_plane *plane, struct intel_crtc *crtc), + TP_PROTO(struct intel_plane *plane, struct intel_crtc *crtc), TP_ARGS(plane, crtc), TP_STRUCT__entry( + __string(dev, __dev_name_kms(plane)) __field(enum pipe, pipe) __field(u32, frame) __field(u32, scanline) __array(int, src, 4) __array(int, dst, 4) - __string(name, plane->name) + __string(name, plane->base.name) ), TP_fast_assign( - __assign_str(name, plane->name); + __assign_str(dev, __dev_name_kms(plane)); + __assign_str(name, plane->base.name); __entry->pipe = crtc->pipe; __entry->frame = intel_crtc_get_vblank_counter(crtc); __entry->scanline = intel_get_crtc_scanline(crtc); - memcpy(__entry->src, &plane->state->src, sizeof(__entry->src)); - memcpy(__entry->dst, &plane->state->dst, sizeof(__entry->dst)); + memcpy(__entry->src, &plane->base.state->src, sizeof(__entry->src)); + memcpy(__entry->dst, &plane->base.state->dst, sizeof(__entry->dst)); ), - TP_printk("pipe %c, plane %s, frame=%u, scanline=%u, " DRM_RECT_FP_FMT " -> " DRM_RECT_FMT, - pipe_name(__entry->pipe), __get_str(name), + TP_printk("dev %s, pipe %c, plane %s, frame=%u, scanline=%u, " DRM_RECT_FP_FMT " -> " DRM_RECT_FMT, + __get_str(dev), pipe_name(__entry->pipe), __get_str(name), __entry->frame, __entry->scanline, DRM_RECT_FP_ARG((const struct drm_rect *)__entry->src), DRM_RECT_ARG((const struct drm_rect *)__entry->dst)) ); TRACE_EVENT(intel_plane_disable_arm, - TP_PROTO(struct drm_plane *plane, struct intel_crtc *crtc), + TP_PROTO(struct intel_plane *plane, struct intel_crtc *crtc), TP_ARGS(plane, crtc), TP_STRUCT__entry( + __string(dev, __dev_name_kms(plane)) __field(enum pipe, pipe) __field(u32, frame) __field(u32, scanline) - __string(name, plane->name) + __string(name, plane->base.name) ), TP_fast_assign( - __assign_str(name, plane->name); + __assign_str(dev, __dev_name_kms(plane)); + __assign_str(name, plane->base.name); __entry->pipe = crtc->pipe; __entry->frame = intel_crtc_get_vblank_counter(crtc); __entry->scanline = intel_get_crtc_scanline(crtc); ), - TP_printk("pipe %c, plane %s, frame=%u, scanline=%u", - pipe_name(__entry->pipe), __get_str(name), + TP_printk("dev %s, pipe %c, plane %s, frame=%u, scanline=%u", + __get_str(dev), pipe_name(__entry->pipe), __get_str(name), __entry->frame, __entry->scanline) ); @@ -369,6 +400,8 @@ TRACE_EVENT(intel_fbc_activate, TP_ARGS(plane), TP_STRUCT__entry( + __string(dev, __dev_name_kms(plane)) + __string(name, plane->base.name) __field(enum pipe, pipe) __field(u32, frame) __field(u32, scanline) @@ -377,13 +410,16 @@ TRACE_EVENT(intel_fbc_activate, TP_fast_assign( struct intel_crtc *crtc = intel_crtc_for_pipe(to_i915(plane->base.dev), plane->pipe); + __assign_str(dev, __dev_name_kms(plane)); + __assign_str(name, plane->base.name) __entry->pipe = crtc->pipe; __entry->frame = intel_crtc_get_vblank_counter(crtc); __entry->scanline = intel_get_crtc_scanline(crtc); ), - TP_printk("pipe %c, frame=%u, scanline=%u", - pipe_name(__entry->pipe), __entry->frame, __entry->scanline) + TP_printk("dev %s, pipe %c, plane %s, frame=%u, scanline=%u", + __get_str(dev), pipe_name(__entry->pipe), __get_str(name), + __entry->frame, __entry->scanline) ); TRACE_EVENT(intel_fbc_deactivate, @@ -391,6 +427,8 @@ TRACE_EVENT(intel_fbc_deactivate, TP_ARGS(plane), TP_STRUCT__entry( + __string(dev, __dev_name_kms(plane)) + __string(name, plane->base.name) __field(enum pipe, pipe) __field(u32, frame) __field(u32, scanline) @@ -399,13 +437,16 @@ TRACE_EVENT(intel_fbc_deactivate, TP_fast_assign( struct intel_crtc *crtc = intel_crtc_for_pipe(to_i915(plane->base.dev), plane->pipe); + __assign_str(dev, __dev_name_kms(plane)); + __assign_str(name, plane->base.name) __entry->pipe = crtc->pipe; __entry->frame = intel_crtc_get_vblank_counter(crtc); __entry->scanline = intel_get_crtc_scanline(crtc); ), - TP_printk("pipe %c, frame=%u, scanline=%u", - pipe_name(__entry->pipe), __entry->frame, __entry->scanline) + TP_printk("dev %s, pipe %c, plane %s, frame=%u, scanline=%u", + __get_str(dev), pipe_name(__entry->pipe), __get_str(name), + __entry->frame, __entry->scanline) ); TRACE_EVENT(intel_fbc_nuke, @@ -413,6 +454,8 @@ TRACE_EVENT(intel_fbc_nuke, TP_ARGS(plane), TP_STRUCT__entry( + __string(dev, __dev_name_kms(plane)) + __string(name, plane->base.name) __field(enum pipe, pipe) __field(u32, frame) __field(u32, scanline) @@ -421,13 +464,16 @@ TRACE_EVENT(intel_fbc_nuke, TP_fast_assign( struct intel_crtc *crtc = intel_crtc_for_pipe(to_i915(plane->base.dev), plane->pipe); + __assign_str(dev, __dev_name_kms(plane)); + __assign_str(name, plane->base.name) __entry->pipe = crtc->pipe; __entry->frame = intel_crtc_get_vblank_counter(crtc); __entry->scanline = intel_get_crtc_scanline(crtc); ), - TP_printk("pipe %c, frame=%u, scanline=%u", - pipe_name(__entry->pipe), __entry->frame, __entry->scanline) + TP_printk("dev %s, pipe %c, plane %s, frame=%u, scanline=%u", + __get_str(dev), pipe_name(__entry->pipe), __get_str(name), + __entry->frame, __entry->scanline) ); TRACE_EVENT(intel_crtc_vblank_work_start, @@ -435,20 +481,22 @@ TRACE_EVENT(intel_crtc_vblank_work_start, TP_ARGS(crtc), TP_STRUCT__entry( + __string(dev, __dev_name_kms(crtc)) __field(enum pipe, pipe) __field(u32, frame) __field(u32, scanline) ), TP_fast_assign( + __assign_str(dev, __dev_name_kms(crtc)); __entry->pipe = crtc->pipe; __entry->frame = intel_crtc_get_vblank_counter(crtc); __entry->scanline = intel_get_crtc_scanline(crtc); ), - TP_printk("pipe %c, frame=%u, scanline=%u", - pipe_name(__entry->pipe), __entry->frame, - __entry->scanline) + TP_printk("dev %s, pipe %c, frame=%u, scanline=%u", + __get_str(dev), pipe_name(__entry->pipe), + __entry->frame, __entry->scanline) ); TRACE_EVENT(intel_crtc_vblank_work_end, @@ -456,20 +504,22 @@ TRACE_EVENT(intel_crtc_vblank_work_end, TP_ARGS(crtc), TP_STRUCT__entry( + __string(dev, __dev_name_kms(crtc)) __field(enum pipe, pipe) __field(u32, frame) __field(u32, scanline) ), TP_fast_assign( + __assign_str(dev, __dev_name_kms(crtc)); __entry->pipe = crtc->pipe; __entry->frame = intel_crtc_get_vblank_counter(crtc); __entry->scanline = intel_get_crtc_scanline(crtc); ), - TP_printk("pipe %c, frame=%u, scanline=%u", - pipe_name(__entry->pipe), __entry->frame, - __entry->scanline) + TP_printk("dev %s, pipe %c, frame=%u, scanline=%u", + __get_str(dev), pipe_name(__entry->pipe), + __entry->frame, __entry->scanline) ); TRACE_EVENT(intel_pipe_update_start, @@ -477,6 +527,7 @@ TRACE_EVENT(intel_pipe_update_start, TP_ARGS(crtc), TP_STRUCT__entry( + __string(dev, __dev_name_kms(crtc)) __field(enum pipe, pipe) __field(u32, frame) __field(u32, scanline) @@ -485,6 +536,7 @@ TRACE_EVENT(intel_pipe_update_start, ), TP_fast_assign( + __assign_str(dev, __dev_name_kms(crtc)); __entry->pipe = crtc->pipe; __entry->frame = intel_crtc_get_vblank_counter(crtc); __entry->scanline = intel_get_crtc_scanline(crtc); @@ -492,9 +544,10 @@ TRACE_EVENT(intel_pipe_update_start, __entry->max = crtc->debug.max_vbl; ), - TP_printk("pipe %c, frame=%u, scanline=%u, min=%u, max=%u", - pipe_name(__entry->pipe), __entry->frame, - __entry->scanline, __entry->min, __entry->max) + TP_printk("dev %s, pipe %c, frame=%u, scanline=%u, min=%u, max=%u", + __get_str(dev), pipe_name(__entry->pipe), + __entry->frame, __entry->scanline, + __entry->min, __entry->max) ); TRACE_EVENT(intel_pipe_update_vblank_evaded, @@ -502,6 +555,7 @@ TRACE_EVENT(intel_pipe_update_vblank_evaded, TP_ARGS(crtc), TP_STRUCT__entry( + __string(dev, __dev_name_kms(crtc)) __field(enum pipe, pipe) __field(u32, frame) __field(u32, scanline) @@ -510,6 +564,7 @@ TRACE_EVENT(intel_pipe_update_vblank_evaded, ), TP_fast_assign( + __assign_str(dev, __dev_name_kms(crtc)); __entry->pipe = crtc->pipe; __entry->frame = crtc->debug.start_vbl_count; __entry->scanline = crtc->debug.scanline_start; @@ -517,9 +572,10 @@ TRACE_EVENT(intel_pipe_update_vblank_evaded, __entry->max = crtc->debug.max_vbl; ), - TP_printk("pipe %c, frame=%u, scanline=%u, min=%u, max=%u", - pipe_name(__entry->pipe), __entry->frame, - __entry->scanline, __entry->min, __entry->max) + TP_printk("dev %s, pipe %c, frame=%u, scanline=%u, min=%u, max=%u", + __get_str(dev), pipe_name(__entry->pipe), + __entry->frame, __entry->scanline, + __entry->min, __entry->max) ); TRACE_EVENT(intel_pipe_update_end, @@ -527,56 +583,64 @@ TRACE_EVENT(intel_pipe_update_end, TP_ARGS(crtc, frame, scanline_end), TP_STRUCT__entry( + __string(dev, __dev_name_kms(crtc)) __field(enum pipe, pipe) __field(u32, frame) __field(u32, scanline) ), TP_fast_assign( + __assign_str(dev, __dev_name_kms(crtc)); __entry->pipe = crtc->pipe; __entry->frame = frame; __entry->scanline = scanline_end; ), - TP_printk("pipe %c, frame=%u, scanline=%u", - pipe_name(__entry->pipe), __entry->frame, - __entry->scanline) + TP_printk("dev %s, pipe %c, frame=%u, scanline=%u", + __get_str(dev), pipe_name(__entry->pipe), + __entry->frame, __entry->scanline) ); TRACE_EVENT(intel_frontbuffer_invalidate, - TP_PROTO(unsigned int frontbuffer_bits, unsigned int origin), - TP_ARGS(frontbuffer_bits, origin), + TP_PROTO(struct drm_i915_private *i915, + unsigned int frontbuffer_bits, unsigned int origin), + TP_ARGS(i915, frontbuffer_bits, origin), TP_STRUCT__entry( + __string(dev, __dev_name_i915(i915)) __field(unsigned int, frontbuffer_bits) __field(unsigned int, origin) ), TP_fast_assign( + __assign_str(dev, __dev_name_i915(i915)); __entry->frontbuffer_bits = frontbuffer_bits; __entry->origin = origin; ), - TP_printk("frontbuffer_bits=0x%08x, origin=%u", - __entry->frontbuffer_bits, __entry->origin) + TP_printk("dev %s, frontbuffer_bits=0x%08x, origin=%u", + __get_str(dev), __entry->frontbuffer_bits, __entry->origin) ); TRACE_EVENT(intel_frontbuffer_flush, - TP_PROTO(unsigned int frontbuffer_bits, unsigned int origin), - TP_ARGS(frontbuffer_bits, origin), + TP_PROTO(struct drm_i915_private *i915, + unsigned int frontbuffer_bits, unsigned int origin), + TP_ARGS(i915, frontbuffer_bits, origin), TP_STRUCT__entry( + __string(dev, __dev_name_i915(i915)) __field(unsigned int, frontbuffer_bits) __field(unsigned int, origin) ), TP_fast_assign( + __assign_str(dev, __dev_name_i915(i915)); __entry->frontbuffer_bits = frontbuffer_bits; __entry->origin = origin; ), - TP_printk("frontbuffer_bits=0x%08x, origin=%u", - __entry->frontbuffer_bits, __entry->origin) + TP_printk("dev %s, frontbuffer_bits=0x%08x, origin=%u", + __get_str(dev), __entry->frontbuffer_bits, __entry->origin) ); #endif /* __INTEL_DISPLAY_TRACE_H__ */ diff --git a/drivers/gpu/drm/i915/display/intel_display_types.h b/drivers/gpu/drm/i915/display/intel_display_types.h index 298d00a11f47..f07395065a69 100644 --- a/drivers/gpu/drm/i915/display/intel_display_types.h +++ b/drivers/gpu/drm/i915/display/intel_display_types.h @@ -969,6 +969,15 @@ struct intel_mpllb_state { u32 mpllb_sscstep; }; +/* Used by dp and fdi links */ +struct intel_link_m_n { + u32 tu; + u32 data_m; + u32 data_n; + u32 link_m; + u32 link_n; +}; + struct intel_crtc_state { /* * uapi (drm) state. This is the software state shown to userspace. @@ -1001,11 +1010,15 @@ struct intel_crtc_state { */ struct { bool active, enable; + /* logical state of LUTs */ struct drm_property_blob *degamma_lut, *gamma_lut, *ctm; struct drm_display_mode mode, pipe_mode, adjusted_mode; enum drm_scaling_filter scaling_filter; } hw; + /* actual state of LUTs */ + struct drm_property_blob *pre_csc_lut, *post_csc_lut; + /** * quirks - bitfield with hw state readout quirks * @@ -1362,6 +1375,7 @@ struct intel_crtc { u16 vmax_vblank_start; struct intel_display_power_domain_set enabled_power_domains; + struct intel_display_power_domain_set hw_readout_power_domains; struct intel_overlay *overlay; struct intel_crtc_state *config; @@ -1799,51 +1813,6 @@ struct intel_dp_mst_encoder { struct intel_connector *connector; }; -static inline enum dpio_channel -vlv_dig_port_to_channel(struct intel_digital_port *dig_port) -{ - switch (dig_port->base.port) { - default: - MISSING_CASE(dig_port->base.port); - fallthrough; - case PORT_B: - case PORT_D: - return DPIO_CH0; - case PORT_C: - return DPIO_CH1; - } -} - -static inline enum dpio_phy -vlv_dig_port_to_phy(struct intel_digital_port *dig_port) -{ - switch (dig_port->base.port) { - default: - MISSING_CASE(dig_port->base.port); - fallthrough; - case PORT_B: - case PORT_C: - return DPIO_PHY0; - case PORT_D: - return DPIO_PHY1; - } -} - -static inline enum dpio_channel -vlv_pipe_to_channel(enum pipe pipe) -{ - switch (pipe) { - default: - MISSING_CASE(pipe); - fallthrough; - case PIPE_A: - case PIPE_C: - return DPIO_CH0; - case PIPE_B: - return DPIO_CH1; - } -} - struct intel_load_detect_pipe { struct drm_atomic_state *restore_state; }; @@ -2040,15 +2009,16 @@ static inline bool intel_crtc_has_type(const struct intel_crtc_state *crtc_state, enum intel_output_type type) { - return crtc_state->output_types & (1 << type); + return crtc_state->output_types & BIT(type); } + static inline bool intel_crtc_has_dp_encoder(const struct intel_crtc_state *crtc_state) { return crtc_state->output_types & - ((1 << INTEL_OUTPUT_DP) | - (1 << INTEL_OUTPUT_DP_MST) | - (1 << INTEL_OUTPUT_EDP)); + (BIT(INTEL_OUTPUT_DP) | + BIT(INTEL_OUTPUT_DP_MST) | + BIT(INTEL_OUTPUT_EDP)); } static inline bool @@ -2057,6 +2027,20 @@ intel_crtc_needs_modeset(const struct intel_crtc_state *crtc_state) return drm_atomic_crtc_needs_modeset(&crtc_state->uapi); } +static inline bool +intel_crtc_needs_fastset(const struct intel_crtc_state *crtc_state) +{ + return crtc_state->update_pipe; +} + +static inline bool +intel_crtc_needs_color_update(const struct intel_crtc_state *crtc_state) +{ + return crtc_state->uapi.color_mgmt_changed || + intel_crtc_needs_fastset(crtc_state) || + intel_crtc_needs_modeset(crtc_state); +} + static inline u32 intel_plane_ggtt_offset(const struct intel_plane_state *plane_state) { return i915_ggtt_offset(plane_state->ggtt_vma); diff --git a/drivers/gpu/drm/i915/display/intel_dkl_phy.c b/drivers/gpu/drm/i915/display/intel_dkl_phy.c index 710b030c7ed5..57cc3edba016 100644 --- a/drivers/gpu/drm/i915/display/intel_dkl_phy.c +++ b/drivers/gpu/drm/i915/display/intel_dkl_phy.c @@ -9,9 +9,10 @@ #include "intel_de.h" #include "intel_display.h" #include "intel_dkl_phy.h" +#include "intel_dkl_phy_regs.h" static void -dkl_phy_set_hip_idx(struct drm_i915_private *i915, i915_reg_t reg, int idx) +dkl_phy_set_hip_idx(struct drm_i915_private *i915, struct intel_dkl_phy_reg reg) { enum tc_port tc_port = DKL_REG_TC_PORT(reg); @@ -19,28 +20,27 @@ dkl_phy_set_hip_idx(struct drm_i915_private *i915, i915_reg_t reg, int idx) intel_de_write(i915, HIP_INDEX_REG(tc_port), - HIP_INDEX_VAL(tc_port, idx)); + HIP_INDEX_VAL(tc_port, reg.bank_idx)); } /** * intel_dkl_phy_read - read a Dekel PHY register * @i915: i915 device instance * @reg: Dekel PHY register - * @ln: lane instance of @reg * * Read the @reg Dekel PHY register. * * Returns the read value. */ u32 -intel_dkl_phy_read(struct drm_i915_private *i915, i915_reg_t reg, int ln) +intel_dkl_phy_read(struct drm_i915_private *i915, struct intel_dkl_phy_reg reg) { u32 val; spin_lock(&i915->display.dkl.phy_lock); - dkl_phy_set_hip_idx(i915, reg, ln); - val = intel_de_read(i915, reg); + dkl_phy_set_hip_idx(i915, reg); + val = intel_de_read(i915, DKL_REG_MMIO(reg)); spin_unlock(&i915->display.dkl.phy_lock); @@ -51,18 +51,17 @@ intel_dkl_phy_read(struct drm_i915_private *i915, i915_reg_t reg, int ln) * intel_dkl_phy_write - write a Dekel PHY register * @i915: i915 device instance * @reg: Dekel PHY register - * @ln: lane instance of @reg * @val: value to write * * Write @val to the @reg Dekel PHY register. */ void -intel_dkl_phy_write(struct drm_i915_private *i915, i915_reg_t reg, int ln, u32 val) +intel_dkl_phy_write(struct drm_i915_private *i915, struct intel_dkl_phy_reg reg, u32 val) { spin_lock(&i915->display.dkl.phy_lock); - dkl_phy_set_hip_idx(i915, reg, ln); - intel_de_write(i915, reg, val); + dkl_phy_set_hip_idx(i915, reg); + intel_de_write(i915, DKL_REG_MMIO(reg), val); spin_unlock(&i915->display.dkl.phy_lock); } @@ -71,7 +70,6 @@ intel_dkl_phy_write(struct drm_i915_private *i915, i915_reg_t reg, int ln, u32 v * intel_dkl_phy_rmw - read-modify-write a Dekel PHY register * @i915: i915 device instance * @reg: Dekel PHY register - * @ln: lane instance of @reg * @clear: mask to clear * @set: mask to set * @@ -79,12 +77,12 @@ intel_dkl_phy_write(struct drm_i915_private *i915, i915_reg_t reg, int ln, u32 v * this value back to the register if the value differs from the read one. */ void -intel_dkl_phy_rmw(struct drm_i915_private *i915, i915_reg_t reg, int ln, u32 clear, u32 set) +intel_dkl_phy_rmw(struct drm_i915_private *i915, struct intel_dkl_phy_reg reg, u32 clear, u32 set) { spin_lock(&i915->display.dkl.phy_lock); - dkl_phy_set_hip_idx(i915, reg, ln); - intel_de_rmw(i915, reg, clear, set); + dkl_phy_set_hip_idx(i915, reg); + intel_de_rmw(i915, DKL_REG_MMIO(reg), clear, set); spin_unlock(&i915->display.dkl.phy_lock); } @@ -93,17 +91,16 @@ intel_dkl_phy_rmw(struct drm_i915_private *i915, i915_reg_t reg, int ln, u32 cle * intel_dkl_phy_posting_read - do a posting read from a Dekel PHY register * @i915: i915 device instance * @reg: Dekel PHY register - * @ln: lane instance of @reg * * Read the @reg Dekel PHY register without returning the read value. */ void -intel_dkl_phy_posting_read(struct drm_i915_private *i915, i915_reg_t reg, int ln) +intel_dkl_phy_posting_read(struct drm_i915_private *i915, struct intel_dkl_phy_reg reg) { spin_lock(&i915->display.dkl.phy_lock); - dkl_phy_set_hip_idx(i915, reg, ln); - intel_de_posting_read(i915, reg); + dkl_phy_set_hip_idx(i915, reg); + intel_de_posting_read(i915, DKL_REG_MMIO(reg)); spin_unlock(&i915->display.dkl.phy_lock); } diff --git a/drivers/gpu/drm/i915/display/intel_dkl_phy.h b/drivers/gpu/drm/i915/display/intel_dkl_phy.h index 260ad121a0b1..570ee36f9386 100644 --- a/drivers/gpu/drm/i915/display/intel_dkl_phy.h +++ b/drivers/gpu/drm/i915/display/intel_dkl_phy.h @@ -8,17 +8,17 @@ #include <linux/types.h> -#include "i915_reg_defs.h" +#include "intel_dkl_phy_regs.h" struct drm_i915_private; u32 -intel_dkl_phy_read(struct drm_i915_private *i915, i915_reg_t reg, int ln); +intel_dkl_phy_read(struct drm_i915_private *i915, struct intel_dkl_phy_reg reg); void -intel_dkl_phy_write(struct drm_i915_private *i915, i915_reg_t reg, int ln, u32 val); +intel_dkl_phy_write(struct drm_i915_private *i915, struct intel_dkl_phy_reg reg, u32 val); void -intel_dkl_phy_rmw(struct drm_i915_private *i915, i915_reg_t reg, int ln, u32 clear, u32 set); +intel_dkl_phy_rmw(struct drm_i915_private *i915, struct intel_dkl_phy_reg reg, u32 clear, u32 set); void -intel_dkl_phy_posting_read(struct drm_i915_private *i915, i915_reg_t reg, int ln); +intel_dkl_phy_posting_read(struct drm_i915_private *i915, struct intel_dkl_phy_reg reg); #endif /* __INTEL_DKL_PHY_H__ */ diff --git a/drivers/gpu/drm/i915/display/intel_dkl_phy_regs.h b/drivers/gpu/drm/i915/display/intel_dkl_phy_regs.h new file mode 100644 index 000000000000..56085b32956d --- /dev/null +++ b/drivers/gpu/drm/i915/display/intel_dkl_phy_regs.h @@ -0,0 +1,204 @@ +/* SPDX-License-Identifier: MIT */ +/* + * Copyright © 2022 Intel Corporation + */ + +#ifndef __INTEL_DKL_PHY_REGS__ +#define __INTEL_DKL_PHY_REGS__ + +#include <linux/types.h> + +struct intel_dkl_phy_reg { + u32 reg:24; + u32 bank_idx:4; +}; + +#define _DKL_PHY1_BASE 0x168000 +#define _DKL_PHY2_BASE 0x169000 +#define _DKL_PHY3_BASE 0x16A000 +#define _DKL_PHY4_BASE 0x16B000 +#define _DKL_PHY5_BASE 0x16C000 +#define _DKL_PHY6_BASE 0x16D000 + +#define DKL_REG_TC_PORT(__reg) \ + (TC_PORT_1 + ((__reg).reg - _DKL_PHY1_BASE) / (_DKL_PHY2_BASE - _DKL_PHY1_BASE)) + +/* DEKEL PHY MMIO Address = Phy base + (internal address & ~index_mask) */ +#define DKL_REG_MMIO(__reg) _MMIO((__reg).reg) + +#define _DKL_REG_PHY_BASE(tc_port) _PORT(tc_port, \ + _DKL_PHY1_BASE, \ + _DKL_PHY2_BASE) + +#define _DKL_BANK_SHIFT 12 +#define _DKL_REG_BANK_OFFSET(phy_offset) \ + ((phy_offset) & ((1 << _DKL_BANK_SHIFT) - 1)) +#define _DKL_REG_BANK_IDX(phy_offset) \ + (((phy_offset) >> _DKL_BANK_SHIFT) & 0xf) + +#define _DKL_REG(tc_port, phy_offset) \ + ((const struct intel_dkl_phy_reg) { \ + .reg = _DKL_REG_PHY_BASE(tc_port) + \ + _DKL_REG_BANK_OFFSET(phy_offset), \ + .bank_idx = _DKL_REG_BANK_IDX(phy_offset), \ + }) + +#define _DKL_REG_LN(tc_port, ln_idx, ln0_offs, ln1_offs) \ + _DKL_REG(tc_port, (ln0_offs) + (ln_idx) * ((ln1_offs) - (ln0_offs))) + +#define _DKL_PCS_DW5_LN0 0x0014 +#define _DKL_PCS_DW5_LN1 0x1014 +#define DKL_PCS_DW5(tc_port, ln) _DKL_REG_LN(tc_port, ln, \ + _DKL_PCS_DW5_LN0, \ + _DKL_PCS_DW5_LN1) +#define DKL_PCS_DW5_CORE_SOFTRESET REG_BIT(11) + +#define _DKL_PLL_DIV0 0x2200 +#define DKL_PLL_DIV0(tc_port) _DKL_REG(tc_port, \ + _DKL_PLL_DIV0) +#define DKL_PLL_DIV0_AFC_STARTUP_MASK REG_GENMASK(27, 25) +#define DKL_PLL_DIV0_AFC_STARTUP(val) REG_FIELD_PREP(DKL_PLL_DIV0_AFC_STARTUP_MASK, (val)) +#define DKL_PLL_DIV0_INTEG_COEFF(x) ((x) << 16) +#define DKL_PLL_DIV0_INTEG_COEFF_MASK (0x1F << 16) +#define DKL_PLL_DIV0_PROP_COEFF(x) ((x) << 12) +#define DKL_PLL_DIV0_PROP_COEFF_MASK (0xF << 12) +#define DKL_PLL_DIV0_FBPREDIV_SHIFT (8) +#define DKL_PLL_DIV0_FBPREDIV(x) ((x) << DKL_PLL_DIV0_FBPREDIV_SHIFT) +#define DKL_PLL_DIV0_FBPREDIV_MASK (0xF << DKL_PLL_DIV0_FBPREDIV_SHIFT) +#define DKL_PLL_DIV0_FBDIV_INT(x) ((x) << 0) +#define DKL_PLL_DIV0_FBDIV_INT_MASK (0xFF << 0) +#define DKL_PLL_DIV0_MASK (DKL_PLL_DIV0_INTEG_COEFF_MASK | \ + DKL_PLL_DIV0_PROP_COEFF_MASK | \ + DKL_PLL_DIV0_FBPREDIV_MASK | \ + DKL_PLL_DIV0_FBDIV_INT_MASK) + +#define _DKL_PLL_DIV1 0x2204 +#define DKL_PLL_DIV1(tc_port) _DKL_REG(tc_port, \ + _DKL_PLL_DIV1) +#define DKL_PLL_DIV1_IREF_TRIM(x) ((x) << 16) +#define DKL_PLL_DIV1_IREF_TRIM_MASK (0x1F << 16) +#define DKL_PLL_DIV1_TDC_TARGET_CNT(x) ((x) << 0) +#define DKL_PLL_DIV1_TDC_TARGET_CNT_MASK (0xFF << 0) + +#define _DKL_PLL_SSC 0x2210 +#define DKL_PLL_SSC(tc_port) _DKL_REG(tc_port, \ + _DKL_PLL_SSC) +#define DKL_PLL_SSC_IREF_NDIV_RATIO(x) ((x) << 29) +#define DKL_PLL_SSC_IREF_NDIV_RATIO_MASK (0x7 << 29) +#define DKL_PLL_SSC_STEP_LEN(x) ((x) << 16) +#define DKL_PLL_SSC_STEP_LEN_MASK (0xFF << 16) +#define DKL_PLL_SSC_STEP_NUM(x) ((x) << 11) +#define DKL_PLL_SSC_STEP_NUM_MASK (0x7 << 11) +#define DKL_PLL_SSC_EN (1 << 9) + +#define _DKL_PLL_BIAS 0x2214 +#define DKL_PLL_BIAS(tc_port) _DKL_REG(tc_port, \ + _DKL_PLL_BIAS) +#define DKL_PLL_BIAS_FRAC_EN_H (1 << 30) +#define DKL_PLL_BIAS_FBDIV_SHIFT (8) +#define DKL_PLL_BIAS_FBDIV_FRAC(x) ((x) << DKL_PLL_BIAS_FBDIV_SHIFT) +#define DKL_PLL_BIAS_FBDIV_FRAC_MASK (0x3FFFFF << DKL_PLL_BIAS_FBDIV_SHIFT) + +#define _DKL_PLL_TDC_COLDST_BIAS 0x2218 +#define DKL_PLL_TDC_COLDST_BIAS(tc_port) _DKL_REG(tc_port, \ + _DKL_PLL_TDC_COLDST_BIAS) +#define DKL_PLL_TDC_SSC_STEP_SIZE(x) ((x) << 8) +#define DKL_PLL_TDC_SSC_STEP_SIZE_MASK (0xFF << 8) +#define DKL_PLL_TDC_FEED_FWD_GAIN(x) ((x) << 0) +#define DKL_PLL_TDC_FEED_FWD_GAIN_MASK (0xFF << 0) + +#define _DKL_REFCLKIN_CTL 0x212C +#define DKL_REFCLKIN_CTL(tc_port) _DKL_REG(tc_port, \ + _DKL_REFCLKIN_CTL) +/* Bits are the same as MG_REFCLKIN_CTL */ + +#define _DKL_CLKTOP2_HSCLKCTL 0x20D4 +#define DKL_CLKTOP2_HSCLKCTL(rc_port) _DKL_REG(tc_port, \ + _DKL_CLKTOP2_HSCLKCTL) +/* Bits are the same as MG_CLKTOP2_HSCLKCTL */ + +#define _DKL_CLKTOP2_CORECLKCTL1 0x20D8 +#define DKL_CLKTOP2_CORECLKCTL1(tc_port) _DKL_REG(tc_port, \ + _DKL_CLKTOP2_CORECLKCTL1) +/* Bits are the same as MG_CLKTOP2_CORECLKCTL1 */ + +#define _DKL_TX_DPCNTL0_LN0 0x02C0 +#define _DKL_TX_DPCNTL0_LN1 0x12C0 +#define DKL_TX_DPCNTL0(tc_port, ln) _DKL_REG_LN(tc_port, ln, \ + _DKL_TX_DPCNTL0_LN0, \ + _DKL_TX_DPCNTL0_LN1) +#define DKL_TX_PRESHOOT_COEFF(x) ((x) << 13) +#define DKL_TX_PRESHOOT_COEFF_MASK (0x1f << 13) +#define DKL_TX_DE_EMPHASIS_COEFF(x) ((x) << 8) +#define DKL_TX_DE_EMPAHSIS_COEFF_MASK (0x1f << 8) +#define DKL_TX_VSWING_CONTROL(x) ((x) << 0) +#define DKL_TX_VSWING_CONTROL_MASK (0x7 << 0) + +#define _DKL_TX_DPCNTL1_LN0 0x02C4 +#define _DKL_TX_DPCNTL1_LN1 0x12C4 +#define DKL_TX_DPCNTL1(tc_port, ln) _DKL_REG_LN(tc_port, ln, \ + _DKL_TX_DPCNTL1_LN0, \ + _DKL_TX_DPCNTL1_LN1) +/* Bits are the same as DKL_TX_DPCNTRL0 */ + +#define _DKL_TX_DPCNTL2_LN0 0x02C8 +#define _DKL_TX_DPCNTL2_LN1 0x12C8 +#define DKL_TX_DPCNTL2(tc_port, ln) _DKL_REG_LN(tc_port, ln, \ + _DKL_TX_DPCNTL2_LN0, \ + _DKL_TX_DPCNTL2_LN1) +#define DKL_TX_DP20BITMODE REG_BIT(2) +#define DKL_TX_DPCNTL2_CFG_LOADGENSELECT_TX1_MASK REG_GENMASK(4, 3) +#define DKL_TX_DPCNTL2_CFG_LOADGENSELECT_TX1(val) REG_FIELD_PREP(DKL_TX_DPCNTL2_CFG_LOADGENSELECT_TX1_MASK, (val)) +#define DKL_TX_DPCNTL2_CFG_LOADGENSELECT_TX2_MASK REG_GENMASK(6, 5) +#define DKL_TX_DPCNTL2_CFG_LOADGENSELECT_TX2(val) REG_FIELD_PREP(DKL_TX_DPCNTL2_CFG_LOADGENSELECT_TX2_MASK, (val)) + +#define _DKL_TX_FW_CALIB_LN0 0x02F8 +#define _DKL_TX_FW_CALIB_LN1 0x12F8 +#define DKL_TX_FW_CALIB(tc_port, ln) _DKL_REG_LN(tc_port, ln, \ + _DKL_TX_FW_CALIB_LN0, \ + _DKL_TX_FW_CALIB_LN1) +#define DKL_TX_CFG_DISABLE_WAIT_INIT (1 << 7) + +#define _DKL_TX_PMD_LANE_SUS_LN0 0x0D00 +#define _DKL_TX_PMD_LANE_SUS_LN1 0x1D00 +#define DKL_TX_PMD_LANE_SUS(tc_port, ln) _DKL_REG_LN(tc_port, ln, \ + _DKL_TX_PMD_LANE_SUS_LN0, \ + _DKL_TX_PMD_LANE_SUS_LN1) + +#define _DKL_TX_DW17_LN0 0x0DC4 +#define _DKL_TX_DW17_LN1 0x1DC4 +#define DKL_TX_DW17(tc_port, ln) _DKL_REG_LN(tc_port, ln, \ + _DKL_TX_DW17_LN0, \ + _DKL_TX_DW17_LN1) + +#define _DKL_TX_DW18_LN0 0x0DC8 +#define _DKL_TX_DW18_LN1 0x1DC8 +#define DKL_TX_DW18(tc_port, ln) _DKL_REG_LN(tc_port, ln, \ + _DKL_TX_DW18_LN0, \ + _DKL_TX_DW18_LN1) + +#define _DKL_DP_MODE_LN0 0x00A0 +#define _DKL_DP_MODE_LN1 0x10A0 +#define DKL_DP_MODE(tc_port, ln) _DKL_REG_LN(tc_port, ln, \ + _DKL_DP_MODE_LN0, \ + _DKL_DP_MODE_LN1) + +#define _DKL_CMN_UC_DW27 0x236C +#define DKL_CMN_UC_DW_27(tc_port) _DKL_REG(tc_port, \ + _DKL_CMN_UC_DW27) +#define DKL_CMN_UC_DW27_UC_HEALTH (0x1 << 15) + +/* + * Each Dekel PHY is addressed through a 4KB aperture. Each PHY has more than + * 4KB of register space, so a separate index is programmed in HIP_INDEX_REG0 + * or HIP_INDEX_REG1, based on the port number, to set the upper 2 address + * bits that point the 4KB window into the full PHY register space. + */ +#define _HIP_INDEX_REG0 0x1010A0 +#define _HIP_INDEX_REG1 0x1010A4 +#define HIP_INDEX_REG(tc_port) _MMIO((tc_port) < 4 ? _HIP_INDEX_REG0 \ + : _HIP_INDEX_REG1) +#define _HIP_INDEX_SHIFT(tc_port) (8 * ((tc_port) % 4)) +#define HIP_INDEX_VAL(tc_port, val) ((val) << _HIP_INDEX_SHIFT(tc_port)) + +#endif /* __INTEL_DKL_PHY_REGS__ */ diff --git a/drivers/gpu/drm/i915/display/intel_dmc.c b/drivers/gpu/drm/i915/display/intel_dmc.c index e52ecc0738a6..eff3add70611 100644 --- a/drivers/gpu/drm/i915/display/intel_dmc.c +++ b/drivers/gpu/drm/i915/display/intel_dmc.c @@ -52,8 +52,8 @@ #define DISPLAY_VER12_DMC_MAX_FW_SIZE ICL_DMC_MAX_FW_SIZE -#define DG2_DMC_PATH DMC_PATH(dg2, 2, 07) -#define DG2_DMC_VERSION_REQUIRED DMC_VERSION(2, 07) +#define DG2_DMC_PATH DMC_PATH(dg2, 2, 08) +#define DG2_DMC_VERSION_REQUIRED DMC_VERSION(2, 8) MODULE_FIRMWARE(DG2_DMC_PATH); #define ADLP_DMC_PATH DMC_PATH(adlp, 2, 16) @@ -1065,12 +1065,13 @@ static int intel_dmc_debugfs_status_show(struct seq_file *m, void *unused) seq_printf(m, "fw loaded: %s\n", str_yes_no(intel_dmc_has_payload(i915))); seq_printf(m, "path: %s\n", dmc->fw_path); - seq_printf(m, "Pipe A fw support: %s\n", + seq_printf(m, "Pipe A fw needed: %s\n", str_yes_no(GRAPHICS_VER(i915) >= 12)); seq_printf(m, "Pipe A fw loaded: %s\n", str_yes_no(dmc->dmc_info[DMC_FW_PIPEA].payload)); - seq_printf(m, "Pipe B fw support: %s\n", - str_yes_no(IS_ALDERLAKE_P(i915))); + seq_printf(m, "Pipe B fw needed: %s\n", + str_yes_no(IS_ALDERLAKE_P(i915) || + DISPLAY_VER(i915) >= 14)); seq_printf(m, "Pipe B fw loaded: %s\n", str_yes_no(dmc->dmc_info[DMC_FW_PIPEB].payload)); @@ -1081,22 +1082,19 @@ static int intel_dmc_debugfs_status_show(struct seq_file *m, void *unused) DMC_VERSION_MINOR(dmc->version)); if (DISPLAY_VER(i915) >= 12) { - if (IS_DGFX(i915)) { + i915_reg_t dc3co_reg; + + if (IS_DGFX(i915) || DISPLAY_VER(i915) >= 14) { + dc3co_reg = DG1_DMC_DEBUG3; dc5_reg = DG1_DMC_DEBUG_DC5_COUNT; } else { + dc3co_reg = TGL_DMC_DEBUG3; dc5_reg = TGL_DMC_DEBUG_DC5_COUNT; dc6_reg = TGL_DMC_DEBUG_DC6_COUNT; } - /* - * NOTE: DMC_DEBUG3 is a general purpose reg. - * According to B.Specs:49196 DMC f/w reuses DC5/6 counter - * reg for DC3CO debugging and validation, - * but TGL DMC f/w is using DMC_DEBUG3 reg for DC3CO counter. - */ seq_printf(m, "DC3CO count: %d\n", - intel_de_read(i915, IS_DGFX(i915) ? - DG1_DMC_DEBUG3 : TGL_DMC_DEBUG3)); + intel_de_read(i915, dc3co_reg)); } else { dc5_reg = IS_BROXTON(i915) ? BXT_DMC_DC3_DC5_COUNT : SKL_DMC_DC3_DC5_COUNT; diff --git a/drivers/gpu/drm/i915/display/intel_dp.c b/drivers/gpu/drm/i915/display/intel_dp.c index 2b5bc95a8b0d..67089711d9e2 100644 --- a/drivers/gpu/drm/i915/display/intel_dp.c +++ b/drivers/gpu/drm/i915/display/intel_dp.c @@ -46,6 +46,7 @@ #include "g4x_dp.h" #include "i915_debugfs.h" #include "i915_drv.h" +#include "i915_reg.h" #include "intel_atomic.h" #include "intel_audio.h" #include "intel_backlight.h" @@ -2306,6 +2307,7 @@ bool intel_dp_initial_fastset_check(struct intel_encoder *encoder, { struct drm_i915_private *i915 = to_i915(encoder->base.dev); struct intel_dp *intel_dp = enc_to_intel_dp(encoder); + bool fastset = true; /* * If BIOS has set an unsupported or non-standard link rate for some @@ -2313,9 +2315,10 @@ bool intel_dp_initial_fastset_check(struct intel_encoder *encoder, */ if (intel_dp_rate_index(intel_dp->source_rates, intel_dp->num_source_rates, crtc_state->port_clock) < 0) { - drm_dbg_kms(&i915->drm, "Forcing full modeset due to unsupported link rate\n"); + drm_dbg_kms(&i915->drm, "[ENCODER:%d:%s] Forcing full modeset due to unsupported link rate\n", + encoder->base.base.id, encoder->base.name); crtc_state->uapi.connectors_changed = true; - return false; + fastset = false; } /* @@ -2326,18 +2329,20 @@ bool intel_dp_initial_fastset_check(struct intel_encoder *encoder, * Remove once we have readout for DSC. */ if (crtc_state->dsc.compression_enable) { - drm_dbg_kms(&i915->drm, "Forcing full modeset due to DSC being enabled\n"); + drm_dbg_kms(&i915->drm, "[ENCODER:%d:%s] Forcing full modeset due to DSC being enabled\n", + encoder->base.base.id, encoder->base.name); crtc_state->uapi.mode_changed = true; - return false; + fastset = false; } if (CAN_PSR(intel_dp)) { - drm_dbg_kms(&i915->drm, "Forcing full modeset to compute PSR state\n"); + drm_dbg_kms(&i915->drm, "[ENCODER:%d:%s] Forcing full modeset to compute PSR state\n", + encoder->base.base.id, encoder->base.name); crtc_state->uapi.mode_changed = true; - return false; + fastset = false; } - return true; + return fastset; } static void intel_dp_get_pcon_dsc_cap(struct intel_dp *intel_dp) @@ -2686,7 +2691,6 @@ void intel_dp_configure_protocol_converter(struct intel_dp *intel_dp, str_enable_disable(tmp)); } - bool intel_dp_get_colorimetry_status(struct intel_dp *intel_dp) { u8 dprx = 0; @@ -4873,6 +4877,12 @@ void intel_dp_encoder_flush_work(struct drm_encoder *encoder) intel_pps_vdd_off_sync(intel_dp); + /* + * Ensure power off delay is respected on module remove, so that we can + * reduce delays at driver probe. See pps_init_timestamps(). + */ + intel_pps_wait_power_cycle(intel_dp); + intel_dp_aux_fini(intel_dp); } @@ -5174,19 +5184,6 @@ intel_dp_add_properties(struct intel_dp *intel_dp, struct drm_connector *connect if (has_gamut_metadata_dip(dev_priv, port)) drm_connector_attach_hdr_output_metadata_property(connector); - if (intel_dp_is_edp(intel_dp)) { - u32 allowed_scalers; - - allowed_scalers = BIT(DRM_MODE_SCALE_ASPECT) | BIT(DRM_MODE_SCALE_FULLSCREEN); - if (!HAS_GMCH(dev_priv)) - allowed_scalers |= BIT(DRM_MODE_SCALE_CENTER); - - drm_connector_attach_scaling_mode_property(connector, allowed_scalers); - - connector->state->scaling_mode = DRM_MODE_SCALE_ASPECT; - - } - if (HAS_VRR(dev_priv)) drm_connector_attach_vrr_capable_property(connector); } @@ -5199,8 +5196,7 @@ intel_edp_add_properties(struct intel_dp *intel_dp) const struct drm_display_mode *fixed_mode = intel_panel_preferred_fixed_mode(connector); - if (!fixed_mode) - return; + intel_attach_scaling_mode_property(&connector->base); drm_connector_set_panel_orientation_with_quirk(&connector->base, i915->display.vbt.orientation, @@ -5208,16 +5204,43 @@ intel_edp_add_properties(struct intel_dp *intel_dp) fixed_mode->vdisplay); } +static void intel_edp_backlight_setup(struct intel_dp *intel_dp, + struct intel_connector *connector) +{ + struct drm_i915_private *i915 = dp_to_i915(intel_dp); + enum pipe pipe = INVALID_PIPE; + + if (IS_VALLEYVIEW(i915) || IS_CHERRYVIEW(i915)) { + /* + * Figure out the current pipe for the initial backlight setup. + * If the current pipe isn't valid, try the PPS pipe, and if that + * fails just assume pipe A. + */ + pipe = vlv_active_pipe(intel_dp); + + if (pipe != PIPE_A && pipe != PIPE_B) + pipe = intel_dp->pps.pps_pipe; + + if (pipe != PIPE_A && pipe != PIPE_B) + pipe = PIPE_A; + + drm_dbg_kms(&i915->drm, + "[CONNECTOR:%d:%s] using pipe %c for initial backlight setup\n", + connector->base.base.id, connector->base.name, + pipe_name(pipe)); + } + + intel_backlight_setup(connector, pipe); +} + static bool intel_edp_init_connector(struct intel_dp *intel_dp, struct intel_connector *intel_connector) { struct drm_i915_private *dev_priv = dp_to_i915(intel_dp); - struct drm_device *dev = &dev_priv->drm; struct drm_connector *connector = &intel_connector->base; struct drm_display_mode *fixed_mode; struct intel_encoder *encoder = &dp_to_dig_port(intel_dp)->base; bool has_dpcd; - enum pipe pipe = INVALID_PIPE; struct edid *edid; if (!intel_dp_is_edp(intel_dp)) @@ -5230,7 +5253,7 @@ static bool intel_edp_init_connector(struct intel_dp *intel_dp, * with an already powered-on LVDS power sequencer. */ if (intel_get_lvds_encoder(dev_priv)) { - drm_WARN_ON(dev, + drm_WARN_ON(&dev_priv->drm, !(HAS_PCH_IBX(dev_priv) || HAS_PCH_CPT(dev_priv))); drm_info(&dev_priv->drm, "LVDS was detected, not registering eDP\n"); @@ -5246,11 +5269,12 @@ static bool intel_edp_init_connector(struct intel_dp *intel_dp, if (!has_dpcd) { /* if this fails, presume the device is a ghost */ drm_info(&dev_priv->drm, - "failed to retrieve link info, disabling eDP\n"); + "[ENCODER:%d:%s] failed to retrieve link info, disabling eDP\n", + encoder->base.base.id, encoder->base.name); goto out_vdd_off; } - mutex_lock(&dev->mode_config.mutex); + mutex_lock(&dev_priv->drm.mode_config.mutex); edid = drm_get_edid(connector, &intel_dp->aux.ddc); if (!edid) { /* Fallback to EDID from ACPI OpRegion, if any */ @@ -5275,9 +5299,7 @@ static bool intel_edp_init_connector(struct intel_dp *intel_dp, intel_bios_init_panel(dev_priv, &intel_connector->panel, encoder->devdata, IS_ERR(edid) ? NULL : edid); - intel_panel_add_edid_fixed_modes(intel_connector, - intel_connector->panel.vbt.drrs_type != DRRS_TYPE_NONE || - intel_vrr_is_capable(intel_connector)); + intel_panel_add_edid_fixed_modes(intel_connector, true); /* MSO requires information from the EDID */ intel_edp_mso_init(intel_dp); @@ -5290,30 +5312,18 @@ static bool intel_edp_init_connector(struct intel_dp *intel_dp, if (!intel_panel_preferred_fixed_mode(intel_connector)) intel_panel_add_vbt_lfp_fixed_mode(intel_connector); - mutex_unlock(&dev->mode_config.mutex); + mutex_unlock(&dev_priv->drm.mode_config.mutex); - if (IS_VALLEYVIEW(dev_priv) || IS_CHERRYVIEW(dev_priv)) { - /* - * Figure out the current pipe for the initial backlight setup. - * If the current pipe isn't valid, try the PPS pipe, and if that - * fails just assume pipe A. - */ - pipe = vlv_active_pipe(intel_dp); - - if (pipe != PIPE_A && pipe != PIPE_B) - pipe = intel_dp->pps.pps_pipe; - - if (pipe != PIPE_A && pipe != PIPE_B) - pipe = PIPE_A; - - drm_dbg_kms(&dev_priv->drm, - "using pipe %c for initial backlight setup\n", - pipe_name(pipe)); + if (!intel_panel_preferred_fixed_mode(intel_connector)) { + drm_info(&dev_priv->drm, + "[ENCODER:%d:%s] failed to find fixed mode for the panel, disabling eDP\n", + encoder->base.base.id, encoder->base.name); + goto out_vdd_off; } intel_panel_init(intel_connector); - intel_backlight_setup(intel_connector, pipe); + intel_edp_backlight_setup(intel_dp, intel_connector); intel_edp_add_properties(intel_dp); @@ -5415,7 +5425,6 @@ intel_dp_init_connector(struct intel_digital_port *dig_port, if (!HAS_GMCH(dev_priv)) connector->interlace_allowed = true; - connector->doublescan_allowed = 0; intel_connector->polled = DRM_CONNECTOR_POLL_HPD; diff --git a/drivers/gpu/drm/i915/display/intel_dp_aux.c b/drivers/gpu/drm/i915/display/intel_dp_aux.c index 48c375c65a41..664bebdecea7 100644 --- a/drivers/gpu/drm/i915/display/intel_dp_aux.c +++ b/drivers/gpu/drm/i915/display/intel_dp_aux.c @@ -4,6 +4,7 @@ */ #include "i915_drv.h" +#include "i915_reg.h" #include "i915_trace.h" #include "intel_display_types.h" #include "intel_dp_aux.h" diff --git a/drivers/gpu/drm/i915/display/intel_dp_hdcp.c b/drivers/gpu/drm/i915/display/intel_dp_hdcp.c index 88689124c013..e0c177161407 100644 --- a/drivers/gpu/drm/i915/display/intel_dp_hdcp.c +++ b/drivers/gpu/drm/i915/display/intel_dp_hdcp.c @@ -11,6 +11,7 @@ #include <drm/display/drm_hdcp_helper.h> #include <drm/drm_print.h> +#include "i915_reg.h" #include "intel_ddi.h" #include "intel_de.h" #include "intel_display_types.h" @@ -19,28 +20,20 @@ #include "intel_hdcp.h" #include "intel_hdcp_regs.h" -static unsigned int transcoder_to_stream_enc_status(enum transcoder cpu_transcoder) +static u32 transcoder_to_stream_enc_status(enum transcoder cpu_transcoder) { - u32 stream_enc_mask; - switch (cpu_transcoder) { case TRANSCODER_A: - stream_enc_mask = HDCP_STATUS_STREAM_A_ENC; - break; + return HDCP_STATUS_STREAM_A_ENC; case TRANSCODER_B: - stream_enc_mask = HDCP_STATUS_STREAM_B_ENC; - break; + return HDCP_STATUS_STREAM_B_ENC; case TRANSCODER_C: - stream_enc_mask = HDCP_STATUS_STREAM_C_ENC; - break; + return HDCP_STATUS_STREAM_C_ENC; case TRANSCODER_D: - stream_enc_mask = HDCP_STATUS_STREAM_D_ENC; - break; + return HDCP_STATUS_STREAM_D_ENC; default: - stream_enc_mask = 0; + return 0; } - - return stream_enc_mask; } static void intel_dp_hdcp_wait_for_cp_irq(struct intel_hdcp *hdcp, int timeout) diff --git a/drivers/gpu/drm/i915/display/intel_dp_mst.c b/drivers/gpu/drm/i915/display/intel_dp_mst.c index 03604a37931c..4077a979a924 100644 --- a/drivers/gpu/drm/i915/display/intel_dp_mst.c +++ b/drivers/gpu/drm/i915/display/intel_dp_mst.c @@ -29,6 +29,7 @@ #include <drm/drm_probe_helper.h> #include "i915_drv.h" +#include "i915_reg.h" #include "intel_atomic.h" #include "intel_audio.h" #include "intel_connector.h" @@ -793,7 +794,35 @@ static bool intel_dp_mst_get_hw_state(struct intel_connector *connector) return false; } -static struct drm_connector *intel_dp_add_mst_connector(struct drm_dp_mst_topology_mgr *mgr, struct drm_dp_mst_port *port, const char *pathprop) +static int intel_dp_mst_add_properties(struct intel_dp *intel_dp, + struct drm_connector *connector, + const char *pathprop) +{ + struct drm_i915_private *i915 = to_i915(connector->dev); + + drm_object_attach_property(&connector->base, + i915->drm.mode_config.path_property, 0); + drm_object_attach_property(&connector->base, + i915->drm.mode_config.tile_property, 0); + + intel_attach_force_audio_property(connector); + intel_attach_broadcast_rgb_property(connector); + + /* + * Reuse the prop from the SST connector because we're + * not allowed to create new props after device registration. + */ + connector->max_bpc_property = + intel_dp->attached_connector->base.max_bpc_property; + if (connector->max_bpc_property) + drm_connector_attach_max_bpc_property(connector, 6, 12); + + return drm_connector_set_path_property(connector, pathprop); +} + +static struct drm_connector *intel_dp_add_mst_connector(struct drm_dp_mst_topology_mgr *mgr, + struct drm_dp_mst_port *port, + const char *pathprop) { struct intel_dp *intel_dp = container_of(mgr, struct intel_dp, mst_mgr); struct intel_digital_port *dig_port = dp_to_dig_port(intel_dp); @@ -833,28 +862,14 @@ static struct drm_connector *intel_dp_add_mst_connector(struct drm_dp_mst_topolo goto err; } - drm_object_attach_property(&connector->base, dev->mode_config.path_property, 0); - drm_object_attach_property(&connector->base, dev->mode_config.tile_property, 0); - - ret = drm_connector_set_path_property(connector, pathprop); + ret = intel_dp_mst_add_properties(intel_dp, connector, pathprop); if (ret) goto err; - intel_attach_force_audio_property(connector); - intel_attach_broadcast_rgb_property(connector); - ret = intel_dp_hdcp_init(dig_port, intel_connector); if (ret) drm_dbg_kms(&dev_priv->drm, "[%s:%d] HDCP MST init failed, skipping.\n", connector->name, connector->base.id); - /* - * Reuse the prop from the SST connector because we're - * not allowed to create new props after device registration. - */ - connector->max_bpc_property = - intel_dp->attached_connector->base.max_bpc_property; - if (connector->max_bpc_property) - drm_connector_attach_max_bpc_property(connector, 6, 12); return connector; diff --git a/drivers/gpu/drm/i915/display/intel_dpio_phy.c b/drivers/gpu/drm/i915/display/intel_dpio_phy.c index 8732b8722ed7..7eb7440b3180 100644 --- a/drivers/gpu/drm/i915/display/intel_dpio_phy.c +++ b/drivers/gpu/drm/i915/display/intel_dpio_phy.c @@ -21,6 +21,7 @@ * DEALINGS IN THE SOFTWARE. */ +#include "i915_reg.h" #include "intel_ddi.h" #include "intel_ddi_buf_trans.h" #include "intel_de.h" @@ -655,6 +656,48 @@ bxt_ddi_phy_get_lane_lat_optim_mask(struct intel_encoder *encoder) return mask; } +enum dpio_channel vlv_dig_port_to_channel(struct intel_digital_port *dig_port) +{ + switch (dig_port->base.port) { + default: + MISSING_CASE(dig_port->base.port); + fallthrough; + case PORT_B: + case PORT_D: + return DPIO_CH0; + case PORT_C: + return DPIO_CH1; + } +} + +enum dpio_phy vlv_dig_port_to_phy(struct intel_digital_port *dig_port) +{ + switch (dig_port->base.port) { + default: + MISSING_CASE(dig_port->base.port); + fallthrough; + case PORT_B: + case PORT_C: + return DPIO_PHY0; + case PORT_D: + return DPIO_PHY1; + } +} + +enum dpio_channel vlv_pipe_to_channel(enum pipe pipe) +{ + switch (pipe) { + default: + MISSING_CASE(pipe); + fallthrough; + case PIPE_A: + case PIPE_C: + return DPIO_CH0; + case PIPE_B: + return DPIO_CH1; + } +} + void chv_set_phy_signal_level(struct intel_encoder *encoder, const struct intel_crtc_state *crtc_state, u32 deemph_reg_value, u32 margin_reg_value, diff --git a/drivers/gpu/drm/i915/display/intel_dpio_phy.h b/drivers/gpu/drm/i915/display/intel_dpio_phy.h index 9c3d008e8e1a..9c7725dacb47 100644 --- a/drivers/gpu/drm/i915/display/intel_dpio_phy.h +++ b/drivers/gpu/drm/i915/display/intel_dpio_phy.h @@ -8,13 +8,24 @@ #include <linux/types.h> -enum dpio_channel; -enum dpio_phy; +enum pipe; enum port; struct drm_i915_private; struct intel_crtc_state; +struct intel_digital_port; struct intel_encoder; +enum dpio_channel { + DPIO_CH0, + DPIO_CH1, +}; + +enum dpio_phy { + DPIO_PHY0, + DPIO_PHY1, + DPIO_PHY2, +}; + void bxt_port_to_phy_channel(struct drm_i915_private *dev_priv, enum port port, enum dpio_phy *phy, enum dpio_channel *ch); void bxt_ddi_phy_set_signal_levels(struct intel_encoder *encoder, @@ -30,6 +41,10 @@ void bxt_ddi_phy_set_lane_optim_mask(struct intel_encoder *encoder, u8 lane_lat_optim_mask); u8 bxt_ddi_phy_get_lane_lat_optim_mask(struct intel_encoder *encoder); +enum dpio_channel vlv_dig_port_to_channel(struct intel_digital_port *dig_port); +enum dpio_phy vlv_dig_port_to_phy(struct intel_digital_port *dig_port); +enum dpio_channel vlv_pipe_to_channel(enum pipe pipe); + void chv_set_phy_signal_level(struct intel_encoder *encoder, const struct intel_crtc_state *crtc_state, u32 deemph_reg_value, u32 margin_reg_value, diff --git a/drivers/gpu/drm/i915/display/intel_dpll.c b/drivers/gpu/drm/i915/display/intel_dpll.c index b15ba78d64d6..c236aafe9be0 100644 --- a/drivers/gpu/drm/i915/display/intel_dpll.c +++ b/drivers/gpu/drm/i915/display/intel_dpll.c @@ -6,10 +6,12 @@ #include <linux/kernel.h> #include <linux/string_helpers.h> +#include "i915_reg.h" #include "intel_crtc.h" #include "intel_de.h" #include "intel_display.h" #include "intel_display_types.h" +#include "intel_dpio_phy.h" #include "intel_dpll.h" #include "intel_lvds.h" #include "intel_panel.h" diff --git a/drivers/gpu/drm/i915/display/intel_dpll_mgr.c b/drivers/gpu/drm/i915/display/intel_dpll_mgr.c index 64dd603dc69a..1974eb580ed1 100644 --- a/drivers/gpu/drm/i915/display/intel_dpll_mgr.c +++ b/drivers/gpu/drm/i915/display/intel_dpll_mgr.c @@ -23,15 +23,18 @@ #include <linux/string_helpers.h> +#include "i915_reg.h" #include "intel_de.h" #include "intel_display_types.h" #include "intel_dkl_phy.h" +#include "intel_dkl_phy_regs.h" #include "intel_dpio_phy.h" #include "intel_dpll.h" #include "intel_dpll_mgr.h" +#include "intel_hti.h" +#include "intel_mg_phy_regs.h" #include "intel_pch_refclk.h" #include "intel_tc.h" -#include "intel_tc_phy_regs.h" /** * DOC: Display PLLs @@ -153,28 +156,6 @@ intel_get_shared_dpll_by_id(struct drm_i915_private *dev_priv, return &dev_priv->display.dpll.shared_dplls[id]; } -/** - * intel_get_shared_dpll_id - get the id of a DPLL - * @dev_priv: i915 device instance - * @pll: the DPLL - * - * Returns: - * The id of @pll - */ -enum intel_dpll_id -intel_get_shared_dpll_id(struct drm_i915_private *dev_priv, - struct intel_shared_dpll *pll) -{ - long pll_idx = pll - dev_priv->display.dpll.shared_dplls; - - if (drm_WARN_ON(&dev_priv->drm, - pll_idx < 0 || - pll_idx >= dev_priv->display.dpll.num_shared_dpll)) - return -1; - - return pll_idx; -} - /* For ILK+ */ void assert_shared_dpll(struct drm_i915_private *dev_priv, struct intel_shared_dpll *pll, @@ -385,20 +366,30 @@ intel_reference_shared_dpll(struct intel_atomic_state *state, if (shared_dpll[id].pipe_mask == 0) shared_dpll[id].hw_state = *pll_state; - drm_dbg(&i915->drm, "using %s for pipe %c\n", pll->info->name, - pipe_name(crtc->pipe)); + drm_WARN_ON(&i915->drm, (shared_dpll[id].pipe_mask & BIT(crtc->pipe)) != 0); shared_dpll[id].pipe_mask |= BIT(crtc->pipe); + + drm_dbg_kms(&i915->drm, "[CRTC:%d:%s] reserving %s\n", + crtc->base.base.id, crtc->base.name, pll->info->name); } static void intel_unreference_shared_dpll(struct intel_atomic_state *state, const struct intel_crtc *crtc, const struct intel_shared_dpll *pll) { + struct drm_i915_private *i915 = to_i915(state->base.dev); struct intel_shared_dpll_state *shared_dpll; + const enum intel_dpll_id id = pll->info->id; shared_dpll = intel_atomic_get_shared_dpll_state(&state->base); - shared_dpll[pll->info->id].pipe_mask &= ~BIT(crtc->pipe); + + drm_WARN_ON(&i915->drm, (shared_dpll[id].pipe_mask & BIT(crtc->pipe)) == 0); + + shared_dpll[id].pipe_mask &= ~BIT(crtc->pipe); + + drm_dbg_kms(&i915->drm, "[CRTC:%d:%s] releasing %s\n", + crtc->base.base.id, crtc->base.name, pll->info->name); } static void intel_put_dpll(struct intel_atomic_state *state, @@ -709,8 +700,6 @@ struct hsw_wrpll_rnp { static unsigned hsw_wrpll_get_budget_for_freq(int clock) { - unsigned budget; - switch (clock) { case 25175000: case 25200000: @@ -743,21 +732,18 @@ static unsigned hsw_wrpll_get_budget_for_freq(int clock) case 222750000: case 296703000: case 297000000: - budget = 0; - break; + return 0; case 233500000: case 245250000: case 247750000: case 253250000: case 298000000: - budget = 1500; - break; + return 1500; case 169128000: case 169500000: case 179500000: case 202000000: - budget = 2000; - break; + return 2000; case 256250000: case 262500000: case 270000000: @@ -767,18 +753,13 @@ static unsigned hsw_wrpll_get_budget_for_freq(int clock) case 281250000: case 286000000: case 291750000: - budget = 4000; - break; + return 4000; case 267250000: case 268500000: - budget = 5000; - break; + return 5000; default: - budget = 1000; - break; + return 1000; } - - return budget; } static void hsw_wrpll_update_rnp(u64 freq2k, unsigned int budget, @@ -3184,14 +3165,6 @@ static void icl_update_active_dpll(struct intel_atomic_state *state, icl_set_active_port_dpll(crtc_state, port_dpll_id); } -static u32 intel_get_hti_plls(struct drm_i915_private *i915) -{ - if (!(i915->hti_state & HDPORT_ENABLED)) - return 0; - - return REG_FIELD_GET(HDPORT_DPLL_USED_MASK, i915->hti_state); -} - static int icl_compute_combo_phy_dpll(struct intel_atomic_state *state, struct intel_crtc *crtc) { @@ -3266,7 +3239,7 @@ static int icl_get_combo_phy_dpll(struct intel_atomic_state *state, } /* Eliminate DPLLs from consideration if reserved by HTI */ - dpll_mask &= ~intel_get_hti_plls(dev_priv); + dpll_mask &= ~intel_hti_dpll_mask(dev_priv); port_dpll->pll = intel_find_shared_dpll(state, crtc, &port_dpll->hw_state, @@ -3510,11 +3483,11 @@ static bool dkl_pll_get_hw_state(struct drm_i915_private *dev_priv, * they are on different building blocks */ hw_state->mg_refclkin_ctl = intel_dkl_phy_read(dev_priv, - DKL_REFCLKIN_CTL(tc_port), 2); + DKL_REFCLKIN_CTL(tc_port)); hw_state->mg_refclkin_ctl &= MG_REFCLKIN_CTL_OD_2_MUX_MASK; hw_state->mg_clktop2_hsclkctl = - intel_dkl_phy_read(dev_priv, DKL_CLKTOP2_HSCLKCTL(tc_port), 2); + intel_dkl_phy_read(dev_priv, DKL_CLKTOP2_HSCLKCTL(tc_port)); hw_state->mg_clktop2_hsclkctl &= MG_CLKTOP2_HSCLKCTL_TLINEDRV_CLKSEL_MASK | MG_CLKTOP2_HSCLKCTL_CORE_INPUTSEL_MASK | @@ -3522,32 +3495,32 @@ static bool dkl_pll_get_hw_state(struct drm_i915_private *dev_priv, MG_CLKTOP2_HSCLKCTL_DSDIV_RATIO_MASK; hw_state->mg_clktop2_coreclkctl1 = - intel_dkl_phy_read(dev_priv, DKL_CLKTOP2_CORECLKCTL1(tc_port), 2); + intel_dkl_phy_read(dev_priv, DKL_CLKTOP2_CORECLKCTL1(tc_port)); hw_state->mg_clktop2_coreclkctl1 &= MG_CLKTOP2_CORECLKCTL1_A_DIVRATIO_MASK; - hw_state->mg_pll_div0 = intel_dkl_phy_read(dev_priv, DKL_PLL_DIV0(tc_port), 2); + hw_state->mg_pll_div0 = intel_dkl_phy_read(dev_priv, DKL_PLL_DIV0(tc_port)); val = DKL_PLL_DIV0_MASK; if (dev_priv->display.vbt.override_afc_startup) val |= DKL_PLL_DIV0_AFC_STARTUP_MASK; hw_state->mg_pll_div0 &= val; - hw_state->mg_pll_div1 = intel_dkl_phy_read(dev_priv, DKL_PLL_DIV1(tc_port), 2); + hw_state->mg_pll_div1 = intel_dkl_phy_read(dev_priv, DKL_PLL_DIV1(tc_port)); hw_state->mg_pll_div1 &= (DKL_PLL_DIV1_IREF_TRIM_MASK | DKL_PLL_DIV1_TDC_TARGET_CNT_MASK); - hw_state->mg_pll_ssc = intel_dkl_phy_read(dev_priv, DKL_PLL_SSC(tc_port), 2); + hw_state->mg_pll_ssc = intel_dkl_phy_read(dev_priv, DKL_PLL_SSC(tc_port)); hw_state->mg_pll_ssc &= (DKL_PLL_SSC_IREF_NDIV_RATIO_MASK | DKL_PLL_SSC_STEP_LEN_MASK | DKL_PLL_SSC_STEP_NUM_MASK | DKL_PLL_SSC_EN); - hw_state->mg_pll_bias = intel_dkl_phy_read(dev_priv, DKL_PLL_BIAS(tc_port), 2); + hw_state->mg_pll_bias = intel_dkl_phy_read(dev_priv, DKL_PLL_BIAS(tc_port)); hw_state->mg_pll_bias &= (DKL_PLL_BIAS_FRAC_EN_H | DKL_PLL_BIAS_FBDIV_FRAC_MASK); hw_state->mg_pll_tdc_coldst_bias = - intel_dkl_phy_read(dev_priv, DKL_PLL_TDC_COLDST_BIAS(tc_port), 2); + intel_dkl_phy_read(dev_priv, DKL_PLL_TDC_COLDST_BIAS(tc_port)); hw_state->mg_pll_tdc_coldst_bias &= (DKL_PLL_TDC_SSC_STEP_SIZE_MASK | DKL_PLL_TDC_FEED_FWD_GAIN_MASK); @@ -3736,57 +3709,57 @@ static void dkl_pll_write(struct drm_i915_private *dev_priv, * though on different building block */ /* All the registers are RMW */ - val = intel_dkl_phy_read(dev_priv, DKL_REFCLKIN_CTL(tc_port), 2); + val = intel_dkl_phy_read(dev_priv, DKL_REFCLKIN_CTL(tc_port)); val &= ~MG_REFCLKIN_CTL_OD_2_MUX_MASK; val |= hw_state->mg_refclkin_ctl; - intel_dkl_phy_write(dev_priv, DKL_REFCLKIN_CTL(tc_port), 2, val); + intel_dkl_phy_write(dev_priv, DKL_REFCLKIN_CTL(tc_port), val); - val = intel_dkl_phy_read(dev_priv, DKL_CLKTOP2_CORECLKCTL1(tc_port), 2); + val = intel_dkl_phy_read(dev_priv, DKL_CLKTOP2_CORECLKCTL1(tc_port)); val &= ~MG_CLKTOP2_CORECLKCTL1_A_DIVRATIO_MASK; val |= hw_state->mg_clktop2_coreclkctl1; - intel_dkl_phy_write(dev_priv, DKL_CLKTOP2_CORECLKCTL1(tc_port), 2, val); + intel_dkl_phy_write(dev_priv, DKL_CLKTOP2_CORECLKCTL1(tc_port), val); - val = intel_dkl_phy_read(dev_priv, DKL_CLKTOP2_HSCLKCTL(tc_port), 2); + val = intel_dkl_phy_read(dev_priv, DKL_CLKTOP2_HSCLKCTL(tc_port)); val &= ~(MG_CLKTOP2_HSCLKCTL_TLINEDRV_CLKSEL_MASK | MG_CLKTOP2_HSCLKCTL_CORE_INPUTSEL_MASK | MG_CLKTOP2_HSCLKCTL_HSDIV_RATIO_MASK | MG_CLKTOP2_HSCLKCTL_DSDIV_RATIO_MASK); val |= hw_state->mg_clktop2_hsclkctl; - intel_dkl_phy_write(dev_priv, DKL_CLKTOP2_HSCLKCTL(tc_port), 2, val); + intel_dkl_phy_write(dev_priv, DKL_CLKTOP2_HSCLKCTL(tc_port), val); val = DKL_PLL_DIV0_MASK; if (dev_priv->display.vbt.override_afc_startup) val |= DKL_PLL_DIV0_AFC_STARTUP_MASK; - intel_dkl_phy_rmw(dev_priv, DKL_PLL_DIV0(tc_port), 2, val, + intel_dkl_phy_rmw(dev_priv, DKL_PLL_DIV0(tc_port), val, hw_state->mg_pll_div0); - val = intel_dkl_phy_read(dev_priv, DKL_PLL_DIV1(tc_port), 2); + val = intel_dkl_phy_read(dev_priv, DKL_PLL_DIV1(tc_port)); val &= ~(DKL_PLL_DIV1_IREF_TRIM_MASK | DKL_PLL_DIV1_TDC_TARGET_CNT_MASK); val |= hw_state->mg_pll_div1; - intel_dkl_phy_write(dev_priv, DKL_PLL_DIV1(tc_port), 2, val); + intel_dkl_phy_write(dev_priv, DKL_PLL_DIV1(tc_port), val); - val = intel_dkl_phy_read(dev_priv, DKL_PLL_SSC(tc_port), 2); + val = intel_dkl_phy_read(dev_priv, DKL_PLL_SSC(tc_port)); val &= ~(DKL_PLL_SSC_IREF_NDIV_RATIO_MASK | DKL_PLL_SSC_STEP_LEN_MASK | DKL_PLL_SSC_STEP_NUM_MASK | DKL_PLL_SSC_EN); val |= hw_state->mg_pll_ssc; - intel_dkl_phy_write(dev_priv, DKL_PLL_SSC(tc_port), 2, val); + intel_dkl_phy_write(dev_priv, DKL_PLL_SSC(tc_port), val); - val = intel_dkl_phy_read(dev_priv, DKL_PLL_BIAS(tc_port), 2); + val = intel_dkl_phy_read(dev_priv, DKL_PLL_BIAS(tc_port)); val &= ~(DKL_PLL_BIAS_FRAC_EN_H | DKL_PLL_BIAS_FBDIV_FRAC_MASK); val |= hw_state->mg_pll_bias; - intel_dkl_phy_write(dev_priv, DKL_PLL_BIAS(tc_port), 2, val); + intel_dkl_phy_write(dev_priv, DKL_PLL_BIAS(tc_port), val); - val = intel_dkl_phy_read(dev_priv, DKL_PLL_TDC_COLDST_BIAS(tc_port), 2); + val = intel_dkl_phy_read(dev_priv, DKL_PLL_TDC_COLDST_BIAS(tc_port)); val &= ~(DKL_PLL_TDC_SSC_STEP_SIZE_MASK | DKL_PLL_TDC_FEED_FWD_GAIN_MASK); val |= hw_state->mg_pll_tdc_coldst_bias; - intel_dkl_phy_write(dev_priv, DKL_PLL_TDC_COLDST_BIAS(tc_port), 2, val); + intel_dkl_phy_write(dev_priv, DKL_PLL_TDC_COLDST_BIAS(tc_port), val); - intel_dkl_phy_posting_read(dev_priv, DKL_PLL_TDC_COLDST_BIAS(tc_port), 2); + intel_dkl_phy_posting_read(dev_priv, DKL_PLL_TDC_COLDST_BIAS(tc_port)); } static void icl_pll_power_enable(struct drm_i915_private *dev_priv, @@ -4188,6 +4161,8 @@ void intel_shared_dpll_init(struct drm_i915_private *dev_priv) const struct dpll_info *dpll_info; int i; + mutex_init(&dev_priv->display.dpll.lock); + if (IS_DG2(dev_priv)) /* No shared DPLLs on DG2; port PLLs are part of the PHY */ dpll_mgr = NULL; @@ -4232,7 +4207,6 @@ void intel_shared_dpll_init(struct drm_i915_private *dev_priv) dev_priv->display.dpll.mgr = dpll_mgr; dev_priv->display.dpll.num_shared_dpll = i; - mutex_init(&dev_priv->display.dpll.lock); } /** diff --git a/drivers/gpu/drm/i915/display/intel_dpll_mgr.h b/drivers/gpu/drm/i915/display/intel_dpll_mgr.h index 3247dc300ae4..3854f1b4299a 100644 --- a/drivers/gpu/drm/i915/display/intel_dpll_mgr.h +++ b/drivers/gpu/drm/i915/display/intel_dpll_mgr.h @@ -328,9 +328,6 @@ struct intel_shared_dpll { struct intel_shared_dpll * intel_get_shared_dpll_by_id(struct drm_i915_private *dev_priv, enum intel_dpll_id id); -enum intel_dpll_id -intel_get_shared_dpll_id(struct drm_i915_private *dev_priv, - struct intel_shared_dpll *pll); void assert_shared_dpll(struct drm_i915_private *dev_priv, struct intel_shared_dpll *pll, bool state); diff --git a/drivers/gpu/drm/i915/display/intel_dpt.c b/drivers/gpu/drm/i915/display/intel_dpt.c index ac587647e1f5..ad1a37b515fb 100644 --- a/drivers/gpu/drm/i915/display/intel_dpt.c +++ b/drivers/gpu/drm/i915/display/intel_dpt.c @@ -5,6 +5,7 @@ #include "gem/i915_gem_domain.h" #include "gem/i915_gem_internal.h" +#include "gem/i915_gem_lmem.h" #include "gt/gen8_ppgtt.h" #include "i915_drv.h" diff --git a/drivers/gpu/drm/i915/display/intel_drrs.c b/drivers/gpu/drm/i915/display/intel_drrs.c index 7da4a9cbe4ba..5b9e44443814 100644 --- a/drivers/gpu/drm/i915/display/intel_drrs.c +++ b/drivers/gpu/drm/i915/display/intel_drrs.c @@ -4,6 +4,7 @@ */ #include "i915_drv.h" +#include "i915_reg.h" #include "intel_atomic.h" #include "intel_de.h" #include "intel_display_types.h" @@ -284,16 +285,124 @@ void intel_drrs_flush(struct drm_i915_private *dev_priv, } /** - * intel_crtc_drrs_init - Init DRRS for CRTC + * intel_drrs_crtc_init - Init DRRS for CRTC * @crtc: crtc * * This function is called only once at driver load to initialize basic * DRRS stuff. * */ -void intel_crtc_drrs_init(struct intel_crtc *crtc) +void intel_drrs_crtc_init(struct intel_crtc *crtc) { INIT_DELAYED_WORK(&crtc->drrs.work, intel_drrs_downclock_work); mutex_init(&crtc->drrs.mutex); crtc->drrs.cpu_transcoder = INVALID_TRANSCODER; } + +static int intel_drrs_debugfs_status_show(struct seq_file *m, void *unused) +{ + struct intel_crtc *crtc = m->private; + const struct intel_crtc_state *crtc_state; + int ret; + + ret = drm_modeset_lock_single_interruptible(&crtc->base.mutex); + if (ret) + return ret; + + crtc_state = to_intel_crtc_state(crtc->base.state); + + mutex_lock(&crtc->drrs.mutex); + + seq_printf(m, "DRRS enabled: %s\n", + str_yes_no(crtc_state->has_drrs)); + + seq_printf(m, "DRRS active: %s\n", + str_yes_no(intel_drrs_is_active(crtc))); + + seq_printf(m, "DRRS refresh rate: %s\n", + crtc->drrs.refresh_rate == DRRS_REFRESH_RATE_LOW ? + "low" : "high"); + + seq_printf(m, "DRRS busy frontbuffer bits: 0x%x\n", + crtc->drrs.busy_frontbuffer_bits); + + mutex_unlock(&crtc->drrs.mutex); + + drm_modeset_unlock(&crtc->base.mutex); + + return 0; +} + +DEFINE_SHOW_ATTRIBUTE(intel_drrs_debugfs_status); + +static int intel_drrs_debugfs_ctl_set(void *data, u64 val) +{ + struct intel_crtc *crtc = data; + struct drm_i915_private *i915 = to_i915(crtc->base.dev); + struct intel_crtc_state *crtc_state; + struct drm_crtc_commit *commit; + int ret; + + ret = drm_modeset_lock_single_interruptible(&crtc->base.mutex); + if (ret) + return ret; + + crtc_state = to_intel_crtc_state(crtc->base.state); + + if (!crtc_state->hw.active || + !crtc_state->has_drrs) + goto out; + + commit = crtc_state->uapi.commit; + if (commit) { + ret = wait_for_completion_interruptible(&commit->hw_done); + if (ret) + goto out; + } + + drm_dbg(&i915->drm, + "Manually %sactivating DRRS\n", val ? "" : "de"); + + if (val) + intel_drrs_activate(crtc_state); + else + intel_drrs_deactivate(crtc_state); + +out: + drm_modeset_unlock(&crtc->base.mutex); + + return ret; +} + +DEFINE_SIMPLE_ATTRIBUTE(intel_drrs_debugfs_ctl_fops, + NULL, intel_drrs_debugfs_ctl_set, "%llu\n"); + +void intel_drrs_crtc_debugfs_add(struct intel_crtc *crtc) +{ + debugfs_create_file("i915_drrs_status", 0444, crtc->base.debugfs_entry, + crtc, &intel_drrs_debugfs_status_fops); + + debugfs_create_file("i915_drrs_ctl", 0644, crtc->base.debugfs_entry, + crtc, &intel_drrs_debugfs_ctl_fops); +} + +static int intel_drrs_debugfs_type_show(struct seq_file *m, void *unused) +{ + struct intel_connector *connector = m->private; + + seq_printf(m, "DRRS type: %s\n", + intel_drrs_type_str(intel_panel_drrs_type(connector))); + + return 0; +} + +DEFINE_SHOW_ATTRIBUTE(intel_drrs_debugfs_type); + +void intel_drrs_connector_debugfs_add(struct intel_connector *connector) +{ + if (intel_panel_drrs_type(connector) == DRRS_TYPE_NONE) + return; + + debugfs_create_file("i915_drrs_type", 0444, connector->base.debugfs_entry, + connector, &intel_drrs_debugfs_type_fops); +} diff --git a/drivers/gpu/drm/i915/display/intel_drrs.h b/drivers/gpu/drm/i915/display/intel_drrs.h index 3ad1be1ad9c1..8ef5f93a80ff 100644 --- a/drivers/gpu/drm/i915/display/intel_drrs.h +++ b/drivers/gpu/drm/i915/display/intel_drrs.h @@ -23,6 +23,8 @@ void intel_drrs_invalidate(struct drm_i915_private *dev_priv, unsigned int frontbuffer_bits); void intel_drrs_flush(struct drm_i915_private *dev_priv, unsigned int frontbuffer_bits); -void intel_crtc_drrs_init(struct intel_crtc *crtc); +void intel_drrs_crtc_init(struct intel_crtc *crtc); +void intel_drrs_crtc_debugfs_add(struct intel_crtc *crtc); +void intel_drrs_connector_debugfs_add(struct intel_connector *connector); #endif /* __INTEL_DRRS_H__ */ diff --git a/drivers/gpu/drm/i915/display/intel_dsb.c b/drivers/gpu/drm/i915/display/intel_dsb.c index fc9c3e41c333..1e1c6107d51b 100644 --- a/drivers/gpu/drm/i915/display/intel_dsb.c +++ b/drivers/gpu/drm/i915/display/intel_dsb.c @@ -7,6 +7,7 @@ #include "gem/i915_gem_internal.h" #include "i915_drv.h" +#include "i915_reg.h" #include "intel_de.h" #include "intel_display_types.h" #include "intel_dsb.h" diff --git a/drivers/gpu/drm/i915/display/intel_dsi_vbt.c b/drivers/gpu/drm/i915/display/intel_dsi_vbt.c index 75e8cc4337c9..fce69fa446d5 100644 --- a/drivers/gpu/drm/i915/display/intel_dsi_vbt.c +++ b/drivers/gpu/drm/i915/display/intel_dsi_vbt.c @@ -137,9 +137,9 @@ static enum port intel_dsi_seq_port_to_port(struct intel_dsi *intel_dsi, return ffs(intel_dsi->ports) - 1; if (seq_port) { - if (intel_dsi->ports & PORT_B) + if (intel_dsi->ports & BIT(PORT_B)) return PORT_B; - else if (intel_dsi->ports & PORT_C) + else if (intel_dsi->ports & BIT(PORT_C)) return PORT_C; } diff --git a/drivers/gpu/drm/i915/display/intel_dvo.c b/drivers/gpu/drm/i915/display/intel_dvo.c index 5572e43026e4..c86f9890754d 100644 --- a/drivers/gpu/drm/i915/display/intel_dvo.c +++ b/drivers/gpu/drm/i915/display/intel_dvo.c @@ -32,6 +32,7 @@ #include <drm/drm_crtc.h> #include "i915_drv.h" +#include "i915_reg.h" #include "intel_connector.h" #include "intel_de.h" #include "intel_display_types.h" @@ -491,8 +492,8 @@ void intel_dvo_init(struct drm_i915_private *dev_priv) intel_encoder->pipe_mask = ~0; if (dvo->type != INTEL_DVO_CHIP_LVDS) - intel_encoder->cloneable = (1 << INTEL_OUTPUT_ANALOG) | - (1 << INTEL_OUTPUT_DVO); + intel_encoder->cloneable = BIT(INTEL_OUTPUT_ANALOG) | + BIT(INTEL_OUTPUT_DVO); switch (dvo->type) { case INTEL_DVO_CHIP_TMDS: @@ -515,8 +516,6 @@ void intel_dvo_init(struct drm_i915_private *dev_priv) drm_connector_helper_add(connector, &intel_dvo_connector_helper_funcs); connector->display_info.subpixel_order = SubPixelHorizontalRGB; - connector->interlace_allowed = false; - connector->doublescan_allowed = false; intel_connector_attach_encoder(intel_connector, intel_encoder); if (dvo->type == INTEL_DVO_CHIP_LVDS) { diff --git a/drivers/gpu/drm/i915/display/intel_dvo_dev.h b/drivers/gpu/drm/i915/display/intel_dvo_dev.h index 50205f064d93..ecff7b190856 100644 --- a/drivers/gpu/drm/i915/display/intel_dvo_dev.h +++ b/drivers/gpu/drm/i915/display/intel_dvo_dev.h @@ -23,12 +23,12 @@ #ifndef __INTEL_DVO_DEV_H__ #define __INTEL_DVO_DEV_H__ -#include <linux/i2c.h> - -#include <drm/drm_crtc.h> - #include "i915_reg_defs.h" +enum drm_connector_status; +struct drm_display_mode; +struct i2c_adapter; + struct intel_dvo_device { const char *name; int type; diff --git a/drivers/gpu/drm/i915/display/intel_fb.c b/drivers/gpu/drm/i915/display/intel_fb.c index eefa33c555ac..63137ae5ab21 100644 --- a/drivers/gpu/drm/i915/display/intel_fb.c +++ b/drivers/gpu/drm/i915/display/intel_fb.c @@ -301,6 +301,19 @@ static bool plane_caps_contain_all(u8 caps, u8 mask) } /** + * intel_fb_is_tiled_modifier: Check if a modifier is a tiled modifier type + * @modifier: Modifier to check + * + * Returns: + * Returns %true if @modifier is a tiled modifier. + */ +bool intel_fb_is_tiled_modifier(u64 modifier) +{ + return plane_caps_contain_any(lookup_modifier(modifier)->plane_caps, + INTEL_PLANE_CAP_TILING_MASK); +} + +/** * intel_fb_is_ccs_modifier: Check if a modifier is a CCS modifier type * @modifier: Modifier to check * diff --git a/drivers/gpu/drm/i915/display/intel_fb.h b/drivers/gpu/drm/i915/display/intel_fb.h index 12386f13a4e0..4662b812b934 100644 --- a/drivers/gpu/drm/i915/display/intel_fb.h +++ b/drivers/gpu/drm/i915/display/intel_fb.h @@ -29,6 +29,7 @@ struct intel_plane_state; #define INTEL_PLANE_CAP_TILING_Yf BIT(5) #define INTEL_PLANE_CAP_TILING_4 BIT(6) +bool intel_fb_is_tiled_modifier(u64 modifier); bool intel_fb_is_ccs_modifier(u64 modifier); bool intel_fb_is_rc_ccs_cc_modifier(u64 modifier); bool intel_fb_is_mc_ccs_modifier(u64 modifier); diff --git a/drivers/gpu/drm/i915/display/intel_fb_pin.c b/drivers/gpu/drm/i915/display/intel_fb_pin.c index 1dddd6abd77b..6900acbb1381 100644 --- a/drivers/gpu/drm/i915/display/intel_fb_pin.c +++ b/drivers/gpu/drm/i915/display/intel_fb_pin.c @@ -167,7 +167,6 @@ retry: ret = i915_gem_object_attach_phys(obj, alignment); else if (!ret && HAS_LMEM(dev_priv)) ret = i915_gem_object_migrate(obj, &ww, INTEL_REGION_LMEM_0); - /* TODO: Do we need to sync when migration becomes async? */ if (!ret) ret = i915_gem_object_pin_pages(obj); if (ret) diff --git a/drivers/gpu/drm/i915/display/intel_fbc.c b/drivers/gpu/drm/i915/display/intel_fbc.c index f38175304928..b5ee5ea0d010 100644 --- a/drivers/gpu/drm/i915/display/intel_fbc.c +++ b/drivers/gpu/drm/i915/display/intel_fbc.c @@ -670,6 +670,7 @@ static void intel_fbc_nuke(struct intel_fbc *fbc) { struct drm_i915_private *i915 = fbc->i915; + lockdep_assert_held(&fbc->lock); drm_WARN_ON(&i915->drm, fbc->flip_pending); trace_intel_fbc_nuke(fbc->state.plane); @@ -679,6 +680,8 @@ static void intel_fbc_nuke(struct intel_fbc *fbc) static void intel_fbc_activate(struct intel_fbc *fbc) { + lockdep_assert_held(&fbc->lock); + intel_fbc_hw_activate(fbc); intel_fbc_nuke(fbc); @@ -687,9 +690,7 @@ static void intel_fbc_activate(struct intel_fbc *fbc) static void intel_fbc_deactivate(struct intel_fbc *fbc, const char *reason) { - struct drm_i915_private *i915 = fbc->i915; - - drm_WARN_ON(&i915->drm, !mutex_is_locked(&fbc->lock)); + lockdep_assert_held(&fbc->lock); if (fbc->active) intel_fbc_hw_deactivate(fbc); @@ -1009,7 +1010,8 @@ static bool intel_fbc_is_fence_ok(const struct intel_plane_state *plane_state) { struct drm_i915_private *i915 = to_i915(plane_state->uapi.plane->dev); - /* The use of a CPU fence is one of two ways to detect writes by the + /* + * The use of a CPU fence is one of two ways to detect writes by the * CPU to the scanout and trigger updates to the FBC. * * The other method is by software tracking (see @@ -1019,12 +1021,6 @@ static bool intel_fbc_is_fence_ok(const struct intel_plane_state *plane_state) * Note that is possible for a tiled surface to be unmappable (and * so have no fence associated with it) due to aperture constraints * at the time of pinning. - * - * FIXME with 90/270 degree rotation we should use the fence on - * the normal GTT view (the rotated view doesn't even have a - * fence). Would need changes to the FBC fence Y offset as well. - * For now this will effectively disable FBC with 90/270 degree - * rotation. */ return DISPLAY_VER(i915) >= 9 || (plane_state->flags & PLANE_HAS_FENCE && @@ -1187,7 +1183,7 @@ static bool intel_fbc_can_flip_nuke(struct intel_atomic_state *state, const struct drm_framebuffer *old_fb = old_plane_state->hw.fb; const struct drm_framebuffer *new_fb = new_plane_state->hw.fb; - if (drm_atomic_crtc_needs_modeset(&new_crtc_state->uapi)) + if (intel_crtc_needs_modeset(new_crtc_state)) return false; if (!intel_fbc_is_ok(old_plane_state) || @@ -1227,6 +1223,8 @@ static bool __intel_fbc_pre_update(struct intel_atomic_state *state, struct intel_fbc *fbc = plane->fbc; bool need_vblank_wait = false; + lockdep_assert_held(&fbc->lock); + fbc->flip_pending = true; if (intel_fbc_can_flip_nuke(state, crtc, plane)) @@ -1284,7 +1282,7 @@ static void __intel_fbc_disable(struct intel_fbc *fbc) struct drm_i915_private *i915 = fbc->i915; struct intel_plane *plane = fbc->state.plane; - drm_WARN_ON(&i915->drm, !mutex_is_locked(&fbc->lock)); + lockdep_assert_held(&fbc->lock); drm_WARN_ON(&i915->drm, fbc->active); drm_dbg_kms(&i915->drm, "Disabling FBC on [PLANE:%d:%s]\n", @@ -1299,9 +1297,9 @@ static void __intel_fbc_disable(struct intel_fbc *fbc) static void __intel_fbc_post_update(struct intel_fbc *fbc) { - struct drm_i915_private *i915 = fbc->i915; + lockdep_assert_held(&fbc->lock); - drm_WARN_ON(&i915->drm, !mutex_is_locked(&fbc->lock)); + fbc->flip_pending = false; if (!fbc->busy_bits) intel_fbc_activate(fbc); @@ -1324,10 +1322,8 @@ void intel_fbc_post_update(struct intel_atomic_state *state, mutex_lock(&fbc->lock); - if (fbc->state.plane == plane) { - fbc->flip_pending = false; + if (fbc->state.plane == plane) __intel_fbc_post_update(fbc); - } mutex_unlock(&fbc->lock); } @@ -1437,6 +1433,8 @@ static void __intel_fbc_enable(struct intel_atomic_state *state, intel_atomic_get_new_plane_state(state, plane); struct intel_fbc *fbc = plane->fbc; + lockdep_assert_held(&fbc->lock); + if (fbc->state.plane) { if (fbc->state.plane != plane) return; @@ -1522,7 +1520,8 @@ void intel_fbc_update(struct intel_atomic_state *state, mutex_lock(&fbc->lock); - if (crtc_state->update_pipe && plane_state->no_fbc_reason) { + if (intel_crtc_needs_fastset(crtc_state) && + plane_state->no_fbc_reason) { if (fbc->state.plane == plane) __intel_fbc_disable(fbc); } else { diff --git a/drivers/gpu/drm/i915/display/intel_fbdev.c b/drivers/gpu/drm/i915/display/intel_fbdev.c index 112aa0447a0d..5575d7abdc09 100644 --- a/drivers/gpu/drm/i915/display/intel_fbdev.c +++ b/drivers/gpu/drm/i915/display/intel_fbdev.c @@ -124,6 +124,8 @@ static const struct fb_ops intelfb_ops = { .owner = THIS_MODULE, DRM_FB_HELPER_DEFAULT_OPS, .fb_set_par = intel_fbdev_set_par, + .fb_read = drm_fb_helper_cfb_read, + .fb_write = drm_fb_helper_cfb_write, .fb_fillrect = drm_fb_helper_cfb_fillrect, .fb_copyarea = drm_fb_helper_cfb_copyarea, .fb_imageblit = drm_fb_helper_cfb_imageblit, @@ -175,7 +177,7 @@ static int intelfb_alloc(struct drm_fb_helper *helper, } if (IS_ERR(obj)) { - drm_err(&dev_priv->drm, "failed to allocate framebuffer\n"); + drm_err(&dev_priv->drm, "failed to allocate framebuffer (%pe)\n", obj); return PTR_ERR(obj); } @@ -254,9 +256,9 @@ static int intelfb_create(struct drm_fb_helper *helper, goto out_unlock; } - info = drm_fb_helper_alloc_fbi(helper); + info = drm_fb_helper_alloc_info(helper); if (IS_ERR(info)) { - drm_err(&dev_priv->drm, "Failed to allocate fb_info\n"); + drm_err(&dev_priv->drm, "Failed to allocate fb_info (%pe)\n", info); ret = PTR_ERR(info); goto out_unpin; } @@ -291,7 +293,7 @@ static int intelfb_create(struct drm_fb_helper *helper, vaddr = i915_vma_pin_iomap(vma); if (IS_ERR(vaddr)) { drm_err(&dev_priv->drm, - "Failed to remap framebuffer into virtual memory\n"); + "Failed to remap framebuffer into virtual memory (%pe)\n", vaddr); ret = PTR_ERR(vaddr); goto out_unpin; } @@ -584,7 +586,7 @@ void intel_fbdev_unregister(struct drm_i915_private *dev_priv) if (!current_is_async()) intel_fbdev_sync(ifbdev); - drm_fb_helper_unregister_fbi(&ifbdev->helper); + drm_fb_helper_unregister_info(&ifbdev->helper); } void intel_fbdev_fini(struct drm_i915_private *dev_priv) @@ -627,7 +629,7 @@ void intel_fbdev_set_suspend(struct drm_device *dev, int state, bool synchronous if (!ifbdev || !ifbdev->vma) goto set_suspend; - info = ifbdev->helper.fbdev; + info = ifbdev->helper.info; if (synchronous) { /* Flush any pending work to turn the console on, and then diff --git a/drivers/gpu/drm/i915/display/intel_fdi.c b/drivers/gpu/drm/i915/display/intel_fdi.c index 7f47e5c85c81..063f1da4f229 100644 --- a/drivers/gpu/drm/i915/display/intel_fdi.c +++ b/drivers/gpu/drm/i915/display/intel_fdi.c @@ -5,6 +5,7 @@ #include <linux/string_helpers.h> +#include "i915_reg.h" #include "intel_atomic.h" #include "intel_crtc.h" #include "intel_ddi.h" diff --git a/drivers/gpu/drm/i915/display/intel_fifo_underrun.h b/drivers/gpu/drm/i915/display/intel_fifo_underrun.h index e04f22ac1f49..2e47d7d3c101 100644 --- a/drivers/gpu/drm/i915/display/intel_fifo_underrun.h +++ b/drivers/gpu/drm/i915/display/intel_fifo_underrun.h @@ -8,9 +8,8 @@ #include <linux/types.h> -#include "intel_display.h" - struct drm_i915_private; +enum pipe; bool intel_set_cpu_fifo_underrun_reporting(struct drm_i915_private *dev_priv, enum pipe pipe, bool enable); diff --git a/drivers/gpu/drm/i915/display/intel_frontbuffer.c b/drivers/gpu/drm/i915/display/intel_frontbuffer.c index d80e3e8a9b01..17a7aa8b28c2 100644 --- a/drivers/gpu/drm/i915/display/intel_frontbuffer.c +++ b/drivers/gpu/drm/i915/display/intel_frontbuffer.c @@ -88,7 +88,7 @@ static void frontbuffer_flush(struct drm_i915_private *i915, if (!frontbuffer_bits) return; - trace_intel_frontbuffer_flush(frontbuffer_bits, origin); + trace_intel_frontbuffer_flush(i915, frontbuffer_bits, origin); might_sleep(); intel_drrs_flush(i915, frontbuffer_bits); @@ -176,7 +176,7 @@ void __intel_fb_invalidate(struct intel_frontbuffer *front, spin_unlock(&i915->display.fb_tracking.lock); } - trace_intel_frontbuffer_invalidate(frontbuffer_bits, origin); + trace_intel_frontbuffer_invalidate(i915, frontbuffer_bits, origin); might_sleep(); intel_psr_invalidate(i915, frontbuffer_bits, origin); diff --git a/drivers/gpu/drm/i915/display/intel_global_state.c b/drivers/gpu/drm/i915/display/intel_global_state.c index 7a19215ad844..02b593b1e2ea 100644 --- a/drivers/gpu/drm/i915/display/intel_global_state.c +++ b/drivers/gpu/drm/i915/display/intel_global_state.c @@ -45,14 +45,14 @@ void intel_atomic_global_obj_init(struct drm_i915_private *dev_priv, obj->state = state; obj->funcs = funcs; - list_add_tail(&obj->head, &dev_priv->global_obj_list); + list_add_tail(&obj->head, &dev_priv->display.global.obj_list); } void intel_atomic_global_obj_cleanup(struct drm_i915_private *dev_priv) { struct intel_global_obj *obj, *next; - list_for_each_entry_safe(obj, next, &dev_priv->global_obj_list, head) { + list_for_each_entry_safe(obj, next, &dev_priv->display.global.obj_list, head) { list_del(&obj->head); drm_WARN_ON(&dev_priv->drm, kref_read(&obj->state->ref) != 1); diff --git a/drivers/gpu/drm/i915/display/intel_global_state.h b/drivers/gpu/drm/i915/display/intel_global_state.h index 1f16fa3073c9..f01ee0bb3e5a 100644 --- a/drivers/gpu/drm/i915/display/intel_global_state.h +++ b/drivers/gpu/drm/i915/display/intel_global_state.h @@ -27,7 +27,7 @@ struct intel_global_obj { }; #define intel_for_each_global_obj(obj, dev_priv) \ - list_for_each_entry(obj, &(dev_priv)->global_obj_list, head) + list_for_each_entry(obj, &(dev_priv)->display.global.obj_list, head) #define for_each_new_global_obj_in_state(__state, obj, new_obj_state, __i) \ for ((__i) = 0; \ diff --git a/drivers/gpu/drm/i915/display/intel_gmbus.c b/drivers/gpu/drm/i915/display/intel_gmbus.c index 74443f57f62d..a5840a28a69d 100644 --- a/drivers/gpu/drm/i915/display/intel_gmbus.c +++ b/drivers/gpu/drm/i915/display/intel_gmbus.c @@ -34,6 +34,8 @@ #include <drm/display/drm_hdcp_helper.h> #include "i915_drv.h" +#include "i915_irq.h" +#include "i915_reg.h" #include "intel_de.h" #include "intel_display_types.h" #include "intel_gmbus.h" @@ -49,9 +51,27 @@ struct intel_gmbus { struct drm_i915_private *i915; }; +enum gmbus_gpio { + GPIOA, + GPIOB, + GPIOC, + GPIOD, + GPIOE, + GPIOF, + GPIOG, + GPIOH, + __GPIOI_UNUSED, + GPIOJ, + GPIOK, + GPIOL, + GPIOM, + GPION, + GPIOO, +}; + struct gmbus_pin { const char *name; - enum i915_gpio gpio; + enum gmbus_gpio gpio; }; /* Map gmbus pin pairs to names and registers. */ diff --git a/drivers/gpu/drm/i915/display/intel_hdcp_regs.h b/drivers/gpu/drm/i915/display/intel_hdcp_regs.h index 2a3733e8966c..8023c85c7fa0 100644 --- a/drivers/gpu/drm/i915/display/intel_hdcp_regs.h +++ b/drivers/gpu/drm/i915/display/intel_hdcp_regs.h @@ -6,7 +6,7 @@ #ifndef __INTEL_HDCP_REGS_H__ #define __INTEL_HDCP_REGS_H__ -#include "i915_reg_defs.h" +#include "intel_display_reg_defs.h" /* HDCP Key Registers */ #define HDCP_KEY_CONF _MMIO(0x66c00) diff --git a/drivers/gpu/drm/i915/display/intel_hdmi.c b/drivers/gpu/drm/i915/display/intel_hdmi.c index 7816b2a33fee..bac85d88054f 100644 --- a/drivers/gpu/drm/i915/display/intel_hdmi.c +++ b/drivers/gpu/drm/i915/display/intel_hdmi.c @@ -42,6 +42,7 @@ #include "i915_debugfs.h" #include "i915_drv.h" +#include "i915_reg.h" #include "intel_atomic.h" #include "intel_connector.h" #include "intel_ddi.h" @@ -2057,13 +2058,6 @@ static bool hdmi_bpc_possible(const struct intel_crtc_state *crtc_state, int bpc if (!intel_hdmi_source_bpc_possible(dev_priv, bpc)) return false; - /* - * HDMI deep color affects the clocks, so it's only possible - * when not cloning with other encoder types. - */ - if (bpc > 8 && crtc_state->output_types != BIT(INTEL_OUTPUT_HDMI)) - return false; - /* Display Wa_1405510057:icl,ehl */ if (intel_hdmi_is_ycbcr420(crtc_state) && bpc == 10 && DISPLAY_VER(dev_priv) == 11 && @@ -2190,9 +2184,13 @@ static bool intel_hdmi_has_audio(struct intel_encoder *encoder, } static enum intel_output_format -intel_hdmi_output_format(struct intel_connector *connector, +intel_hdmi_output_format(const struct intel_crtc_state *crtc_state, + struct intel_connector *connector, bool ycbcr_420_output) { + if (!crtc_state->has_hdmi_sink) + return INTEL_OUTPUT_FORMAT_RGB; + if (connector->base.ycbcr_420_allowed && ycbcr_420_output) return INTEL_OUTPUT_FORMAT_YCBCR420; else @@ -2211,7 +2209,8 @@ static int intel_hdmi_compute_output_format(struct intel_encoder *encoder, bool ycbcr_420_only = drm_mode_is_420_only(info, adjusted_mode); int ret; - crtc_state->output_format = intel_hdmi_output_format(connector, ycbcr_420_only); + crtc_state->output_format = + intel_hdmi_output_format(crtc_state, connector, ycbcr_420_only); if (ycbcr_420_only && !intel_hdmi_is_ycbcr420(crtc_state)) { drm_dbg_kms(&i915->drm, @@ -2226,13 +2225,19 @@ static int intel_hdmi_compute_output_format(struct intel_encoder *encoder, !drm_mode_is_420_also(info, adjusted_mode)) return ret; - crtc_state->output_format = intel_hdmi_output_format(connector, true); + crtc_state->output_format = intel_hdmi_output_format(crtc_state, connector, true); ret = intel_hdmi_compute_clock(encoder, crtc_state, respect_downstream_limits); } return ret; } +static bool intel_hdmi_is_cloned(const struct intel_crtc_state *crtc_state) +{ + return crtc_state->uapi.encoder_mask && + !is_power_of_2(crtc_state->uapi.encoder_mask); +} + int intel_hdmi_compute_config(struct intel_encoder *encoder, struct intel_crtc_state *pipe_config, struct drm_connector_state *conn_state) @@ -2248,8 +2253,9 @@ int intel_hdmi_compute_config(struct intel_encoder *encoder, return -EINVAL; pipe_config->output_format = INTEL_OUTPUT_FORMAT_RGB; - pipe_config->has_hdmi_sink = intel_has_hdmi_sink(intel_hdmi, - conn_state); + pipe_config->has_hdmi_sink = + intel_has_hdmi_sink(intel_hdmi, conn_state) && + !intel_hdmi_is_cloned(pipe_config); if (pipe_config->has_hdmi_sink) pipe_config->has_infoframe = true; @@ -2257,9 +2263,6 @@ int intel_hdmi_compute_config(struct intel_encoder *encoder, if (adjusted_mode->flags & DRM_MODE_FLAG_DBLCLK) pipe_config->pixel_multiplier = 2; - if (HAS_PCH_SPLIT(dev_priv) && !HAS_DDI(dev_priv)) - pipe_config->has_pch_encoder = true; - pipe_config->has_audio = intel_hdmi_has_audio(encoder, pipe_config, conn_state); @@ -2355,7 +2358,7 @@ intel_hdmi_unset_edid(struct drm_connector *connector) } static void -intel_hdmi_dp_dual_mode_detect(struct drm_connector *connector, bool has_edid) +intel_hdmi_dp_dual_mode_detect(struct drm_connector *connector) { struct drm_i915_private *dev_priv = to_i915(connector->dev); struct intel_hdmi *hdmi = intel_attached_hdmi(to_intel_connector(connector)); @@ -2371,16 +2374,10 @@ intel_hdmi_dp_dual_mode_detect(struct drm_connector *connector, bool has_edid) * CONFIG1 pin, but no such luck on our hardware. * * The only method left to us is to check the VBT to see - * if the port is a dual mode capable DP port. But let's - * only do that when we sucesfully read the EDID, to avoid - * confusing log messages about DP dual mode adaptors when - * there's nothing connected to the port. + * if the port is a dual mode capable DP port. */ if (type == DRM_DP_DUAL_MODE_UNKNOWN) { - /* An overridden EDID imply that we want this port for testing. - * Make sure not to set limits for that port. - */ - if (has_edid && !connector->override_edid && + if (!connector->force && intel_bios_is_port_dp_dual_mode(dev_priv, port)) { drm_dbg_kms(&dev_priv->drm, "Assuming DP dual mode adaptor presence based on VBT\n"); @@ -2435,18 +2432,18 @@ intel_hdmi_set_edid(struct drm_connector *connector) intel_gmbus_force_bit(i2c, false); } - intel_hdmi_dp_dual_mode_detect(connector, edid != NULL); - - intel_display_power_put(dev_priv, POWER_DOMAIN_GMBUS, wakeref); - to_intel_connector(connector)->detect_edid = edid; if (edid && edid->input & DRM_EDID_INPUT_DIGITAL) { intel_hdmi->has_audio = drm_detect_monitor_audio(edid); intel_hdmi->has_hdmi_sink = drm_detect_hdmi_monitor(edid); + intel_hdmi_dp_dual_mode_detect(connector); + connected = true; } + intel_display_power_put(dev_priv, POWER_DOMAIN_GMBUS, wakeref); + cec_notifier_set_phys_addr_from_edid(intel_hdmi->cec_notifier, edid); return connected; @@ -2956,9 +2953,8 @@ void intel_hdmi_init_connector(struct intel_digital_port *dig_port, ddc); drm_connector_helper_add(connector, &intel_hdmi_connector_helper_funcs); - connector->interlace_allowed = 1; - connector->doublescan_allowed = 0; - connector->stereo_allowed = 1; + connector->interlace_allowed = true; + connector->stereo_allowed = true; if (DISPLAY_VER(dev_priv) >= 10) connector->ycbcr_420_allowed = true; diff --git a/drivers/gpu/drm/i915/display/intel_hdmi.h b/drivers/gpu/drm/i915/display/intel_hdmi.h index 93f65a917c36..774dda2376ed 100644 --- a/drivers/gpu/drm/i915/display/intel_hdmi.h +++ b/drivers/gpu/drm/i915/display/intel_hdmi.h @@ -6,20 +6,20 @@ #ifndef __INTEL_HDMI_H__ #define __INTEL_HDMI_H__ -#include <linux/hdmi.h> #include <linux/types.h> +enum hdmi_infoframe_type; +enum port; struct drm_connector; +struct drm_connector_state; struct drm_encoder; struct drm_i915_private; struct intel_connector; +struct intel_crtc_state; struct intel_digital_port; struct intel_encoder; -struct intel_crtc_state; struct intel_hdmi; -struct drm_connector_state; union hdmi_infoframe; -enum port; void intel_hdmi_init_connector(struct intel_digital_port *dig_port, struct intel_connector *intel_connector); diff --git a/drivers/gpu/drm/i915/display/intel_hotplug.c b/drivers/gpu/drm/i915/display/intel_hotplug.c index f7a2f485b177..907ab7526cb4 100644 --- a/drivers/gpu/drm/i915/display/intel_hotplug.c +++ b/drivers/gpu/drm/i915/display/intel_hotplug.c @@ -90,6 +90,9 @@ enum hpd_pin intel_hpd_pin_default(struct drm_i915_private *dev_priv, return HPD_PORT_A + port - PORT_A; } +/* Threshold == 5 for long IRQs, 50 for short */ +#define HPD_STORM_DEFAULT_THRESHOLD 50 + #define HPD_STORM_DETECT_PERIOD 1000 #define HPD_STORM_REENABLE_DELAY (2 * 60 * 1000) #define HPD_RETRY_DELAY 1000 @@ -175,14 +178,13 @@ static bool intel_hpd_irq_storm_detect(struct drm_i915_private *dev_priv, static void intel_hpd_irq_storm_switch_to_polling(struct drm_i915_private *dev_priv) { - struct drm_device *dev = &dev_priv->drm; struct drm_connector_list_iter conn_iter; struct intel_connector *connector; bool hpd_disabled = false; lockdep_assert_held(&dev_priv->irq_lock); - drm_connector_list_iter_begin(dev, &conn_iter); + drm_connector_list_iter_begin(&dev_priv->drm, &conn_iter); for_each_intel_connector_iter(connector, &conn_iter) { enum hpd_pin pin; @@ -208,7 +210,7 @@ intel_hpd_irq_storm_switch_to_polling(struct drm_i915_private *dev_priv) /* Enable polling and queue hotplug re-enabling. */ if (hpd_disabled) { - drm_kms_helper_poll_enable(dev); + drm_kms_helper_poll_enable(&dev_priv->drm); mod_delayed_work(system_wq, &dev_priv->display.hotplug.reenable_work, msecs_to_jiffies(HPD_STORM_REENABLE_DELAY)); } @@ -219,7 +221,6 @@ static void intel_hpd_irq_storm_reenable_work(struct work_struct *work) struct drm_i915_private *dev_priv = container_of(work, typeof(*dev_priv), display.hotplug.reenable_work.work); - struct drm_device *dev = &dev_priv->drm; struct drm_connector_list_iter conn_iter; struct intel_connector *connector; intel_wakeref_t wakeref; @@ -229,7 +230,7 @@ static void intel_hpd_irq_storm_reenable_work(struct work_struct *work) spin_lock_irq(&dev_priv->irq_lock); - drm_connector_list_iter_begin(dev, &conn_iter); + drm_connector_list_iter_begin(&dev_priv->drm, &conn_iter); for_each_intel_connector_iter(connector, &conn_iter) { pin = intel_connector_hpd_pin(connector); if (pin == HPD_NONE || @@ -367,14 +368,13 @@ static void i915_hotplug_work_func(struct work_struct *work) struct drm_i915_private *dev_priv = container_of(work, struct drm_i915_private, display.hotplug.hotplug_work.work); - struct drm_device *dev = &dev_priv->drm; struct drm_connector_list_iter conn_iter; struct intel_connector *connector; u32 changed = 0, retry = 0; u32 hpd_event_bits; u32 hpd_retry_bits; - mutex_lock(&dev->mode_config.mutex); + mutex_lock(&dev_priv->drm.mode_config.mutex); drm_dbg_kms(&dev_priv->drm, "running encoder hotplug functions\n"); spin_lock_irq(&dev_priv->irq_lock); @@ -389,7 +389,7 @@ static void i915_hotplug_work_func(struct work_struct *work) spin_unlock_irq(&dev_priv->irq_lock); - drm_connector_list_iter_begin(dev, &conn_iter); + drm_connector_list_iter_begin(&dev_priv->drm, &conn_iter); for_each_intel_connector_iter(connector, &conn_iter) { enum hpd_pin pin; u32 hpd_bit; @@ -426,10 +426,10 @@ static void i915_hotplug_work_func(struct work_struct *work) } } drm_connector_list_iter_end(&conn_iter); - mutex_unlock(&dev->mode_config.mutex); + mutex_unlock(&dev_priv->drm.mode_config.mutex); if (changed) - drm_kms_helper_hotplug_event(dev); + drm_kms_helper_hotplug_event(&dev_priv->drm); /* Remove shared HPD pins that have changed */ retry &= ~changed; @@ -612,16 +612,15 @@ static void i915_hpd_poll_init_work(struct work_struct *work) struct drm_i915_private *dev_priv = container_of(work, struct drm_i915_private, display.hotplug.poll_init_work); - struct drm_device *dev = &dev_priv->drm; struct drm_connector_list_iter conn_iter; struct intel_connector *connector; bool enabled; - mutex_lock(&dev->mode_config.mutex); + mutex_lock(&dev_priv->drm.mode_config.mutex); enabled = READ_ONCE(dev_priv->display.hotplug.poll_enabled); - drm_connector_list_iter_begin(dev, &conn_iter); + drm_connector_list_iter_begin(&dev_priv->drm, &conn_iter); for_each_intel_connector_iter(connector, &conn_iter) { enum hpd_pin pin; @@ -638,16 +637,16 @@ static void i915_hpd_poll_init_work(struct work_struct *work) drm_connector_list_iter_end(&conn_iter); if (enabled) - drm_kms_helper_poll_enable(dev); + drm_kms_helper_poll_enable(&dev_priv->drm); - mutex_unlock(&dev->mode_config.mutex); + mutex_unlock(&dev_priv->drm.mode_config.mutex); /* * We might have missed any hotplugs that happened while we were * in the middle of disabling polling */ if (!enabled) - drm_helper_hpd_irq_event(dev); + drm_helper_hpd_irq_event(&dev_priv->drm); } /** @@ -711,14 +710,23 @@ void intel_hpd_poll_disable(struct drm_i915_private *dev_priv) schedule_work(&dev_priv->display.hotplug.poll_init_work); } -void intel_hpd_init_work(struct drm_i915_private *dev_priv) +void intel_hpd_init_early(struct drm_i915_private *i915) { - INIT_DELAYED_WORK(&dev_priv->display.hotplug.hotplug_work, + INIT_DELAYED_WORK(&i915->display.hotplug.hotplug_work, i915_hotplug_work_func); - INIT_WORK(&dev_priv->display.hotplug.dig_port_work, i915_digport_work_func); - INIT_WORK(&dev_priv->display.hotplug.poll_init_work, i915_hpd_poll_init_work); - INIT_DELAYED_WORK(&dev_priv->display.hotplug.reenable_work, + INIT_WORK(&i915->display.hotplug.dig_port_work, i915_digport_work_func); + INIT_WORK(&i915->display.hotplug.poll_init_work, i915_hpd_poll_init_work); + INIT_DELAYED_WORK(&i915->display.hotplug.reenable_work, intel_hpd_irq_storm_reenable_work); + + i915->display.hotplug.hpd_storm_threshold = HPD_STORM_DEFAULT_THRESHOLD; + /* If we have MST support, we want to avoid doing short HPD IRQ storm + * detection, as short HPD storms will occur as a natural part of + * sideband messaging with MST. + * On older platforms however, IRQ storms can occur with both long and + * short pulses, as seen on some G4x systems. + */ + i915->display.hotplug.hpd_short_storm_enabled = !HAS_DP_MST(i915); } void intel_hpd_cancel_work(struct drm_i915_private *dev_priv) @@ -767,3 +775,169 @@ void intel_hpd_enable(struct drm_i915_private *dev_priv, enum hpd_pin pin) dev_priv->display.hotplug.stats[pin].state = HPD_ENABLED; spin_unlock_irq(&dev_priv->irq_lock); } + +static int i915_hpd_storm_ctl_show(struct seq_file *m, void *data) +{ + struct drm_i915_private *dev_priv = m->private; + struct intel_hotplug *hotplug = &dev_priv->display.hotplug; + + /* Synchronize with everything first in case there's been an HPD + * storm, but we haven't finished handling it in the kernel yet + */ + intel_synchronize_irq(dev_priv); + flush_work(&dev_priv->display.hotplug.dig_port_work); + flush_delayed_work(&dev_priv->display.hotplug.hotplug_work); + + seq_printf(m, "Threshold: %d\n", hotplug->hpd_storm_threshold); + seq_printf(m, "Detected: %s\n", + str_yes_no(delayed_work_pending(&hotplug->reenable_work))); + + return 0; +} + +static ssize_t i915_hpd_storm_ctl_write(struct file *file, + const char __user *ubuf, size_t len, + loff_t *offp) +{ + struct seq_file *m = file->private_data; + struct drm_i915_private *dev_priv = m->private; + struct intel_hotplug *hotplug = &dev_priv->display.hotplug; + unsigned int new_threshold; + int i; + char *newline; + char tmp[16]; + + if (len >= sizeof(tmp)) + return -EINVAL; + + if (copy_from_user(tmp, ubuf, len)) + return -EFAULT; + + tmp[len] = '\0'; + + /* Strip newline, if any */ + newline = strchr(tmp, '\n'); + if (newline) + *newline = '\0'; + + if (strcmp(tmp, "reset") == 0) + new_threshold = HPD_STORM_DEFAULT_THRESHOLD; + else if (kstrtouint(tmp, 10, &new_threshold) != 0) + return -EINVAL; + + if (new_threshold > 0) + drm_dbg_kms(&dev_priv->drm, + "Setting HPD storm detection threshold to %d\n", + new_threshold); + else + drm_dbg_kms(&dev_priv->drm, "Disabling HPD storm detection\n"); + + spin_lock_irq(&dev_priv->irq_lock); + hotplug->hpd_storm_threshold = new_threshold; + /* Reset the HPD storm stats so we don't accidentally trigger a storm */ + for_each_hpd_pin(i) + hotplug->stats[i].count = 0; + spin_unlock_irq(&dev_priv->irq_lock); + + /* Re-enable hpd immediately if we were in an irq storm */ + flush_delayed_work(&dev_priv->display.hotplug.reenable_work); + + return len; +} + +static int i915_hpd_storm_ctl_open(struct inode *inode, struct file *file) +{ + return single_open(file, i915_hpd_storm_ctl_show, inode->i_private); +} + +static const struct file_operations i915_hpd_storm_ctl_fops = { + .owner = THIS_MODULE, + .open = i915_hpd_storm_ctl_open, + .read = seq_read, + .llseek = seq_lseek, + .release = single_release, + .write = i915_hpd_storm_ctl_write +}; + +static int i915_hpd_short_storm_ctl_show(struct seq_file *m, void *data) +{ + struct drm_i915_private *dev_priv = m->private; + + seq_printf(m, "Enabled: %s\n", + str_yes_no(dev_priv->display.hotplug.hpd_short_storm_enabled)); + + return 0; +} + +static int +i915_hpd_short_storm_ctl_open(struct inode *inode, struct file *file) +{ + return single_open(file, i915_hpd_short_storm_ctl_show, + inode->i_private); +} + +static ssize_t i915_hpd_short_storm_ctl_write(struct file *file, + const char __user *ubuf, + size_t len, loff_t *offp) +{ + struct seq_file *m = file->private_data; + struct drm_i915_private *dev_priv = m->private; + struct intel_hotplug *hotplug = &dev_priv->display.hotplug; + char *newline; + char tmp[16]; + int i; + bool new_state; + + if (len >= sizeof(tmp)) + return -EINVAL; + + if (copy_from_user(tmp, ubuf, len)) + return -EFAULT; + + tmp[len] = '\0'; + + /* Strip newline, if any */ + newline = strchr(tmp, '\n'); + if (newline) + *newline = '\0'; + + /* Reset to the "default" state for this system */ + if (strcmp(tmp, "reset") == 0) + new_state = !HAS_DP_MST(dev_priv); + else if (kstrtobool(tmp, &new_state) != 0) + return -EINVAL; + + drm_dbg_kms(&dev_priv->drm, "%sabling HPD short storm detection\n", + new_state ? "En" : "Dis"); + + spin_lock_irq(&dev_priv->irq_lock); + hotplug->hpd_short_storm_enabled = new_state; + /* Reset the HPD storm stats so we don't accidentally trigger a storm */ + for_each_hpd_pin(i) + hotplug->stats[i].count = 0; + spin_unlock_irq(&dev_priv->irq_lock); + + /* Re-enable hpd immediately if we were in an irq storm */ + flush_delayed_work(&dev_priv->display.hotplug.reenable_work); + + return len; +} + +static const struct file_operations i915_hpd_short_storm_ctl_fops = { + .owner = THIS_MODULE, + .open = i915_hpd_short_storm_ctl_open, + .read = seq_read, + .llseek = seq_lseek, + .release = single_release, + .write = i915_hpd_short_storm_ctl_write, +}; + +void intel_hpd_debugfs_register(struct drm_i915_private *i915) +{ + struct drm_minor *minor = i915->drm.primary; + + debugfs_create_file("i915_hpd_storm_ctl", 0644, minor->debugfs_root, + i915, &i915_hpd_storm_ctl_fops); + debugfs_create_file("i915_hpd_short_storm_ctl", 0644, minor->debugfs_root, + i915, &i915_hpd_short_storm_ctl_fops); +} diff --git a/drivers/gpu/drm/i915/display/intel_hotplug.h b/drivers/gpu/drm/i915/display/intel_hotplug.h index b87e95d606e6..424ae5dbf5a0 100644 --- a/drivers/gpu/drm/i915/display/intel_hotplug.h +++ b/drivers/gpu/drm/i915/display/intel_hotplug.h @@ -22,11 +22,12 @@ void intel_hpd_irq_handler(struct drm_i915_private *dev_priv, u32 pin_mask, u32 long_mask); void intel_hpd_trigger_irq(struct intel_digital_port *dig_port); void intel_hpd_init(struct drm_i915_private *dev_priv); -void intel_hpd_init_work(struct drm_i915_private *dev_priv); +void intel_hpd_init_early(struct drm_i915_private *i915); void intel_hpd_cancel_work(struct drm_i915_private *dev_priv); enum hpd_pin intel_hpd_pin_default(struct drm_i915_private *dev_priv, enum port port); bool intel_hpd_disable(struct drm_i915_private *dev_priv, enum hpd_pin pin); void intel_hpd_enable(struct drm_i915_private *dev_priv, enum hpd_pin pin); +void intel_hpd_debugfs_register(struct drm_i915_private *i915); #endif /* __INTEL_HOTPLUG_H__ */ diff --git a/drivers/gpu/drm/i915/display/intel_hti.c b/drivers/gpu/drm/i915/display/intel_hti.c new file mode 100644 index 000000000000..12a1f4ce1a77 --- /dev/null +++ b/drivers/gpu/drm/i915/display/intel_hti.c @@ -0,0 +1,38 @@ +// SPDX-License-Identifier: MIT +/* + * Copyright © 2022 Intel Corporation + */ + +#include "i915_drv.h" +#include "intel_de.h" +#include "intel_display.h" +#include "intel_hti.h" +#include "intel_hti_regs.h" + +void intel_hti_init(struct drm_i915_private *i915) +{ + /* + * If the platform has HTI, we need to find out whether it has reserved + * any display resources before we create our display outputs. + */ + if (INTEL_INFO(i915)->display.has_hti) + i915->display.hti.state = intel_de_read(i915, HDPORT_STATE); +} + +bool intel_hti_uses_phy(struct drm_i915_private *i915, enum phy phy) +{ + return i915->display.hti.state & HDPORT_ENABLED && + i915->display.hti.state & HDPORT_DDI_USED(phy); +} + +u32 intel_hti_dpll_mask(struct drm_i915_private *i915) +{ + if (!(i915->display.hti.state & HDPORT_ENABLED)) + return 0; + + /* + * Note: This is subtle. The values must coincide with what's defined + * for the platform. + */ + return REG_FIELD_GET(HDPORT_DPLL_USED_MASK, i915->display.hti.state); +} diff --git a/drivers/gpu/drm/i915/display/intel_hti.h b/drivers/gpu/drm/i915/display/intel_hti.h new file mode 100644 index 000000000000..2893d6668657 --- /dev/null +++ b/drivers/gpu/drm/i915/display/intel_hti.h @@ -0,0 +1,18 @@ +/* SPDX-License-Identifier: MIT */ +/* + * Copyright © 2022 Intel Corporation + */ + +#ifndef __INTEL_HTI_H__ +#define __INTEL_HTI_H__ + +#include <linux/types.h> + +struct drm_i915_private; +enum phy; + +void intel_hti_init(struct drm_i915_private *i915); +bool intel_hti_uses_phy(struct drm_i915_private *i915, enum phy phy); +u32 intel_hti_dpll_mask(struct drm_i915_private *i915); + +#endif /* __INTEL_HTI_H__ */ diff --git a/drivers/gpu/drm/i915/display/intel_hti_regs.h b/drivers/gpu/drm/i915/display/intel_hti_regs.h new file mode 100644 index 000000000000..e206f2837fc8 --- /dev/null +++ b/drivers/gpu/drm/i915/display/intel_hti_regs.h @@ -0,0 +1,16 @@ +/* SPDX-License-Identifier: MIT */ +/* + * Copyright © 2022 Intel Corporation + */ + +#ifndef __INTEL_HTI_REGS_H__ +#define __INTEL_HTI_REGS_H__ + +#include "i915_reg_defs.h" + +#define HDPORT_STATE _MMIO(0x45050) +#define HDPORT_DPLL_USED_MASK REG_GENMASK(15, 12) +#define HDPORT_DDI_USED(phy) REG_BIT(2 * (phy) + 1) +#define HDPORT_ENABLED REG_BIT(0) + +#endif /* __INTEL_HTI_REGS_H__ */ diff --git a/drivers/gpu/drm/i915/display/intel_lpe_audio.c b/drivers/gpu/drm/i915/display/intel_lpe_audio.c index dca6003ccac8..8aaaef4d7856 100644 --- a/drivers/gpu/drm/i915/display/intel_lpe_audio.c +++ b/drivers/gpu/drm/i915/display/intel_lpe_audio.c @@ -71,6 +71,8 @@ #include <drm/intel_lpe_audio.h> #include "i915_drv.h" +#include "i915_irq.h" +#include "i915_reg.h" #include "intel_de.h" #include "intel_lpe_audio.h" #include "intel_pci_config.h" @@ -80,8 +82,7 @@ static struct platform_device * lpe_audio_platdev_create(struct drm_i915_private *dev_priv) { - struct drm_device *dev = &dev_priv->drm; - struct pci_dev *pdev = to_pci_dev(dev->dev); + struct pci_dev *pdev = to_pci_dev(dev_priv->drm.dev); struct platform_device_info pinfo = {}; struct resource *rsc; struct platform_device *platdev; @@ -101,14 +102,14 @@ lpe_audio_platdev_create(struct drm_i915_private *dev_priv) rsc[0].flags = IORESOURCE_IRQ; rsc[0].name = "hdmi-lpe-audio-irq"; - rsc[1].start = pci_resource_start(pdev, GTTMMADR_BAR) + + rsc[1].start = pci_resource_start(pdev, GEN4_GTTMMADR_BAR) + I915_HDMI_LPE_AUDIO_BASE; - rsc[1].end = pci_resource_start(pdev, GTTMMADR_BAR) + + rsc[1].end = pci_resource_start(pdev, GEN4_GTTMMADR_BAR) + I915_HDMI_LPE_AUDIO_BASE + I915_HDMI_LPE_AUDIO_SIZE - 1; rsc[1].flags = IORESOURCE_MEM; rsc[1].name = "hdmi-lpe-audio-mmio"; - pinfo.parent = dev->dev; + pinfo.parent = dev_priv->drm.dev; pinfo.name = "hdmi-lpe-audio"; pinfo.id = -1; pinfo.res = rsc; diff --git a/drivers/gpu/drm/i915/display/intel_lspcon.c b/drivers/gpu/drm/i915/display/intel_lspcon.c index 15d59de8810e..9ff1c0b223ad 100644 --- a/drivers/gpu/drm/i915/display/intel_lspcon.c +++ b/drivers/gpu/drm/i915/display/intel_lspcon.c @@ -28,6 +28,7 @@ #include <drm/drm_atomic_helper.h> #include <drm/drm_edid.h> +#include "i915_reg.h" #include "intel_de.h" #include "intel_display_types.h" #include "intel_dp.h" diff --git a/drivers/gpu/drm/i915/display/intel_lvds.c b/drivers/gpu/drm/i915/display/intel_lvds.c index e5352239b2a2..7bf1bdfd03ec 100644 --- a/drivers/gpu/drm/i915/display/intel_lvds.c +++ b/drivers/gpu/drm/i915/display/intel_lvds.c @@ -39,6 +39,7 @@ #include <drm/drm_edid.h> #include "i915_drv.h" +#include "i915_reg.h" #include "intel_atomic.h" #include "intel_backlight.h" #include "intel_connector.h" @@ -78,9 +79,9 @@ struct intel_lvds_encoder { struct intel_connector *attached_connector; }; -static struct intel_lvds_encoder *to_lvds_encoder(struct drm_encoder *encoder) +static struct intel_lvds_encoder *to_lvds_encoder(struct intel_encoder *encoder) { - return container_of(encoder, struct intel_lvds_encoder, base.base); + return container_of(encoder, struct intel_lvds_encoder, base); } bool intel_lvds_port_enabled(struct drm_i915_private *dev_priv, @@ -103,7 +104,7 @@ static bool intel_lvds_get_hw_state(struct intel_encoder *encoder, enum pipe *pipe) { struct drm_i915_private *dev_priv = to_i915(encoder->base.dev); - struct intel_lvds_encoder *lvds_encoder = to_lvds_encoder(&encoder->base); + struct intel_lvds_encoder *lvds_encoder = to_lvds_encoder(encoder); intel_wakeref_t wakeref; bool ret; @@ -123,7 +124,7 @@ static void intel_lvds_get_config(struct intel_encoder *encoder, struct intel_crtc_state *pipe_config) { struct drm_i915_private *dev_priv = to_i915(encoder->base.dev); - struct intel_lvds_encoder *lvds_encoder = to_lvds_encoder(&encoder->base); + struct intel_lvds_encoder *lvds_encoder = to_lvds_encoder(encoder); u32 tmp, flags = 0; pipe_config->output_types |= BIT(INTEL_OUTPUT_LVDS); @@ -229,7 +230,7 @@ static void intel_pre_enable_lvds(struct intel_atomic_state *state, const struct intel_crtc_state *pipe_config, const struct drm_connector_state *conn_state) { - struct intel_lvds_encoder *lvds_encoder = to_lvds_encoder(&encoder->base); + struct intel_lvds_encoder *lvds_encoder = to_lvds_encoder(encoder); struct drm_i915_private *dev_priv = to_i915(encoder->base.dev); struct intel_crtc *crtc = to_intel_crtc(pipe_config->uapi.crtc); const struct drm_display_mode *adjusted_mode = &pipe_config->hw.adjusted_mode; @@ -312,7 +313,7 @@ static void intel_enable_lvds(struct intel_atomic_state *state, const struct drm_connector_state *conn_state) { struct drm_device *dev = encoder->base.dev; - struct intel_lvds_encoder *lvds_encoder = to_lvds_encoder(&encoder->base); + struct intel_lvds_encoder *lvds_encoder = to_lvds_encoder(encoder); struct drm_i915_private *dev_priv = to_i915(dev); intel_de_write(dev_priv, lvds_encoder->reg, @@ -334,7 +335,7 @@ static void intel_disable_lvds(struct intel_atomic_state *state, const struct intel_crtc_state *old_crtc_state, const struct drm_connector_state *old_conn_state) { - struct intel_lvds_encoder *lvds_encoder = to_lvds_encoder(&encoder->base); + struct intel_lvds_encoder *lvds_encoder = to_lvds_encoder(encoder); struct drm_i915_private *dev_priv = to_i915(encoder->base.dev); intel_de_write(dev_priv, PP_CONTROL(0), @@ -413,7 +414,7 @@ static int intel_lvds_compute_config(struct intel_encoder *intel_encoder, { struct drm_i915_private *dev_priv = to_i915(intel_encoder->base.dev); struct intel_lvds_encoder *lvds_encoder = - to_lvds_encoder(&intel_encoder->base); + to_lvds_encoder(intel_encoder); struct intel_connector *intel_connector = lvds_encoder->attached_connector; struct drm_display_mode *adjusted_mode = &pipe_config->hw.adjusted_mode; @@ -775,7 +776,7 @@ bool intel_is_dual_link_lvds(struct drm_i915_private *dev_priv) { struct intel_encoder *encoder = intel_get_lvds_encoder(dev_priv); - return encoder && to_lvds_encoder(&encoder->base)->is_dual_link; + return encoder && to_lvds_encoder(encoder)->is_dual_link; } static bool compute_is_dual_link_lvds(struct intel_lvds_encoder *lvds_encoder) @@ -814,6 +815,11 @@ static bool compute_is_dual_link_lvds(struct intel_lvds_encoder *lvds_encoder) return (val & LVDS_CLKB_POWER_MASK) == LVDS_CLKB_POWER_UP; } +static void intel_lvds_add_properties(struct drm_connector *connector) +{ + intel_attach_scaling_mode_property(connector); +} + /** * intel_lvds_init - setup LVDS connectors on this device * @dev_priv: i915 device @@ -823,7 +829,6 @@ static bool compute_is_dual_link_lvds(struct intel_lvds_encoder *lvds_encoder) */ void intel_lvds_init(struct drm_i915_private *dev_priv) { - struct drm_device *dev = &dev_priv->drm; struct intel_lvds_encoder *lvds_encoder; struct intel_encoder *intel_encoder; struct intel_connector *intel_connector; @@ -833,11 +838,10 @@ void intel_lvds_init(struct drm_i915_private *dev_priv) i915_reg_t lvds_reg; u32 lvds; u8 pin; - u32 allowed_scalers; /* Skip init on machines we know falsely report LVDS */ if (dmi_check_system(intel_no_lvds)) { - drm_WARN(dev, !dev_priv->display.vbt.int_lvds_support, + drm_WARN(&dev_priv->drm, !dev_priv->display.vbt.int_lvds_support, "Useless DMI match. Internal LVDS support disabled by VBT\n"); return; } @@ -886,10 +890,10 @@ void intel_lvds_init(struct drm_i915_private *dev_priv) intel_encoder = &lvds_encoder->base; encoder = &intel_encoder->base; connector = &intel_connector->base; - drm_connector_init(dev, &intel_connector->base, &intel_lvds_connector_funcs, + drm_connector_init(&dev_priv->drm, &intel_connector->base, &intel_lvds_connector_funcs, DRM_MODE_CONNECTOR_LVDS); - drm_encoder_init(dev, &intel_encoder->base, &intel_lvds_enc_funcs, + drm_encoder_init(&dev_priv->drm, &intel_encoder->base, &intel_lvds_enc_funcs, DRM_MODE_ENCODER_LVDS, "LVDS"); intel_encoder->enable = intel_enable_lvds; @@ -920,17 +924,10 @@ void intel_lvds_init(struct drm_i915_private *dev_priv) drm_connector_helper_add(connector, &intel_lvds_connector_helper_funcs); connector->display_info.subpixel_order = SubPixelHorizontalRGB; - connector->interlace_allowed = false; - connector->doublescan_allowed = false; lvds_encoder->reg = lvds_reg; - /* create the scaling mode property */ - allowed_scalers = BIT(DRM_MODE_SCALE_ASPECT); - allowed_scalers |= BIT(DRM_MODE_SCALE_FULLSCREEN); - allowed_scalers |= BIT(DRM_MODE_SCALE_CENTER); - drm_connector_attach_scaling_mode_property(connector, allowed_scalers); - connector->state->scaling_mode = DRM_MODE_SCALE_ASPECT; + intel_lvds_add_properties(connector); intel_lvds_pps_get_hw_state(dev_priv, &lvds_encoder->init_pps); lvds_encoder->init_lvds_val = lvds; @@ -947,7 +944,7 @@ void intel_lvds_init(struct drm_i915_private *dev_priv) * Attempt to get the fixed panel mode from DDC. Assume that the * preferred mode is the right one. */ - mutex_lock(&dev->mode_config.mutex); + mutex_lock(&dev_priv->drm.mode_config.mutex); if (vga_switcheroo_handler_flags() & VGA_SWITCHEROO_CAN_SWITCH_DDC) edid = drm_get_edid_switcheroo(connector, intel_gmbus_get_adapter(dev_priv, pin)); @@ -971,8 +968,7 @@ void intel_lvds_init(struct drm_i915_private *dev_priv) IS_ERR(edid) ? NULL : edid); /* Try EDID first */ - intel_panel_add_edid_fixed_modes(intel_connector, - intel_connector->panel.vbt.drrs_type != DRRS_TYPE_NONE); + intel_panel_add_edid_fixed_modes(intel_connector, true); /* Failed to get EDID, what about VBT? */ if (!intel_panel_preferred_fixed_mode(intel_connector)) @@ -986,7 +982,7 @@ void intel_lvds_init(struct drm_i915_private *dev_priv) if (!intel_panel_preferred_fixed_mode(intel_connector)) intel_panel_add_encoder_fixed_mode(intel_connector, intel_encoder); - mutex_unlock(&dev->mode_config.mutex); + mutex_unlock(&dev_priv->drm.mode_config.mutex); /* If we still don't have a mode after all that, give up. */ if (!intel_panel_preferred_fixed_mode(intel_connector)) diff --git a/drivers/gpu/drm/i915/display/intel_tc_phy_regs.h b/drivers/gpu/drm/i915/display/intel_mg_phy_regs.h index 5a545086f959..0e8248bce52d 100644 --- a/drivers/gpu/drm/i915/display/intel_tc_phy_regs.h +++ b/drivers/gpu/drm/i915/display/intel_mg_phy_regs.h @@ -3,10 +3,10 @@ * Copyright © 2022 Intel Corporation */ -#ifndef __INTEL_TC_PHY_REGS__ -#define __INTEL_TC_PHY_REGS__ +#ifndef __INTEL_MG_PHY_REGS__ +#define __INTEL_MG_PHY_REGS__ -#include "i915_reg_defs.h" +#include "intel_display_reg_defs.h" #define MG_PHY_PORT_LN(ln, tc_port, ln0p1, ln0p2, ln1p1) \ _MMIO(_PORT(tc_port, ln0p1, ln0p2) + (ln) * ((ln1p1) - (ln0p1))) @@ -277,4 +277,4 @@ _MG_PLL_TDC_COLDST_BIAS_PORT1, \ _MG_PLL_TDC_COLDST_BIAS_PORT2) -#endif /* __INTEL_TC_PHY_REGS__ */ +#endif /* __INTEL_MG_PHY_REGS__ */ diff --git a/drivers/gpu/drm/i915/display/intel_modeset_setup.c b/drivers/gpu/drm/i915/display/intel_modeset_setup.c index cbfabd58b75a..96395bfbd41d 100644 --- a/drivers/gpu/drm/i915/display/intel_modeset_setup.c +++ b/drivers/gpu/drm/i915/display/intel_modeset_setup.c @@ -10,6 +10,7 @@ #include <drm/drm_atomic_state_helper.h> #include "i915_drv.h" +#include "i915_reg.h" #include "intel_atomic.h" #include "intel_bw.h" #include "intel_color.h" @@ -155,6 +156,12 @@ static void intel_crtc_copy_hw_to_uapi_state(struct intel_crtc_state *crtc_state crtc_state->uapi.adjusted_mode = crtc_state->hw.adjusted_mode; crtc_state->uapi.scaling_filter = crtc_state->hw.scaling_filter; + /* assume 1:1 mapping */ + drm_property_replace_blob(&crtc_state->hw.degamma_lut, + crtc_state->pre_csc_lut); + drm_property_replace_blob(&crtc_state->hw.gamma_lut, + crtc_state->post_csc_lut); + drm_property_replace_blob(&crtc_state->uapi.degamma_lut, crtc_state->hw.degamma_lut); drm_property_replace_blob(&crtc_state->uapi.gamma_lut, @@ -205,13 +212,21 @@ static bool intel_crtc_has_encoders(struct intel_crtc *crtc) static struct intel_connector *intel_encoder_find_connector(struct intel_encoder *encoder) { - struct drm_device *dev = encoder->base.dev; + struct drm_i915_private *i915 = to_i915(encoder->base.dev); + struct drm_connector_list_iter conn_iter; struct intel_connector *connector; + struct intel_connector *found_connector = NULL; - for_each_connector_on_encoder(dev, &encoder->base, connector) - return connector; + drm_connector_list_iter_begin(&i915->drm, &conn_iter); + for_each_intel_connector_iter(connector, &conn_iter) { + if (&encoder->base == connector->base.encoder) { + found_connector = connector; + break; + } + } + drm_connector_list_iter_end(&conn_iter); - return NULL; + return found_connector; } static void intel_sanitize_fifo_underrun_reporting(const struct intel_crtc_state *crtc_state) diff --git a/drivers/gpu/drm/i915/display/intel_modeset_verify.c b/drivers/gpu/drm/i915/display/intel_modeset_verify.c index 0fdcf2e6d57f..842d70f0dfd2 100644 --- a/drivers/gpu/drm/i915/display/intel_modeset_verify.c +++ b/drivers/gpu/drm/i915/display/intel_modeset_verify.c @@ -227,7 +227,8 @@ void intel_modeset_verify_crtc(struct intel_crtc *crtc, struct intel_crtc_state *old_crtc_state, struct intel_crtc_state *new_crtc_state) { - if (!intel_crtc_needs_modeset(new_crtc_state) && !new_crtc_state->update_pipe) + if (!intel_crtc_needs_modeset(new_crtc_state) && + !intel_crtc_needs_fastset(new_crtc_state)) return; intel_wm_state_verify(crtc, new_crtc_state); diff --git a/drivers/gpu/drm/i915/display/intel_opregion.c b/drivers/gpu/drm/i915/display/intel_opregion.c index caa07ef34f21..e0184745632c 100644 --- a/drivers/gpu/drm/i915/display/intel_opregion.c +++ b/drivers/gpu/drm/i915/display/intel_opregion.c @@ -463,7 +463,6 @@ static u32 asle_set_backlight(struct drm_i915_private *dev_priv, u32 bclp) struct intel_connector *connector; struct drm_connector_list_iter conn_iter; struct opregion_asle *asle = dev_priv->display.opregion.asle; - struct drm_device *dev = &dev_priv->drm; drm_dbg(&dev_priv->drm, "bclp = 0x%08x\n", bclp); @@ -480,7 +479,7 @@ static u32 asle_set_backlight(struct drm_i915_private *dev_priv, u32 bclp) if (bclp > 255) return ASLC_BACKLIGHT_FAILED; - drm_modeset_lock(&dev->mode_config.connection_mutex, NULL); + drm_modeset_lock(&dev_priv->drm.mode_config.connection_mutex, NULL); /* * Update backlight on all connectors that support backlight (usually @@ -488,13 +487,13 @@ static u32 asle_set_backlight(struct drm_i915_private *dev_priv, u32 bclp) */ drm_dbg_kms(&dev_priv->drm, "updating opregion backlight %d/255\n", bclp); - drm_connector_list_iter_begin(dev, &conn_iter); + drm_connector_list_iter_begin(&dev_priv->drm, &conn_iter); for_each_intel_connector_iter(connector, &conn_iter) intel_backlight_set_acpi(connector->base.state, bclp, 255); drm_connector_list_iter_end(&conn_iter); asle->cblv = DIV_ROUND_UP(bclp * 100, 255) | ASLE_CBLV_VALID; - drm_modeset_unlock(&dev->mode_config.connection_mutex); + drm_modeset_unlock(&dev_priv->drm.mode_config.connection_mutex); return 0; diff --git a/drivers/gpu/drm/i915/display/intel_panel.c b/drivers/gpu/drm/i915/display/intel_panel.c index 41cec9dc4223..1640726bfbf6 100644 --- a/drivers/gpu/drm/i915/display/intel_panel.c +++ b/drivers/gpu/drm/i915/display/intel_panel.c @@ -31,6 +31,7 @@ #include <linux/kernel.h> #include <linux/pwm.h> +#include "i915_reg.h" #include "intel_backlight.h" #include "intel_connector.h" #include "intel_de.h" @@ -85,9 +86,10 @@ static bool is_alt_drrs_mode(const struct drm_display_mode *mode, static bool is_alt_fixed_mode(const struct drm_display_mode *mode, const struct drm_display_mode *preferred_mode) { - return drm_mode_match(mode, preferred_mode, - DRM_MODE_MATCH_FLAGS | - DRM_MODE_MATCH_3D_FLAGS) && + u32 sync_flags = DRM_MODE_FLAG_PHSYNC | DRM_MODE_FLAG_NHSYNC | + DRM_MODE_FLAG_PVSYNC | DRM_MODE_FLAG_NVSYNC; + + return (mode->flags & ~sync_flags) == (preferred_mode->flags & ~sync_flags) && mode->hdisplay == preferred_mode->hdisplay && mode->vdisplay == preferred_mode->vdisplay; } @@ -147,12 +149,24 @@ int intel_panel_get_modes(struct intel_connector *connector) return num_modes; } -enum drrs_type intel_panel_drrs_type(struct intel_connector *connector) +static bool has_drrs_modes(struct intel_connector *connector) { - if (list_empty(&connector->panel.fixed_modes) || - list_is_singular(&connector->panel.fixed_modes)) - return DRRS_TYPE_NONE; + const struct drm_display_mode *mode1; + + list_for_each_entry(mode1, &connector->panel.fixed_modes, head) { + const struct drm_display_mode *mode2 = mode1; + + list_for_each_entry_continue(mode2, &connector->panel.fixed_modes, head) { + if (is_alt_drrs_mode(mode1, mode2)) + return true; + } + } + + return false; +} +enum drrs_type intel_panel_drrs_type(struct intel_connector *connector) +{ return connector->panel.vbt.drrs_type; } @@ -653,6 +667,9 @@ int intel_panel_init(struct intel_connector *connector) intel_backlight_init_funcs(panel); + if (!has_drrs_modes(connector)) + connector->panel.vbt.drrs_type = DRRS_TYPE_NONE; + drm_dbg_kms(connector->base.dev, "[CONNECTOR:%d:%s] DRRS type: %s\n", connector->base.base.id, connector->base.name, diff --git a/drivers/gpu/drm/i915/display/intel_pch_display.c b/drivers/gpu/drm/i915/display/intel_pch_display.c index 837152dca063..cecc0d007cf3 100644 --- a/drivers/gpu/drm/i915/display/intel_pch_display.c +++ b/drivers/gpu/drm/i915/display/intel_pch_display.c @@ -4,6 +4,7 @@ */ #include "g4x_dp.h" +#include "i915_reg.h" #include "intel_crt.h" #include "intel_de.h" #include "intel_display_types.h" diff --git a/drivers/gpu/drm/i915/display/intel_pch_refclk.c b/drivers/gpu/drm/i915/display/intel_pch_refclk.c index a66097cdc1e0..08a94365b7d1 100644 --- a/drivers/gpu/drm/i915/display/intel_pch_refclk.c +++ b/drivers/gpu/drm/i915/display/intel_pch_refclk.c @@ -3,6 +3,7 @@ * Copyright © 2021 Intel Corporation */ +#include "i915_reg.h" #include "intel_de.h" #include "intel_display_types.h" #include "intel_panel.h" diff --git a/drivers/gpu/drm/i915/display/intel_pipe_crc.c b/drivers/gpu/drm/i915/display/intel_pipe_crc.c index 8ac263f471be..e9774670e3f6 100644 --- a/drivers/gpu/drm/i915/display/intel_pipe_crc.c +++ b/drivers/gpu/drm/i915/display/intel_pipe_crc.c @@ -24,11 +24,12 @@ * */ -#include <linux/circ_buf.h> #include <linux/ctype.h> #include <linux/debugfs.h> #include <linux/seq_file.h> +#include "i915_irq.h" +#include "i915_reg.h" #include "intel_atomic.h" #include "intel_de.h" #include "intel_display_types.h" @@ -75,7 +76,6 @@ static int i9xx_pipe_crc_auto_source(struct drm_i915_private *dev_priv, enum pipe pipe, enum intel_pipe_crc_source *source) { - struct drm_device *dev = &dev_priv->drm; struct intel_encoder *encoder; struct intel_crtc *crtc; struct intel_digital_port *dig_port; @@ -83,8 +83,8 @@ static int i9xx_pipe_crc_auto_source(struct drm_i915_private *dev_priv, *source = INTEL_PIPE_CRC_SOURCE_PIPE; - drm_modeset_lock_all(dev); - for_each_intel_encoder(dev, encoder) { + drm_modeset_lock_all(&dev_priv->drm); + for_each_intel_encoder(&dev_priv->drm, encoder) { if (!encoder->base.crtc) continue; @@ -111,7 +111,7 @@ static int i9xx_pipe_crc_auto_source(struct drm_i915_private *dev_priv, *source = INTEL_PIPE_CRC_SOURCE_DP_D; break; default: - drm_WARN(dev, 1, "nonexisting DP port %c\n", + drm_WARN(&dev_priv->drm, 1, "nonexisting DP port %c\n", port_name(dig_port->base.port)); break; } @@ -120,7 +120,7 @@ static int i9xx_pipe_crc_auto_source(struct drm_i915_private *dev_priv, break; } } - drm_modeset_unlock_all(dev); + drm_modeset_unlock_all(&dev_priv->drm); return ret; } diff --git a/drivers/gpu/drm/i915/display/intel_pps.c b/drivers/gpu/drm/i915/display/intel_pps.c index 21944f5bf3a8..9bbf41a076f7 100644 --- a/drivers/gpu/drm/i915/display/intel_pps.c +++ b/drivers/gpu/drm/i915/display/intel_pps.c @@ -5,10 +5,12 @@ #include "g4x_dp.h" #include "i915_drv.h" +#include "i915_reg.h" #include "intel_de.h" #include "intel_display_power_well.h" #include "intel_display_types.h" #include "intel_dp.h" +#include "intel_dpio_phy.h" #include "intel_dpll.h" #include "intel_lvds.h" #include "intel_pps.h" @@ -1098,7 +1100,13 @@ bool intel_pps_have_panel_power_or_vdd(struct intel_dp *intel_dp) static void pps_init_timestamps(struct intel_dp *intel_dp) { - intel_dp->pps.panel_power_off_time = ktime_get_boottime(); + /* + * Initialize panel power off time to 0, assuming panel power could have + * been toggled between kernel boot and now only by a previously loaded + * and removed i915, which has already ensured sufficient power off + * delay at module remove. + */ + intel_dp->pps.panel_power_off_time = 0; intel_dp->pps.last_power_on = jiffies; intel_dp->pps.last_backlight_off = jiffies; } diff --git a/drivers/gpu/drm/i915/display/intel_psr.c b/drivers/gpu/drm/i915/display/intel_psr.c index 15c3e448aa0e..5b678916e6db 100644 --- a/drivers/gpu/drm/i915/display/intel_psr.c +++ b/drivers/gpu/drm/i915/display/intel_psr.c @@ -27,6 +27,7 @@ #include "display/intel_dp.h" #include "i915_drv.h" +#include "i915_reg.h" #include "intel_atomic.h" #include "intel_crtc.h" #include "intel_de.h" @@ -533,7 +534,7 @@ static void hsw_activate_psr2(struct intel_dp *intel_dp) val |= psr_compute_idle_frames(intel_dp) << EDP_PSR2_IDLE_FRAME_SHIFT; - if (!IS_ALDERLAKE_P(dev_priv)) + if (DISPLAY_VER(dev_priv) <= 13 && !IS_ALDERLAKE_P(dev_priv)) val |= EDP_SU_TRACK_ENABLE; if (DISPLAY_VER(dev_priv) >= 10 && DISPLAY_VER(dev_priv) <= 12) @@ -616,7 +617,7 @@ static void hsw_activate_psr2(struct intel_dp *intel_dp) static bool transcoder_has_psr2(struct drm_i915_private *dev_priv, enum transcoder trans) { - if (IS_ALDERLAKE_P(dev_priv)) + if (IS_ALDERLAKE_P(dev_priv) || DISPLAY_VER(dev_priv) >= 14) return trans == TRANSCODER_A || trans == TRANSCODER_B; else if (DISPLAY_VER(dev_priv) >= 12) return trans == TRANSCODER_A; @@ -696,7 +697,7 @@ dc3co_is_pipe_port_compatible(struct intel_dp *intel_dp, struct drm_i915_private *dev_priv = dp_to_i915(intel_dp); enum port port = dig_port->base.port; - if (IS_ALDERLAKE_P(dev_priv)) + if (IS_ALDERLAKE_P(dev_priv) || DISPLAY_VER(dev_priv) >= 14) return pipe <= PIPE_B && port <= PORT_B; else return pipe == PIPE_A && port == PORT_A; @@ -779,6 +780,7 @@ static bool psr2_granularity_check(struct intel_dp *intel_dp, struct intel_crtc_state *crtc_state) { struct drm_i915_private *dev_priv = dp_to_i915(intel_dp); + const struct drm_dsc_config *vdsc_cfg = &crtc_state->dsc.config; const int crtc_hdisplay = crtc_state->hw.adjusted_mode.crtc_hdisplay; const int crtc_vdisplay = crtc_state->hw.adjusted_mode.crtc_vdisplay; u16 y_granularity = 0; @@ -795,11 +797,11 @@ static bool psr2_granularity_check(struct intel_dp *intel_dp, return intel_dp->psr.su_y_granularity == 4; /* - * adl_p has 1 line granularity. For other platforms with SW tracking we - * can adjust the y coordinates to match sink requirement if multiple of - * 4. + * adl_p and display 14+ platforms has 1 line granularity. + * For other platforms with SW tracking we can adjust the y coordinates + * to match sink requirement if multiple of 4. */ - if (IS_ALDERLAKE_P(dev_priv)) + if (IS_ALDERLAKE_P(dev_priv) || DISPLAY_VER(dev_priv) >= 14) y_granularity = intel_dp->psr.su_y_granularity; else if (intel_dp->psr.su_y_granularity <= 2) y_granularity = 4; @@ -809,6 +811,10 @@ static bool psr2_granularity_check(struct intel_dp *intel_dp, if (y_granularity == 0 || crtc_vdisplay % y_granularity) return false; + if (crtc_state->dsc.compression_enable && + vdsc_cfg->slice_height % y_granularity) + return false; + crtc_state->su_y_granularity = y_granularity; return true; } @@ -883,7 +889,8 @@ static bool intel_psr2_config_valid(struct intel_dp *intel_dp, * resolution requires DSC to be enabled, priority is given to DSC * over PSR2. */ - if (crtc_state->dsc.compression_enable) { + if (crtc_state->dsc.compression_enable && + (DISPLAY_VER(dev_priv) <= 13 && !IS_ALDERLAKE_P(dev_priv))) { drm_dbg_kms(&dev_priv->drm, "PSR2 cannot be enabled since DSC is enabled\n"); return false; @@ -1469,26 +1476,27 @@ unlock: static u32 man_trk_ctl_enable_bit_get(struct drm_i915_private *dev_priv) { - return IS_ALDERLAKE_P(dev_priv) ? 0 : PSR2_MAN_TRK_CTL_ENABLE; + return IS_ALDERLAKE_P(dev_priv) || DISPLAY_VER(dev_priv) >= 14 ? 0 : + PSR2_MAN_TRK_CTL_ENABLE; } static u32 man_trk_ctl_single_full_frame_bit_get(struct drm_i915_private *dev_priv) { - return IS_ALDERLAKE_P(dev_priv) ? + return IS_ALDERLAKE_P(dev_priv) || DISPLAY_VER(dev_priv) >= 14 ? ADLP_PSR2_MAN_TRK_CTL_SF_SINGLE_FULL_FRAME : PSR2_MAN_TRK_CTL_SF_SINGLE_FULL_FRAME; } static u32 man_trk_ctl_partial_frame_bit_get(struct drm_i915_private *dev_priv) { - return IS_ALDERLAKE_P(dev_priv) ? + return IS_ALDERLAKE_P(dev_priv) || DISPLAY_VER(dev_priv) >= 14 ? ADLP_PSR2_MAN_TRK_CTL_SF_PARTIAL_FRAME_UPDATE : PSR2_MAN_TRK_CTL_SF_PARTIAL_FRAME_UPDATE; } static u32 man_trk_ctl_continuos_full_frame(struct drm_i915_private *dev_priv) { - return IS_ALDERLAKE_P(dev_priv) ? + return IS_ALDERLAKE_P(dev_priv) || DISPLAY_VER(dev_priv) >= 14 ? ADLP_PSR2_MAN_TRK_CTL_SF_CONTINUOS_FULL_FRAME : PSR2_MAN_TRK_CTL_SF_CONTINUOS_FULL_FRAME; } @@ -1627,7 +1635,7 @@ static void psr2_man_trk_ctl_calc(struct intel_crtc_state *crtc_state, if (clip->y1 == -1) goto exit; - if (IS_ALDERLAKE_P(dev_priv)) { + if (IS_ALDERLAKE_P(dev_priv) || DISPLAY_VER(dev_priv) >= 14) { val |= ADLP_PSR2_MAN_TRK_CTL_SU_REGION_START_ADDR(clip->y1); val |= ADLP_PSR2_MAN_TRK_CTL_SU_REGION_END_ADDR(clip->y2 - 1); } else { @@ -1664,14 +1672,19 @@ static void intel_psr2_sel_fetch_pipe_alignment(const struct intel_crtc_state *c struct drm_rect *pipe_clip) { struct drm_i915_private *dev_priv = to_i915(crtc_state->uapi.crtc->dev); - const u16 y_alignment = crtc_state->su_y_granularity; + const struct drm_dsc_config *vdsc_cfg = &crtc_state->dsc.config; + u16 y_alignment; + + /* ADLP aligns the SU region to vdsc slice height in case dsc is enabled */ + if (crtc_state->dsc.compression_enable && + (IS_ALDERLAKE_P(dev_priv) || DISPLAY_VER(dev_priv) >= 14)) + y_alignment = vdsc_cfg->slice_height; + else + y_alignment = crtc_state->su_y_granularity; pipe_clip->y1 -= pipe_clip->y1 % y_alignment; if (pipe_clip->y2 % y_alignment) pipe_clip->y2 = ((pipe_clip->y2 / y_alignment) + 1) * y_alignment; - - if (IS_ALDERLAKE_P(dev_priv) && crtc_state->dsc.compression_enable) - drm_warn(&dev_priv->drm, "Missing PSR2 sel fetch alignment with DSC\n"); } /* @@ -2054,13 +2067,12 @@ static bool __psr_wait_for_idle_locked(struct intel_dp *intel_dp) static int intel_psr_fastset_force(struct drm_i915_private *dev_priv) { struct drm_connector_list_iter conn_iter; - struct drm_device *dev = &dev_priv->drm; struct drm_modeset_acquire_ctx ctx; struct drm_atomic_state *state; struct drm_connector *conn; int err = 0; - state = drm_atomic_state_alloc(dev); + state = drm_atomic_state_alloc(&dev_priv->drm); if (!state) return -ENOMEM; @@ -2069,7 +2081,7 @@ static int intel_psr_fastset_force(struct drm_i915_private *dev_priv) retry: - drm_connector_list_iter_begin(dev, &conn_iter); + drm_connector_list_iter_begin(&dev_priv->drm, &conn_iter); drm_for_each_connector_iter(conn, &conn_iter) { struct drm_connector_state *conn_state; struct drm_crtc_state *crtc_state; diff --git a/drivers/gpu/drm/i915/display/intel_sdvo.c b/drivers/gpu/drm/i915/display/intel_sdvo.c index 774c1dc31a52..329b9d9af667 100644 --- a/drivers/gpu/drm/i915/display/intel_sdvo.c +++ b/drivers/gpu/drm/i915/display/intel_sdvo.c @@ -37,6 +37,7 @@ #include <drm/drm_edid.h> #include "i915_drv.h" +#include "i915_reg.h" #include "intel_atomic.h" #include "intel_connector.h" #include "intel_crtc.h" @@ -199,7 +200,7 @@ to_intel_sdvo_connector(struct drm_connector *connector) container_of((conn_state), struct intel_sdvo_connector_state, base.base) static bool -intel_sdvo_output_setup(struct intel_sdvo *intel_sdvo, u16 flags); +intel_sdvo_output_setup(struct intel_sdvo *intel_sdvo); static bool intel_sdvo_tv_create_property(struct intel_sdvo *intel_sdvo, struct intel_sdvo_connector *intel_sdvo_connector, @@ -1297,13 +1298,28 @@ static bool intel_sdvo_limited_color_range(struct intel_encoder *encoder, return intel_hdmi_limited_color_range(crtc_state, conn_state); } +static bool intel_sdvo_has_audio(struct intel_encoder *encoder, + const struct intel_crtc_state *crtc_state, + const struct drm_connector_state *conn_state) +{ + struct intel_sdvo *intel_sdvo = to_sdvo(encoder); + const struct intel_digital_connector_state *intel_conn_state = + to_intel_digital_connector_state(conn_state); + + if (!crtc_state->has_hdmi_sink) + return false; + + if (intel_conn_state->force_audio == HDMI_AUDIO_AUTO) + return intel_sdvo->has_hdmi_audio; + else + return intel_conn_state->force_audio == HDMI_AUDIO_ON; +} + static int intel_sdvo_compute_config(struct intel_encoder *encoder, struct intel_crtc_state *pipe_config, struct drm_connector_state *conn_state) { struct intel_sdvo *intel_sdvo = to_sdvo(encoder); - struct intel_sdvo_connector_state *intel_sdvo_state = - to_intel_sdvo_connector_state(conn_state); struct intel_sdvo_connector *intel_sdvo_connector = to_intel_sdvo_connector(conn_state->connector); struct drm_display_mode *adjusted_mode = &pipe_config->hw.adjusted_mode; @@ -1362,13 +1378,7 @@ static int intel_sdvo_compute_config(struct intel_encoder *encoder, pipe_config->has_hdmi_sink = intel_has_hdmi_sink(intel_sdvo, conn_state); - if (pipe_config->has_hdmi_sink) { - if (intel_sdvo_state->base.force_audio == HDMI_AUDIO_AUTO) - pipe_config->has_audio = intel_sdvo->has_hdmi_audio; - else - pipe_config->has_audio = - intel_sdvo_state->base.force_audio == HDMI_AUDIO_ON; - } + pipe_config->has_audio = intel_sdvo_has_audio(encoder, pipe_config, conn_state); pipe_config->limited_color_range = intel_sdvo_limited_color_range(encoder, pipe_config, @@ -2290,17 +2300,12 @@ static int intel_sdvo_get_tv_modes(struct drm_connector *connector) static int intel_sdvo_get_lvds_modes(struct drm_connector *connector) { - struct intel_sdvo *intel_sdvo = intel_attached_sdvo(to_intel_connector(connector)); struct drm_i915_private *dev_priv = to_i915(connector->dev); - int num_modes = 0; drm_dbg_kms(&dev_priv->drm, "[CONNECTOR:%d:%s]\n", connector->base.id, connector->name); - num_modes += intel_panel_get_modes(to_intel_connector(connector)); - num_modes += intel_ddc_get_modes(connector, &intel_sdvo->ddc); - - return num_modes; + return intel_panel_get_modes(to_intel_connector(connector)); } static int intel_sdvo_get_modes(struct drm_connector *connector) @@ -2627,7 +2632,7 @@ intel_sdvo_unselect_i2c_bus(struct intel_sdvo *sdvo) } static bool -intel_sdvo_is_hdmi_connector(struct intel_sdvo *intel_sdvo, int device) +intel_sdvo_is_hdmi_connector(struct intel_sdvo *intel_sdvo) { return intel_sdvo_check_supp_encode(intel_sdvo); } @@ -2689,9 +2694,8 @@ intel_sdvo_connector_init(struct intel_sdvo_connector *connector, drm_connector_helper_add(drm_connector, &intel_sdvo_connector_helper_funcs); - connector->base.base.interlace_allowed = 1; - connector->base.base.doublescan_allowed = 0; connector->base.base.display_info.subpixel_order = SubPixelHorizontalRGB; + connector->base.base.interlace_allowed = true; connector->base.get_hw_state = intel_sdvo_connector_get_hw_state; intel_connector_attach_encoder(&connector->base, &encoder->base); @@ -2733,7 +2737,7 @@ static struct intel_sdvo_connector *intel_sdvo_connector_alloc(void) } static bool -intel_sdvo_dvi_init(struct intel_sdvo *intel_sdvo, int device) +intel_sdvo_dvi_init(struct intel_sdvo *intel_sdvo, u16 type) { struct drm_encoder *encoder = &intel_sdvo->base.base; struct drm_connector *connector; @@ -2741,16 +2745,13 @@ intel_sdvo_dvi_init(struct intel_sdvo *intel_sdvo, int device) struct intel_connector *intel_connector; struct intel_sdvo_connector *intel_sdvo_connector; - DRM_DEBUG_KMS("initialising DVI device %d\n", device); + DRM_DEBUG_KMS("initialising DVI type 0x%x\n", type); intel_sdvo_connector = intel_sdvo_connector_alloc(); if (!intel_sdvo_connector) return false; - if (device == 0) - intel_sdvo_connector->output_flag = SDVO_OUTPUT_TMDS0; - else if (device == 1) - intel_sdvo_connector->output_flag = SDVO_OUTPUT_TMDS1; + intel_sdvo_connector->output_flag = type; intel_connector = &intel_sdvo_connector->base; connector = &intel_connector->base; @@ -2770,7 +2771,7 @@ intel_sdvo_dvi_init(struct intel_sdvo *intel_sdvo, int device) encoder->encoder_type = DRM_MODE_ENCODER_TMDS; connector->connector_type = DRM_MODE_CONNECTOR_DVID; - if (intel_sdvo_is_hdmi_connector(intel_sdvo, device)) { + if (intel_sdvo_is_hdmi_connector(intel_sdvo)) { connector->connector_type = DRM_MODE_CONNECTOR_HDMIA; intel_sdvo_connector->is_hdmi = true; } @@ -2787,14 +2788,14 @@ intel_sdvo_dvi_init(struct intel_sdvo *intel_sdvo, int device) } static bool -intel_sdvo_tv_init(struct intel_sdvo *intel_sdvo, int type) +intel_sdvo_tv_init(struct intel_sdvo *intel_sdvo, u16 type) { struct drm_encoder *encoder = &intel_sdvo->base.base; struct drm_connector *connector; struct intel_connector *intel_connector; struct intel_sdvo_connector *intel_sdvo_connector; - DRM_DEBUG_KMS("initialising TV type %d\n", type); + DRM_DEBUG_KMS("initialising TV type 0x%x\n", type); intel_sdvo_connector = intel_sdvo_connector_alloc(); if (!intel_sdvo_connector) @@ -2826,14 +2827,14 @@ err: } static bool -intel_sdvo_analog_init(struct intel_sdvo *intel_sdvo, int device) +intel_sdvo_analog_init(struct intel_sdvo *intel_sdvo, u16 type) { struct drm_encoder *encoder = &intel_sdvo->base.base; struct drm_connector *connector; struct intel_connector *intel_connector; struct intel_sdvo_connector *intel_sdvo_connector; - DRM_DEBUG_KMS("initialising analog device %d\n", device); + DRM_DEBUG_KMS("initialising analog type 0x%x\n", type); intel_sdvo_connector = intel_sdvo_connector_alloc(); if (!intel_sdvo_connector) @@ -2845,10 +2846,7 @@ intel_sdvo_analog_init(struct intel_sdvo *intel_sdvo, int device) encoder->encoder_type = DRM_MODE_ENCODER_DAC; connector->connector_type = DRM_MODE_CONNECTOR_VGA; - if (device == 0) - intel_sdvo_connector->output_flag = SDVO_OUTPUT_RGB0; - else if (device == 1) - intel_sdvo_connector->output_flag = SDVO_OUTPUT_RGB1; + intel_sdvo_connector->output_flag = type; if (intel_sdvo_connector_init(intel_sdvo_connector, intel_sdvo) < 0) { kfree(intel_sdvo_connector); @@ -2859,7 +2857,7 @@ intel_sdvo_analog_init(struct intel_sdvo *intel_sdvo, int device) } static bool -intel_sdvo_lvds_init(struct intel_sdvo *intel_sdvo, int device) +intel_sdvo_lvds_init(struct intel_sdvo *intel_sdvo, u16 type) { struct drm_encoder *encoder = &intel_sdvo->base.base; struct drm_i915_private *i915 = to_i915(encoder->dev); @@ -2867,7 +2865,7 @@ intel_sdvo_lvds_init(struct intel_sdvo *intel_sdvo, int device) struct intel_connector *intel_connector; struct intel_sdvo_connector *intel_sdvo_connector; - DRM_DEBUG_KMS("initialising LVDS device %d\n", device); + DRM_DEBUG_KMS("initialising LVDS type 0x%x\n", type); intel_sdvo_connector = intel_sdvo_connector_alloc(); if (!intel_sdvo_connector) @@ -2878,10 +2876,7 @@ intel_sdvo_lvds_init(struct intel_sdvo *intel_sdvo, int device) encoder->encoder_type = DRM_MODE_ENCODER_LVDS; connector->connector_type = DRM_MODE_CONNECTOR_LVDS; - if (device == 0) - intel_sdvo_connector->output_flag = SDVO_OUTPUT_LVDS0; - else if (device == 1) - intel_sdvo_connector->output_flag = SDVO_OUTPUT_LVDS1; + intel_sdvo_connector->output_flag = type; if (intel_sdvo_connector_init(intel_sdvo_connector, intel_sdvo) < 0) { kfree(intel_sdvo_connector); @@ -2937,63 +2932,61 @@ static u16 intel_sdvo_filter_output_flags(u16 flags) return flags; } -static bool -intel_sdvo_output_setup(struct intel_sdvo *intel_sdvo, u16 flags) +static bool intel_sdvo_output_init(struct intel_sdvo *sdvo, u16 type) { + if (type & SDVO_TMDS_MASK) + return intel_sdvo_dvi_init(sdvo, type); + else if (type & SDVO_TV_MASK) + return intel_sdvo_tv_init(sdvo, type); + else if (type & SDVO_RGB_MASK) + return intel_sdvo_analog_init(sdvo, type); + else if (type & SDVO_LVDS_MASK) + return intel_sdvo_lvds_init(sdvo, type); + else + return false; +} + +static bool +intel_sdvo_output_setup(struct intel_sdvo *intel_sdvo) +{ + static const u16 probe_order[] = { + SDVO_OUTPUT_TMDS0, + SDVO_OUTPUT_TMDS1, + /* TV has no XXX1 function block */ + SDVO_OUTPUT_SVID0, + SDVO_OUTPUT_CVBS0, + SDVO_OUTPUT_YPRPB0, + SDVO_OUTPUT_RGB0, + SDVO_OUTPUT_RGB1, + SDVO_OUTPUT_LVDS0, + SDVO_OUTPUT_LVDS1, + }; struct drm_i915_private *i915 = to_i915(intel_sdvo->base.base.dev); + u16 flags; + int i; - flags = intel_sdvo_filter_output_flags(flags); + flags = intel_sdvo_filter_output_flags(intel_sdvo->caps.output_flags); + + if (flags == 0) { + DRM_DEBUG_KMS("%s: Unknown SDVO output type (0x%04x)\n", + SDVO_NAME(intel_sdvo), intel_sdvo->caps.output_flags); + return false; + } intel_sdvo->controlled_output = flags; intel_sdvo_select_ddc_bus(i915, intel_sdvo); - if (flags & SDVO_OUTPUT_TMDS0) - if (!intel_sdvo_dvi_init(intel_sdvo, 0)) - return false; + for (i = 0; i < ARRAY_SIZE(probe_order); i++) { + u16 type = flags & probe_order[i]; - if (flags & SDVO_OUTPUT_TMDS1) - if (!intel_sdvo_dvi_init(intel_sdvo, 1)) - return false; - - /* TV has no XXX1 function block */ - if (flags & SDVO_OUTPUT_SVID0) - if (!intel_sdvo_tv_init(intel_sdvo, SDVO_OUTPUT_SVID0)) - return false; + if (!type) + continue; - if (flags & SDVO_OUTPUT_CVBS0) - if (!intel_sdvo_tv_init(intel_sdvo, SDVO_OUTPUT_CVBS0)) + if (!intel_sdvo_output_init(intel_sdvo, type)) return false; - - if (flags & SDVO_OUTPUT_YPRPB0) - if (!intel_sdvo_tv_init(intel_sdvo, SDVO_OUTPUT_YPRPB0)) - return false; - - if (flags & SDVO_OUTPUT_RGB0) - if (!intel_sdvo_analog_init(intel_sdvo, 0)) - return false; - - if (flags & SDVO_OUTPUT_RGB1) - if (!intel_sdvo_analog_init(intel_sdvo, 1)) - return false; - - if (flags & SDVO_OUTPUT_LVDS0) - if (!intel_sdvo_lvds_init(intel_sdvo, 0)) - return false; - - if (flags & SDVO_OUTPUT_LVDS1) - if (!intel_sdvo_lvds_init(intel_sdvo, 1)) - return false; - - if (flags == 0) { - unsigned char bytes[2]; - - memcpy(bytes, &intel_sdvo->caps.output_flags, 2); - DRM_DEBUG_KMS("%s: Unknown SDVO output type (0x%02x%02x)\n", - SDVO_NAME(intel_sdvo), - bytes[0], bytes[1]); - return false; } + intel_sdvo->base.pipe_mask = ~0; return true; @@ -3369,8 +3362,7 @@ bool intel_sdvo_init(struct drm_i915_private *dev_priv, intel_sdvo->colorimetry_cap = intel_sdvo_get_colorimetry_cap(intel_sdvo); - if (intel_sdvo_output_setup(intel_sdvo, - intel_sdvo->caps.output_flags) != true) { + if (!intel_sdvo_output_setup(intel_sdvo)) { drm_dbg_kms(&dev_priv->drm, "SDVO output failed to setup on %s\n", SDVO_NAME(intel_sdvo)); @@ -3421,9 +3413,12 @@ bool intel_sdvo_init(struct drm_i915_private *dev_priv, (intel_sdvo->caps.sdvo_inputs_mask & 0x2) ? 'Y' : 'N', /* check currently supported outputs */ intel_sdvo->caps.output_flags & - (SDVO_OUTPUT_TMDS0 | SDVO_OUTPUT_RGB0) ? 'Y' : 'N', + (SDVO_OUTPUT_TMDS0 | SDVO_OUTPUT_RGB0 | + SDVO_OUTPUT_LVDS0 | SDVO_OUTPUT_SVID0 | + SDVO_OUTPUT_CVBS0 | SDVO_OUTPUT_YPRPB0) ? 'Y' : 'N', intel_sdvo->caps.output_flags & - (SDVO_OUTPUT_TMDS1 | SDVO_OUTPUT_RGB1) ? 'Y' : 'N'); + (SDVO_OUTPUT_TMDS1 | SDVO_OUTPUT_RGB1 | + SDVO_OUTPUT_LVDS1) ? 'Y' : 'N'); return true; err_output: diff --git a/drivers/gpu/drm/i915/display/intel_snps_phy.c b/drivers/gpu/drm/i915/display/intel_snps_phy.c index 937cefd6f78f..c799e891f8b5 100644 --- a/drivers/gpu/drm/i915/display/intel_snps_phy.c +++ b/drivers/gpu/drm/i915/display/intel_snps_phy.c @@ -5,6 +5,7 @@ #include <linux/util_macros.h> +#include "i915_reg.h" #include "intel_ddi.h" #include "intel_ddi_buf_trans.h" #include "intel_de.h" diff --git a/drivers/gpu/drm/i915/display/intel_snps_phy_regs.h b/drivers/gpu/drm/i915/display/intel_snps_phy_regs.h index 0543465aaf14..a04d692169d4 100644 --- a/drivers/gpu/drm/i915/display/intel_snps_phy_regs.h +++ b/drivers/gpu/drm/i915/display/intel_snps_phy_regs.h @@ -6,7 +6,7 @@ #ifndef __INTEL_SNPS_PHY_REGS__ #define __INTEL_SNPS_PHY_REGS__ -#include "i915_reg_defs.h" +#include "intel_display_reg_defs.h" #define _SNPS_PHY_A_BASE 0x168000 #define _SNPS_PHY_B_BASE 0x169000 diff --git a/drivers/gpu/drm/i915/display/intel_sprite.c b/drivers/gpu/drm/i915/display/intel_sprite.c index 7649c50b5445..e6b4d24b9cd0 100644 --- a/drivers/gpu/drm/i915/display/intel_sprite.c +++ b/drivers/gpu/drm/i915/display/intel_sprite.c @@ -42,6 +42,7 @@ #include <drm/drm_rect.h> #include "i915_drv.h" +#include "i915_reg.h" #include "i915_vgpu.h" #include "i9xx_plane.h" #include "intel_atomic_plane.h" diff --git a/drivers/gpu/drm/i915/display/intel_sprite.h b/drivers/gpu/drm/i915/display/intel_sprite.h index 4f63e4967731..4635c7ad23f9 100644 --- a/drivers/gpu/drm/i915/display/intel_sprite.h +++ b/drivers/gpu/drm/i915/display/intel_sprite.h @@ -8,14 +8,13 @@ #include <linux/types.h> -#include "intel_display.h" - struct drm_device; struct drm_display_mode; struct drm_file; struct drm_i915_private; struct intel_crtc_state; struct intel_plane_state; +enum pipe; /* * FIXME: We should instead only take spinlocks once for the entire update @@ -34,12 +33,6 @@ int intel_sprite_set_colorkey_ioctl(struct drm_device *dev, void *data, int intel_plane_check_src_coordinates(struct intel_plane_state *plane_state); int chv_plane_check_rotation(const struct intel_plane_state *plane_state); -static inline u8 icl_hdr_plane_mask(void) -{ - return BIT(PLANE_PRIMARY) | - BIT(PLANE_SPRITE0) | BIT(PLANE_SPRITE1); -} - int ivb_plane_min_cdclk(const struct intel_crtc_state *crtc_state, const struct intel_plane_state *plane_state); int hsw_plane_min_cdclk(const struct intel_crtc_state *crtc_state, diff --git a/drivers/gpu/drm/i915/display/intel_tc.c b/drivers/gpu/drm/i915/display/intel_tc.c index e5af955b5600..70624b4b2d38 100644 --- a/drivers/gpu/drm/i915/display/intel_tc.c +++ b/drivers/gpu/drm/i915/display/intel_tc.c @@ -8,9 +8,10 @@ #include "intel_display.h" #include "intel_display_power_map.h" #include "intel_display_types.h" +#include "intel_dkl_phy_regs.h" #include "intel_dp_mst.h" +#include "intel_mg_phy_regs.h" #include "intel_tc.h" -#include "intel_tc_phy_regs.h" static const char *tc_port_mode_name(enum tc_port_mode mode) { @@ -408,14 +409,9 @@ static bool adl_tc_phy_take_ownership(struct intel_digital_port *dig_port, struct drm_i915_private *i915 = to_i915(dig_port->base.base.dev); struct intel_uncore *uncore = &i915->uncore; enum port port = dig_port->base.port; - u32 val; - val = intel_uncore_read(uncore, DDI_BUF_CTL(port)); - if (take) - val |= DDI_BUF_CTL_TC_PHY_OWNERSHIP; - else - val &= ~DDI_BUF_CTL_TC_PHY_OWNERSHIP; - intel_uncore_write(uncore, DDI_BUF_CTL(port), val); + intel_uncore_rmw(uncore, DDI_BUF_CTL(port), DDI_BUF_CTL_TC_PHY_OWNERSHIP, + take ? DDI_BUF_CTL_TC_PHY_OWNERSHIP : 0); return true; } @@ -687,18 +683,58 @@ static void intel_tc_port_link_init_refcount(struct intel_digital_port *dig_port, int refcount) { + dig_port->tc_link_refcount = refcount; +} + +/** + * intel_tc_port_init_mode: Read out HW state and init the given port's TypeC mode + * @dig_port: digital port + * + * Read out the HW state and initialize the TypeC mode of @dig_port. The mode + * will be locked until intel_tc_port_sanitize_mode() is called. + */ +void intel_tc_port_init_mode(struct intel_digital_port *dig_port) +{ struct drm_i915_private *i915 = to_i915(dig_port->base.base.dev); + intel_wakeref_t tc_cold_wref; + enum intel_display_power_domain domain; + + mutex_lock(&dig_port->tc_lock); + drm_WARN_ON(&i915->drm, dig_port->tc_mode != TC_PORT_DISCONNECTED); + drm_WARN_ON(&i915->drm, dig_port->tc_lock_wakeref); drm_WARN_ON(&i915->drm, dig_port->tc_link_refcount); - dig_port->tc_link_refcount = refcount; + + tc_cold_wref = tc_cold_block(dig_port, &domain); + + dig_port->tc_mode = intel_tc_port_get_current_mode(dig_port); + /* Prevent changing dig_port->tc_mode until intel_tc_port_sanitize_mode() is called. */ + intel_tc_port_link_init_refcount(dig_port, 1); + dig_port->tc_lock_wakeref = tc_cold_block(dig_port, &dig_port->tc_lock_power_domain); + + tc_cold_unblock(dig_port, domain, tc_cold_wref); + + drm_dbg_kms(&i915->drm, "Port %s: init mode (%s)\n", + dig_port->tc_port_name, + tc_port_mode_name(dig_port->tc_mode)); + + mutex_unlock(&dig_port->tc_lock); } -void intel_tc_port_sanitize(struct intel_digital_port *dig_port) +/** + * intel_tc_port_sanitize_mode: Sanitize the given port's TypeC mode + * @dig_port: digital port + * + * Sanitize @dig_port's TypeC mode wrt. the encoder's state right after driver + * loading and system resume: + * If the encoder is enabled keep the TypeC mode/PHY connected state locked until + * the encoder is disabled. + * If the encoder is disabled make sure the PHY is disconnected. + */ +void intel_tc_port_sanitize_mode(struct intel_digital_port *dig_port) { struct drm_i915_private *i915 = to_i915(dig_port->base.base.dev); struct intel_encoder *encoder = &dig_port->base; - intel_wakeref_t tc_cold_wref; - enum intel_display_power_domain domain; int active_links = 0; mutex_lock(&dig_port->tc_lock); @@ -708,21 +744,14 @@ void intel_tc_port_sanitize(struct intel_digital_port *dig_port) else if (encoder->base.crtc) active_links = to_intel_crtc(encoder->base.crtc)->active; - drm_WARN_ON(&i915->drm, dig_port->tc_mode != TC_PORT_DISCONNECTED); - drm_WARN_ON(&i915->drm, dig_port->tc_lock_wakeref); - - tc_cold_wref = tc_cold_block(dig_port, &domain); + drm_WARN_ON(&i915->drm, dig_port->tc_link_refcount != 1); + intel_tc_port_link_init_refcount(dig_port, active_links); - dig_port->tc_mode = intel_tc_port_get_current_mode(dig_port); if (active_links) { if (!icl_tc_phy_is_connected(dig_port)) drm_dbg_kms(&i915->drm, "Port %s: PHY disconnected with %d active link(s)\n", dig_port->tc_port_name, active_links); - intel_tc_port_link_init_refcount(dig_port, active_links); - - dig_port->tc_lock_wakeref = tc_cold_block(dig_port, - &dig_port->tc_lock_power_domain); } else { /* * TBT-alt is the default mode in any case the PHY ownership is not @@ -736,9 +765,10 @@ void intel_tc_port_sanitize(struct intel_digital_port *dig_port) dig_port->tc_port_name, tc_port_mode_name(dig_port->tc_mode)); icl_tc_phy_disconnect(dig_port); - } - tc_cold_unblock(dig_port, domain, tc_cold_wref); + tc_cold_unblock(dig_port, dig_port->tc_lock_power_domain, + fetch_and_zero(&dig_port->tc_lock_wakeref)); + } drm_dbg_kms(&i915->drm, "Port %s: sanitize mode (%s)\n", dig_port->tc_port_name, @@ -923,4 +953,6 @@ void intel_tc_port_init(struct intel_digital_port *dig_port, bool is_legacy) dig_port->tc_mode = TC_PORT_DISCONNECTED; dig_port->tc_link_refcount = 0; tc_port_load_fia_params(i915, dig_port); + + intel_tc_port_init_mode(dig_port); } diff --git a/drivers/gpu/drm/i915/display/intel_tc.h b/drivers/gpu/drm/i915/display/intel_tc.h index 6b47b29f551c..d54082e2d5e8 100644 --- a/drivers/gpu/drm/i915/display/intel_tc.h +++ b/drivers/gpu/drm/i915/display/intel_tc.h @@ -24,7 +24,8 @@ int intel_tc_port_fia_max_lane_count(struct intel_digital_port *dig_port); void intel_tc_port_set_fia_lane_count(struct intel_digital_port *dig_port, int required_lanes); -void intel_tc_port_sanitize(struct intel_digital_port *dig_port); +void intel_tc_port_init_mode(struct intel_digital_port *dig_port); +void intel_tc_port_sanitize_mode(struct intel_digital_port *dig_port); void intel_tc_port_lock(struct intel_digital_port *dig_port); void intel_tc_port_unlock(struct intel_digital_port *dig_port); void intel_tc_port_flush_work(struct intel_digital_port *dig_port); diff --git a/drivers/gpu/drm/i915/display/intel_tv.c b/drivers/gpu/drm/i915/display/intel_tv.c index dcf89d701f0f..4d2101ca1692 100644 --- a/drivers/gpu/drm/i915/display/intel_tv.c +++ b/drivers/gpu/drm/i915/display/intel_tv.c @@ -35,6 +35,8 @@ #include <drm/drm_edid.h> #include "i915_drv.h" +#include "i915_reg.h" +#include "i915_irq.h" #include "intel_connector.h" #include "intel_crtc.h" #include "intel_de.h" @@ -1880,18 +1882,56 @@ static const struct drm_encoder_funcs intel_tv_enc_funcs = { .destroy = intel_encoder_destroy, }; +static void intel_tv_add_properties(struct drm_connector *connector) +{ + struct drm_i915_private *i915 = to_i915(connector->dev); + struct drm_connector_state *conn_state = connector->state; + const char *tv_format_names[ARRAY_SIZE(tv_modes)]; + int i; + + /* BIOS margin values */ + conn_state->tv.margins.left = 54; + conn_state->tv.margins.top = 36; + conn_state->tv.margins.right = 46; + conn_state->tv.margins.bottom = 37; + + conn_state->tv.mode = 0; + + /* Create TV properties then attach current values */ + for (i = 0; i < ARRAY_SIZE(tv_modes); i++) { + /* 1080p50/1080p60 not supported on gen3 */ + if (DISPLAY_VER(i915) == 3 && tv_modes[i].oversample == 1) + break; + + tv_format_names[i] = tv_modes[i].name; + } + drm_mode_create_tv_properties(&i915->drm, i, tv_format_names); + + drm_object_attach_property(&connector->base, + i915->drm.mode_config.tv_mode_property, + conn_state->tv.mode); + drm_object_attach_property(&connector->base, + i915->drm.mode_config.tv_left_margin_property, + conn_state->tv.margins.left); + drm_object_attach_property(&connector->base, + i915->drm.mode_config.tv_top_margin_property, + conn_state->tv.margins.top); + drm_object_attach_property(&connector->base, + i915->drm.mode_config.tv_right_margin_property, + conn_state->tv.margins.right); + drm_object_attach_property(&connector->base, + i915->drm.mode_config.tv_bottom_margin_property, + conn_state->tv.margins.bottom); +} + void intel_tv_init(struct drm_i915_private *dev_priv) { - struct drm_device *dev = &dev_priv->drm; struct drm_connector *connector; struct intel_tv *intel_tv; struct intel_encoder *intel_encoder; struct intel_connector *intel_connector; u32 tv_dac_on, tv_dac_off, save_tv_dac; - const char *tv_format_names[ARRAY_SIZE(tv_modes)]; - int i, initial_mode = 0; - struct drm_connector_state *state; if ((intel_de_read(dev_priv, TV_CTL) & TV_FUSE_STATE_MASK) == TV_FUSE_STATE_DISABLED) return; @@ -1937,7 +1977,6 @@ intel_tv_init(struct drm_i915_private *dev_priv) intel_encoder = &intel_tv->base; connector = &intel_connector->base; - state = connector->state; /* * The documentation, for the older chipsets at least, recommend @@ -1951,10 +1990,10 @@ intel_tv_init(struct drm_i915_private *dev_priv) */ intel_connector->polled = DRM_CONNECTOR_POLL_CONNECT; - drm_connector_init(dev, connector, &intel_tv_connector_funcs, + drm_connector_init(&dev_priv->drm, connector, &intel_tv_connector_funcs, DRM_MODE_CONNECTOR_SVIDEO); - drm_encoder_init(dev, &intel_encoder->base, &intel_tv_enc_funcs, + drm_encoder_init(&dev_priv->drm, &intel_encoder->base, &intel_tv_enc_funcs, DRM_MODE_ENCODER_TVDAC, "TV"); intel_encoder->compute_config = intel_tv_compute_config; @@ -1974,41 +2013,7 @@ intel_tv_init(struct drm_i915_private *dev_priv) intel_encoder->cloneable = 0; intel_tv->type = DRM_MODE_CONNECTOR_Unknown; - /* BIOS margin values */ - state->tv.margins.left = 54; - state->tv.margins.top = 36; - state->tv.margins.right = 46; - state->tv.margins.bottom = 37; - - state->tv.mode = initial_mode; - drm_connector_helper_add(connector, &intel_tv_connector_helper_funcs); - connector->interlace_allowed = false; - connector->doublescan_allowed = false; - /* Create TV properties then attach current values */ - for (i = 0; i < ARRAY_SIZE(tv_modes); i++) { - /* 1080p50/1080p60 not supported on gen3 */ - if (DISPLAY_VER(dev_priv) == 3 && - tv_modes[i].oversample == 1) - break; - - tv_format_names[i] = tv_modes[i].name; - } - drm_mode_create_tv_properties(dev, i, tv_format_names); - - drm_object_attach_property(&connector->base, dev->mode_config.tv_mode_property, - state->tv.mode); - drm_object_attach_property(&connector->base, - dev->mode_config.tv_left_margin_property, - state->tv.margins.left); - drm_object_attach_property(&connector->base, - dev->mode_config.tv_top_margin_property, - state->tv.margins.top); - drm_object_attach_property(&connector->base, - dev->mode_config.tv_right_margin_property, - state->tv.margins.right); - drm_object_attach_property(&connector->base, - dev->mode_config.tv_bottom_margin_property, - state->tv.margins.bottom); + intel_tv_add_properties(connector); } diff --git a/drivers/gpu/drm/i915/display/intel_vdsc.c b/drivers/gpu/drm/i915/display/intel_vdsc.c index 269f9792390d..9d3b77b41b5c 100644 --- a/drivers/gpu/drm/i915/display/intel_vdsc.c +++ b/drivers/gpu/drm/i915/display/intel_vdsc.c @@ -10,6 +10,7 @@ #include <drm/display/drm_dsc_helper.h> #include "i915_drv.h" +#include "i915_reg.h" #include "intel_crtc.h" #include "intel_de.h" #include "intel_display_types.h" diff --git a/drivers/gpu/drm/i915/display/intel_vga.c b/drivers/gpu/drm/i915/display/intel_vga.c index b5d058404c14..a69bfcac9a94 100644 --- a/drivers/gpu/drm/i915/display/intel_vga.c +++ b/drivers/gpu/drm/i915/display/intel_vga.c @@ -10,6 +10,7 @@ #include <video/vga.h> #include "i915_drv.h" +#include "i915_reg.h" #include "intel_de.h" #include "intel_vga.h" diff --git a/drivers/gpu/drm/i915/display/intel_vrr.c b/drivers/gpu/drm/i915/display/intel_vrr.c index 5eac99021875..7b1357e82b69 100644 --- a/drivers/gpu/drm/i915/display/intel_vrr.c +++ b/drivers/gpu/drm/i915/display/intel_vrr.c @@ -5,6 +5,7 @@ */ #include "i915_drv.h" +#include "i915_reg.h" #include "intel_de.h" #include "intel_display_types.h" #include "intel_vrr.h" diff --git a/drivers/gpu/drm/i915/display/skl_scaler.c b/drivers/gpu/drm/i915/display/skl_scaler.c index 4092679be21e..d7390067b7d4 100644 --- a/drivers/gpu/drm/i915/display/skl_scaler.c +++ b/drivers/gpu/drm/i915/display/skl_scaler.c @@ -2,6 +2,8 @@ /* * Copyright © 2020 Intel Corporation */ + +#include "i915_reg.h" #include "intel_de.h" #include "intel_display_types.h" #include "intel_fb.h" diff --git a/drivers/gpu/drm/i915/display/skl_universal_plane.c b/drivers/gpu/drm/i915/display/skl_universal_plane.c index 7cb713043408..76490cc59d8f 100644 --- a/drivers/gpu/drm/i915/display/skl_universal_plane.c +++ b/drivers/gpu/drm/i915/display/skl_universal_plane.c @@ -9,6 +9,8 @@ #include <drm/drm_fourcc.h> #include "i915_drv.h" +#include "i915_irq.h" +#include "i915_reg.h" #include "intel_atomic_plane.h" #include "intel_de.h" #include "intel_display_types.h" @@ -246,6 +248,11 @@ bool icl_is_nv12_y_plane(struct drm_i915_private *dev_priv, icl_nv12_y_plane_mask(dev_priv) & BIT(plane_id); } +u8 icl_hdr_plane_mask(void) +{ + return BIT(PLANE_PRIMARY) | BIT(PLANE_SPRITE0) | BIT(PLANE_SPRITE1); +} + bool icl_is_hdr_plane(struct drm_i915_private *dev_priv, enum plane_id plane_id) { return DISPLAY_VER(dev_priv) >= 11 && diff --git a/drivers/gpu/drm/i915/display/skl_universal_plane.h b/drivers/gpu/drm/i915/display/skl_universal_plane.h index 351040b64dc7..be64c201f9b3 100644 --- a/drivers/gpu/drm/i915/display/skl_universal_plane.h +++ b/drivers/gpu/drm/i915/display/skl_universal_plane.h @@ -30,6 +30,7 @@ int skl_calc_main_surface_offset(const struct intel_plane_state *plane_state, bool icl_is_nv12_y_plane(struct drm_i915_private *dev_priv, enum plane_id plane_id); +u8 icl_hdr_plane_mask(void); bool icl_is_hdr_plane(struct drm_i915_private *dev_priv, enum plane_id plane_id); #endif diff --git a/drivers/gpu/drm/i915/display/skl_watermark.c b/drivers/gpu/drm/i915/display/skl_watermark.c index 18178b01375e..e0766d1be966 100644 --- a/drivers/gpu/drm/i915/display/skl_watermark.c +++ b/drivers/gpu/drm/i915/display/skl_watermark.c @@ -1706,26 +1706,10 @@ skl_compute_wm_params(const struct intel_crtc_state *crtc_state, return -EINVAL; } - wp->y_tiled = modifier == I915_FORMAT_MOD_Y_TILED || - modifier == I915_FORMAT_MOD_4_TILED || - modifier == I915_FORMAT_MOD_Yf_TILED || - modifier == I915_FORMAT_MOD_Y_TILED_CCS || - modifier == I915_FORMAT_MOD_Yf_TILED_CCS || - modifier == I915_FORMAT_MOD_Y_TILED_GEN12_RC_CCS || - modifier == I915_FORMAT_MOD_Y_TILED_GEN12_MC_CCS || - modifier == I915_FORMAT_MOD_Y_TILED_GEN12_RC_CCS_CC || - modifier == I915_FORMAT_MOD_4_TILED_DG2_RC_CCS || - modifier == I915_FORMAT_MOD_4_TILED_DG2_MC_CCS || - modifier == I915_FORMAT_MOD_4_TILED_DG2_RC_CCS_CC; wp->x_tiled = modifier == I915_FORMAT_MOD_X_TILED; - wp->rc_surface = modifier == I915_FORMAT_MOD_Y_TILED_CCS || - modifier == I915_FORMAT_MOD_Yf_TILED_CCS || - modifier == I915_FORMAT_MOD_Y_TILED_GEN12_RC_CCS || - modifier == I915_FORMAT_MOD_Y_TILED_GEN12_MC_CCS || - modifier == I915_FORMAT_MOD_Y_TILED_GEN12_RC_CCS_CC || - modifier == I915_FORMAT_MOD_4_TILED_DG2_RC_CCS || - modifier == I915_FORMAT_MOD_4_TILED_DG2_MC_CCS || - modifier == I915_FORMAT_MOD_4_TILED_DG2_RC_CCS_CC; + wp->y_tiled = modifier != I915_FORMAT_MOD_X_TILED && + intel_fb_is_tiled_modifier(modifier); + wp->rc_surface = intel_fb_is_ccs_modifier(modifier); wp->is_planar = intel_format_info_is_yuv_semiplanar(format, modifier); wp->width = width; @@ -2498,7 +2482,7 @@ skl_compute_ddb(struct intel_atomic_state *state) if (old_dbuf_state->joined_mbus != new_dbuf_state->joined_mbus) { /* TODO: Implement vblank synchronized MBUS joining changes */ - ret = intel_modeset_all_pipes(state); + ret = intel_modeset_all_pipes(state, "MBUS joining change"); if (ret) return ret; } @@ -2760,7 +2744,7 @@ static int skl_wm_add_affected_planes(struct intel_atomic_state *state, * power well the hardware state will go out of sync * with the software state. */ - if (!drm_atomic_crtc_needs_modeset(&new_crtc_state->uapi) && + if (!intel_crtc_needs_modeset(new_crtc_state) && skl_plane_selected_wm_equals(plane, &old_crtc_state->wm.skl.optimal, &new_crtc_state->wm.skl.optimal)) diff --git a/drivers/gpu/drm/i915/display/vlv_dsi.c b/drivers/gpu/drm/i915/display/vlv_dsi.c index b3f5ca280ef2..84481030883a 100644 --- a/drivers/gpu/drm/i915/display/vlv_dsi.c +++ b/drivers/gpu/drm/i915/display/vlv_dsi.c @@ -31,6 +31,7 @@ #include <drm/drm_mipi_dsi.h> #include "i915_drv.h" +#include "i915_reg.h" #include "intel_atomic.h" #include "intel_backlight.h" #include "intel_connector.h" @@ -1659,19 +1660,10 @@ static const struct drm_connector_funcs intel_dsi_connector_funcs = { static void vlv_dsi_add_properties(struct intel_connector *connector) { - struct drm_i915_private *dev_priv = to_i915(connector->base.dev); const struct drm_display_mode *fixed_mode = intel_panel_preferred_fixed_mode(connector); - u32 allowed_scalers; - allowed_scalers = BIT(DRM_MODE_SCALE_ASPECT) | BIT(DRM_MODE_SCALE_FULLSCREEN); - if (!HAS_GMCH(dev_priv)) - allowed_scalers |= BIT(DRM_MODE_SCALE_CENTER); - - drm_connector_attach_scaling_mode_property(&connector->base, - allowed_scalers); - - connector->base.state->scaling_mode = DRM_MODE_SCALE_ASPECT; + intel_attach_scaling_mode_property(&connector->base); drm_connector_set_panel_orientation_with_quirk(&connector->base, intel_dsi_get_panel_orientation(connector), @@ -1854,7 +1846,6 @@ static void vlv_dphy_param_init(struct intel_dsi *intel_dsi) void vlv_dsi_init(struct drm_i915_private *dev_priv) { - struct drm_device *dev = &dev_priv->drm; struct intel_dsi *intel_dsi; struct intel_encoder *intel_encoder; struct drm_encoder *encoder; @@ -1891,7 +1882,7 @@ void vlv_dsi_init(struct drm_i915_private *dev_priv) connector = &intel_connector->base; - drm_encoder_init(dev, encoder, &intel_dsi_funcs, DRM_MODE_ENCODER_DSI, + drm_encoder_init(&dev_priv->drm, encoder, &intel_dsi_funcs, DRM_MODE_ENCODER_DSI, "DSI %c", port_name(port)); intel_encoder->compute_config = intel_dsi_compute_config; @@ -1974,20 +1965,18 @@ void vlv_dsi_init(struct drm_i915_private *dev_priv) intel_dsi_vbt_gpio_init(intel_dsi, intel_dsi_get_hw_state(intel_encoder, &pipe)); - drm_connector_init(dev, connector, &intel_dsi_connector_funcs, + drm_connector_init(&dev_priv->drm, connector, &intel_dsi_connector_funcs, DRM_MODE_CONNECTOR_DSI); drm_connector_helper_add(connector, &intel_dsi_connector_helper_funcs); connector->display_info.subpixel_order = SubPixelHorizontalRGB; /*XXX*/ - connector->interlace_allowed = false; - connector->doublescan_allowed = false; intel_connector_attach_encoder(intel_connector, intel_encoder); - mutex_lock(&dev->mode_config.mutex); + mutex_lock(&dev_priv->drm.mode_config.mutex); intel_panel_add_vbt_lfp_fixed_mode(intel_connector); - mutex_unlock(&dev->mode_config.mutex); + mutex_unlock(&dev_priv->drm.mode_config.mutex); if (!intel_panel_preferred_fixed_mode(intel_connector)) { drm_dbg_kms(&dev_priv->drm, "no fixed mode\n"); diff --git a/drivers/gpu/drm/i915/display/vlv_dsi_regs.h b/drivers/gpu/drm/i915/display/vlv_dsi_regs.h index e065b8f2ee08..abbe427e462e 100644 --- a/drivers/gpu/drm/i915/display/vlv_dsi_regs.h +++ b/drivers/gpu/drm/i915/display/vlv_dsi_regs.h @@ -6,7 +6,7 @@ #ifndef __VLV_DSI_REGS_H__ #define __VLV_DSI_REGS_H__ -#include "i915_reg_defs.h" +#include "intel_display_reg_defs.h" #define VLV_MIPI_BASE VLV_DISPLAY_BASE #define BXT_MIPI_BASE 0x60000 diff --git a/drivers/gpu/drm/i915/gem/i915_gem_clflush.c b/drivers/gpu/drm/i915/gem/i915_gem_clflush.c index 0512afdd20d8..b3b398fe689c 100644 --- a/drivers/gpu/drm/i915/gem/i915_gem_clflush.c +++ b/drivers/gpu/drm/i915/gem/i915_gem_clflush.c @@ -113,7 +113,7 @@ bool i915_gem_clflush_object(struct drm_i915_gem_object *obj, clflush = clflush_work_create(obj); if (clflush) { i915_sw_fence_await_reservation(&clflush->base.chain, - obj->base.resv, NULL, true, + obj->base.resv, true, i915_fence_timeout(i915), I915_FENCE_GFP); dma_resv_add_fence(obj->base.resv, &clflush->base.dma, diff --git a/drivers/gpu/drm/i915/gem/i915_gem_context.c b/drivers/gpu/drm/i915/gem/i915_gem_context.c index 1e29b1e6d186..7f2831efc798 100644 --- a/drivers/gpu/drm/i915/gem/i915_gem_context.c +++ b/drivers/gpu/drm/i915/gem/i915_gem_context.c @@ -546,7 +546,7 @@ set_proto_ctx_engines_bond(struct i915_user_extension __user *base, void *data) } if (intel_engine_uses_guc(master)) { - DRM_DEBUG("bonding extension not supported with GuC submission"); + drm_dbg(&i915->drm, "bonding extension not supported with GuC submission"); return -ENODEV; } @@ -1452,7 +1452,7 @@ static void engines_idle_release(struct i915_gem_context *ctx, int err; /* serialises with execbuf */ - set_bit(CONTEXT_CLOSED_BIT, &ce->flags); + intel_context_close(ce); if (!intel_context_pin_if_active(ce)) continue; @@ -2298,7 +2298,6 @@ int i915_gem_context_create_ioctl(struct drm_device *dev, void *data, } args->ctx_id = id; - drm_dbg(&i915->drm, "HW context %d created\n", args->ctx_id); return 0; diff --git a/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c b/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c index 824971a1ceec..fd556a076d05 100644 --- a/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c +++ b/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c @@ -25,43 +25,44 @@ static struct drm_i915_gem_object *dma_buf_to_obj(struct dma_buf *buf) return to_intel_bo(buf->priv); } -static struct sg_table *i915_gem_map_dma_buf(struct dma_buf_attachment *attachment, +static struct sg_table *i915_gem_map_dma_buf(struct dma_buf_attachment *attach, enum dma_data_direction dir) { - struct drm_i915_gem_object *obj = dma_buf_to_obj(attachment->dmabuf); - struct sg_table *st; + struct drm_i915_gem_object *obj = dma_buf_to_obj(attach->dmabuf); + struct sg_table *sgt; struct scatterlist *src, *dst; int ret, i; - /* Copy sg so that we make an independent mapping */ - st = kmalloc(sizeof(struct sg_table), GFP_KERNEL); - if (st == NULL) { + /* + * Make a copy of the object's sgt, so that we can make an independent + * mapping + */ + sgt = kmalloc(sizeof(*sgt), GFP_KERNEL); + if (!sgt) { ret = -ENOMEM; goto err; } - ret = sg_alloc_table(st, obj->mm.pages->orig_nents, GFP_KERNEL); + ret = sg_alloc_table(sgt, obj->mm.pages->orig_nents, GFP_KERNEL); if (ret) goto err_free; - src = obj->mm.pages->sgl; - dst = st->sgl; - for (i = 0; i < obj->mm.pages->orig_nents; i++) { + dst = sgt->sgl; + for_each_sg(obj->mm.pages->sgl, src, obj->mm.pages->orig_nents, i) { sg_set_page(dst, sg_page(src), src->length, 0); dst = sg_next(dst); - src = sg_next(src); } - ret = dma_map_sgtable(attachment->dev, st, dir, DMA_ATTR_SKIP_CPU_SYNC); + ret = dma_map_sgtable(attach->dev, sgt, dir, DMA_ATTR_SKIP_CPU_SYNC); if (ret) goto err_free_sg; - return st; + return sgt; err_free_sg: - sg_free_table(st); + sg_free_table(sgt); err_free: - kfree(st); + kfree(sgt); err: return ERR_PTR(ret); } @@ -72,7 +73,7 @@ static int i915_gem_dmabuf_vmap(struct dma_buf *dma_buf, struct drm_i915_gem_object *obj = dma_buf_to_obj(dma_buf); void *vaddr; - vaddr = i915_gem_object_pin_map_unlocked(obj, I915_MAP_WB); + vaddr = i915_gem_object_pin_map(obj, I915_MAP_WB); if (IS_ERR(vaddr)) return PTR_ERR(vaddr); @@ -96,6 +97,8 @@ static int i915_gem_dmabuf_mmap(struct dma_buf *dma_buf, struct vm_area_struct * struct drm_i915_private *i915 = to_i915(obj->base.dev); int ret; + dma_resv_assert_held(dma_buf->resv); + if (obj->base.size < vma->vm_end - vma->vm_start) return -EINVAL; @@ -236,15 +239,14 @@ struct dma_buf *i915_gem_prime_export(struct drm_gem_object *gem_obj, int flags) static int i915_gem_object_get_pages_dmabuf(struct drm_i915_gem_object *obj) { struct drm_i915_private *i915 = to_i915(obj->base.dev); - struct sg_table *pages; - unsigned int sg_page_sizes; + struct sg_table *sgt; assert_object_held(obj); - pages = dma_buf_map_attachment(obj->base.import_attach, - DMA_BIDIRECTIONAL); - if (IS_ERR(pages)) - return PTR_ERR(pages); + sgt = dma_buf_map_attachment(obj->base.import_attach, + DMA_BIDIRECTIONAL); + if (IS_ERR(sgt)) + return PTR_ERR(sgt); /* * DG1 is special here since it still snoops transactions even with @@ -261,16 +263,15 @@ static int i915_gem_object_get_pages_dmabuf(struct drm_i915_gem_object *obj) (!HAS_LLC(i915) && !IS_DG1(i915))) wbinvd_on_all_cpus(); - sg_page_sizes = i915_sg_dma_sizes(pages->sgl); - __i915_gem_object_set_pages(obj, pages, sg_page_sizes); + __i915_gem_object_set_pages(obj, sgt); return 0; } static void i915_gem_object_put_pages_dmabuf(struct drm_i915_gem_object *obj, - struct sg_table *pages) + struct sg_table *sgt) { - dma_buf_unmap_attachment(obj->base.import_attach, pages, + dma_buf_unmap_attachment(obj->base.import_attach, sgt, DMA_BIDIRECTIONAL); } @@ -313,7 +314,7 @@ struct drm_gem_object *i915_gem_prime_import(struct drm_device *dev, get_dma_buf(dma_buf); obj = i915_gem_object_alloc(); - if (obj == NULL) { + if (!obj) { ret = -ENOMEM; goto fail_detach; } diff --git a/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c index 29d2459bcc90..da09767fda07 100644 --- a/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c +++ b/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c @@ -30,6 +30,7 @@ #include "i915_gem_context.h" #include "i915_gem_evict.h" #include "i915_gem_ioctls.h" +#include "i915_reg.h" #include "i915_trace.h" #include "i915_user_extensions.h" @@ -53,13 +54,13 @@ enum { #define DBG_FORCE_RELOC 0 /* choose one of the above! */ }; -/* __EXEC_OBJECT_NO_RESERVE is BIT(31), defined in i915_vma.h */ -#define __EXEC_OBJECT_HAS_PIN BIT(30) -#define __EXEC_OBJECT_HAS_FENCE BIT(29) -#define __EXEC_OBJECT_USERPTR_INIT BIT(28) -#define __EXEC_OBJECT_NEEDS_MAP BIT(27) -#define __EXEC_OBJECT_NEEDS_BIAS BIT(26) -#define __EXEC_OBJECT_INTERNAL_FLAGS (~0u << 26) /* all of the above + */ +/* __EXEC_OBJECT_ flags > BIT(29) defined in i915_vma.h */ +#define __EXEC_OBJECT_HAS_PIN BIT(29) +#define __EXEC_OBJECT_HAS_FENCE BIT(28) +#define __EXEC_OBJECT_USERPTR_INIT BIT(27) +#define __EXEC_OBJECT_NEEDS_MAP BIT(26) +#define __EXEC_OBJECT_NEEDS_BIAS BIT(25) +#define __EXEC_OBJECT_INTERNAL_FLAGS (~0u << 25) /* all of the above + */ #define __EXEC_OBJECT_RESERVED (__EXEC_OBJECT_HAS_PIN | __EXEC_OBJECT_HAS_FENCE) #define __EXEC_HAS_RELOC BIT(31) @@ -2101,7 +2102,8 @@ static int eb_move_to_gpu(struct i915_execbuffer *eb) eb->composite_fence ? eb->composite_fence : &eb->requests[j]->fence, - flags | __EXEC_OBJECT_NO_RESERVE); + flags | __EXEC_OBJECT_NO_RESERVE | + __EXEC_OBJECT_NO_REQUEST_AWAIT); } } @@ -2148,7 +2150,8 @@ err_skip: return err; } -static int i915_gem_check_execbuffer(struct drm_i915_gem_execbuffer2 *exec) +static int i915_gem_check_execbuffer(struct drm_i915_private *i915, + struct drm_i915_gem_execbuffer2 *exec) { if (exec->flags & __I915_EXEC_ILLEGAL_FLAGS) return -EINVAL; @@ -2161,7 +2164,7 @@ static int i915_gem_check_execbuffer(struct drm_i915_gem_execbuffer2 *exec) } if (exec->DR4 == 0xffffffff) { - DRM_DEBUG("UXA submitting garbage DR4, fixing up\n"); + drm_dbg(&i915->drm, "UXA submitting garbage DR4, fixing up\n"); exec->DR4 = 0; } if (exec->DR1 || exec->DR4) @@ -2799,7 +2802,8 @@ add_timeline_fence_array(struct i915_execbuffer *eb, syncobj = drm_syncobj_find(eb->file, user_fence.handle); if (!syncobj) { - DRM_DEBUG("Invalid syncobj handle provided\n"); + drm_dbg(&eb->i915->drm, + "Invalid syncobj handle provided\n"); return -ENOENT; } @@ -2807,7 +2811,8 @@ add_timeline_fence_array(struct i915_execbuffer *eb, if (!fence && user_fence.flags && !(user_fence.flags & I915_EXEC_FENCE_SIGNAL)) { - DRM_DEBUG("Syncobj handle has no fence\n"); + drm_dbg(&eb->i915->drm, + "Syncobj handle has no fence\n"); drm_syncobj_put(syncobj); return -EINVAL; } @@ -2816,7 +2821,9 @@ add_timeline_fence_array(struct i915_execbuffer *eb, err = dma_fence_chain_find_seqno(&fence, point); if (err && !(user_fence.flags & I915_EXEC_FENCE_SIGNAL)) { - DRM_DEBUG("Syncobj handle missing requested point %llu\n", point); + drm_dbg(&eb->i915->drm, + "Syncobj handle missing requested point %llu\n", + point); dma_fence_put(fence); drm_syncobj_put(syncobj); return err; @@ -2842,7 +2849,8 @@ add_timeline_fence_array(struct i915_execbuffer *eb, * 0) would break the timeline. */ if (user_fence.flags & I915_EXEC_FENCE_WAIT) { - DRM_DEBUG("Trying to wait & signal the same timeline point.\n"); + drm_dbg(&eb->i915->drm, + "Trying to wait & signal the same timeline point.\n"); dma_fence_put(fence); drm_syncobj_put(syncobj); return -EINVAL; @@ -2913,14 +2921,16 @@ static int add_fence_array(struct i915_execbuffer *eb) syncobj = drm_syncobj_find(eb->file, user_fence.handle); if (!syncobj) { - DRM_DEBUG("Invalid syncobj handle provided\n"); + drm_dbg(&eb->i915->drm, + "Invalid syncobj handle provided\n"); return -ENOENT; } if (user_fence.flags & I915_EXEC_FENCE_WAIT) { fence = drm_syncobj_fence_get(syncobj); if (!fence) { - DRM_DEBUG("Syncobj handle has no fence\n"); + drm_dbg(&eb->i915->drm, + "Syncobj handle has no fence\n"); drm_syncobj_put(syncobj); return -EINVAL; } @@ -2954,11 +2964,6 @@ await_fence_array(struct i915_execbuffer *eb, int err; for (n = 0; n < eb->num_fences; n++) { - struct drm_syncobj *syncobj; - unsigned int flags; - - syncobj = ptr_unpack_bits(eb->fences[n].syncobj, &flags, 2); - if (!eb->fences[n].dma_fence) continue; @@ -3520,7 +3525,7 @@ i915_gem_execbuffer2_ioctl(struct drm_device *dev, void *data, return -EINVAL; } - err = i915_gem_check_execbuffer(args); + err = i915_gem_check_execbuffer(i915, args); if (err) return err; diff --git a/drivers/gpu/drm/i915/gem/i915_gem_internal.c b/drivers/gpu/drm/i915/gem/i915_gem_internal.c index 629acb403a2c..f66bcefc09ec 100644 --- a/drivers/gpu/drm/i915/gem/i915_gem_internal.c +++ b/drivers/gpu/drm/i915/gem/i915_gem_internal.c @@ -35,7 +35,6 @@ static int i915_gem_object_get_pages_internal(struct drm_i915_gem_object *obj) struct drm_i915_private *i915 = to_i915(obj->base.dev); struct sg_table *st; struct scatterlist *sg; - unsigned int sg_page_sizes; unsigned int npages; int max_order = MAX_ORDER; unsigned int max_segment; @@ -64,7 +63,6 @@ create_st: sg = st->sgl; st->nents = 0; - sg_page_sizes = 0; do { int order = min(fls(npages) - 1, max_order); @@ -83,7 +81,6 @@ create_st: } while (1); sg_set_page(sg, page, PAGE_SIZE << order, 0); - sg_page_sizes |= PAGE_SIZE << order; st->nents++; npages -= 1 << order; @@ -105,7 +102,7 @@ create_st: goto err; } - __i915_gem_object_set_pages(obj, st, sg_page_sizes); + __i915_gem_object_set_pages(obj, st); return 0; diff --git a/drivers/gpu/drm/i915/gem/i915_gem_mman.c b/drivers/gpu/drm/i915/gem/i915_gem_mman.c index 73d9eda1d6b7..c29efdef8313 100644 --- a/drivers/gpu/drm/i915/gem/i915_gem_mman.c +++ b/drivers/gpu/drm/i915/gem/i915_gem_mman.c @@ -330,7 +330,7 @@ retry: if (ret) goto err_rpm; - ret = intel_gt_reset_trylock(ggtt->vm.gt, &srcu); + ret = intel_gt_reset_lock_interruptible(ggtt->vm.gt, &srcu); if (ret) goto err_pages; @@ -413,7 +413,7 @@ retry: vma->mmo = mmo; if (CONFIG_DRM_I915_USERFAULT_AUTOSUSPEND) - intel_wakeref_auto(&to_gt(i915)->userfault_wakeref, + intel_wakeref_auto(&i915->runtime_pm.userfault_wakeref, msecs_to_jiffies_timeout(CONFIG_DRM_I915_USERFAULT_AUTOSUSPEND)); if (write) { @@ -557,11 +557,13 @@ void i915_gem_object_runtime_pm_release_mmap_offset(struct drm_i915_gem_object * drm_vma_node_unmap(&bo->base.vma_node, bdev->dev_mapping); - if (obj->userfault_count) { - /* rpm wakeref provide exclusive access */ - list_del(&obj->userfault_link); - obj->userfault_count = 0; - } + /* + * We have exclusive access here via runtime suspend. All other callers + * must first grab the rpm wakeref. + */ + GEM_BUG_ON(!obj->userfault_count); + list_del(&obj->userfault_link); + obj->userfault_count = 0; } void i915_gem_object_release_mmap_offset(struct drm_i915_gem_object *obj) @@ -587,13 +589,6 @@ void i915_gem_object_release_mmap_offset(struct drm_i915_gem_object *obj) spin_lock(&obj->mmo.lock); } spin_unlock(&obj->mmo.lock); - - if (obj->userfault_count) { - mutex_lock(&to_gt(to_i915(obj->base.dev))->lmem_userfault_lock); - list_del(&obj->userfault_link); - mutex_unlock(&to_gt(to_i915(obj->base.dev))->lmem_userfault_lock); - obj->userfault_count = 0; - } } static struct i915_mmap_offset * diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object.c b/drivers/gpu/drm/i915/gem/i915_gem_object.c index 369006c5317f..733696057761 100644 --- a/drivers/gpu/drm/i915/gem/i915_gem_object.c +++ b/drivers/gpu/drm/i915/gem/i915_gem_object.c @@ -290,7 +290,21 @@ void __i915_gem_object_pages_fini(struct drm_i915_gem_object *obj) __i915_gem_object_free_mmaps(obj); atomic_set(&obj->mm.pages_pin_count, 0); + + /* + * dma_buf_unmap_attachment() requires reservation to be + * locked. The imported GEM shouldn't share reservation lock + * and ttm_bo_cleanup_memtype_use() shouldn't be invoked for + * dma-buf, so it's safe to take the lock. + */ + if (obj->base.import_attach) + i915_gem_object_lock(obj, NULL); + __i915_gem_object_put_pages(obj); + + if (obj->base.import_attach) + i915_gem_object_unlock(obj); + GEM_BUG_ON(i915_gem_object_has_pages(obj)); } @@ -444,6 +458,16 @@ i915_gem_object_read_from_page_iomap(struct drm_i915_gem_object *obj, u64 offset io_mapping_unmap(src_map); } +static bool object_has_mappable_iomem(struct drm_i915_gem_object *obj) +{ + GEM_BUG_ON(!i915_gem_object_has_iomem(obj)); + + if (IS_DGFX(to_i915(obj->base.dev))) + return i915_ttm_resource_mappable(i915_gem_to_ttm(obj)->resource); + + return true; +} + /** * i915_gem_object_read_from_page - read data from the page of a GEM object * @obj: GEM object to read from @@ -466,7 +490,7 @@ int i915_gem_object_read_from_page(struct drm_i915_gem_object *obj, u64 offset, if (i915_gem_object_has_struct_page(obj)) i915_gem_object_read_from_page_kmap(obj, offset, dst, size); - else if (i915_gem_object_has_iomem(obj)) + else if (i915_gem_object_has_iomem(obj) && object_has_mappable_iomem(obj)) i915_gem_object_read_from_page_iomap(obj, offset, dst, size); else return -ENODEV; diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object.h b/drivers/gpu/drm/i915/gem/i915_gem_object.h index 1723af9b0f6a..3db53769864c 100644 --- a/drivers/gpu/drm/i915/gem/i915_gem_object.h +++ b/drivers/gpu/drm/i915/gem/i915_gem_object.h @@ -403,8 +403,7 @@ i915_gem_object_get_dma_address(struct drm_i915_gem_object *obj, unsigned long n); void __i915_gem_object_set_pages(struct drm_i915_gem_object *obj, - struct sg_table *pages, - unsigned int sg_page_sizes); + struct sg_table *pages); int ____i915_gem_object_get_pages(struct drm_i915_gem_object *obj); int __i915_gem_object_get_pages(struct drm_i915_gem_object *obj); @@ -482,6 +481,10 @@ void *__must_check i915_gem_object_pin_map(struct drm_i915_gem_object *obj, void *__must_check i915_gem_object_pin_map_unlocked(struct drm_i915_gem_object *obj, enum i915_map_type type); +enum i915_map_type i915_coherent_map_type(struct drm_i915_private *i915, + struct drm_i915_gem_object *obj, + bool always_coherent); + void __i915_gem_object_flush_map(struct drm_i915_gem_object *obj, unsigned long offset, unsigned long size); diff --git a/drivers/gpu/drm/i915/gem/i915_gem_pages.c b/drivers/gpu/drm/i915/gem/i915_gem_pages.c index 4df50b049cea..05a27723ebb8 100644 --- a/drivers/gpu/drm/i915/gem/i915_gem_pages.c +++ b/drivers/gpu/drm/i915/gem/i915_gem_pages.c @@ -16,8 +16,7 @@ #include "i915_gem_mman.h" void __i915_gem_object_set_pages(struct drm_i915_gem_object *obj, - struct sg_table *pages, - unsigned int sg_page_sizes) + struct sg_table *pages) { struct drm_i915_private *i915 = to_i915(obj->base.dev); unsigned long supported = RUNTIME_INFO(i915)->page_sizes; @@ -45,8 +44,8 @@ void __i915_gem_object_set_pages(struct drm_i915_gem_object *obj, obj->mm.pages = pages; - GEM_BUG_ON(!sg_page_sizes); - obj->mm.page_sizes.phys = sg_page_sizes; + obj->mm.page_sizes.phys = i915_sg_dma_sizes(pages->sgl); + GEM_BUG_ON(!obj->mm.page_sizes.phys); /* * Calculate the supported page-sizes which fit into the given @@ -466,6 +465,18 @@ void *i915_gem_object_pin_map_unlocked(struct drm_i915_gem_object *obj, return ret; } +enum i915_map_type i915_coherent_map_type(struct drm_i915_private *i915, + struct drm_i915_gem_object *obj, + bool always_coherent) +{ + if (i915_gem_object_is_lmem(obj)) + return I915_MAP_WC; + if (HAS_LLC(i915) || always_coherent) + return I915_MAP_WB; + else + return I915_MAP_WC; +} + void __i915_gem_object_flush_map(struct drm_i915_gem_object *obj, unsigned long offset, unsigned long size) diff --git a/drivers/gpu/drm/i915/gem/i915_gem_phys.c b/drivers/gpu/drm/i915/gem/i915_gem_phys.c index 0d0e46dae559..68453572275b 100644 --- a/drivers/gpu/drm/i915/gem/i915_gem_phys.c +++ b/drivers/gpu/drm/i915/gem/i915_gem_phys.c @@ -79,7 +79,7 @@ static int i915_gem_object_get_pages_phys(struct drm_i915_gem_object *obj) /* We're no longer struct page backed */ obj->mem_flags &= ~I915_BO_FLAG_STRUCT_PAGE; - __i915_gem_object_set_pages(obj, st, sg->length); + __i915_gem_object_set_pages(obj, st); return 0; @@ -209,11 +209,8 @@ static int i915_gem_object_shmem_to_phys(struct drm_i915_gem_object *obj) return 0; err_xfer: - if (!IS_ERR_OR_NULL(pages)) { - unsigned int sg_page_sizes = i915_sg_dma_sizes(pages->sgl); - - __i915_gem_object_set_pages(obj, pages, sg_page_sizes); - } + if (!IS_ERR_OR_NULL(pages)) + __i915_gem_object_set_pages(obj, pages); return err; } diff --git a/drivers/gpu/drm/i915/gem/i915_gem_pm.c b/drivers/gpu/drm/i915/gem/i915_gem_pm.c index 3428f735e786..0d812f4d787d 100644 --- a/drivers/gpu/drm/i915/gem/i915_gem_pm.c +++ b/drivers/gpu/drm/i915/gem/i915_gem_pm.c @@ -22,9 +22,12 @@ void i915_gem_suspend(struct drm_i915_private *i915) { + struct intel_gt *gt; + unsigned int i; + GEM_TRACE("%s\n", dev_name(i915->drm.dev)); - intel_wakeref_auto(&to_gt(i915)->userfault_wakeref, 0); + intel_wakeref_auto(&i915->runtime_pm.userfault_wakeref, 0); flush_workqueue(i915->wq); /* @@ -36,7 +39,8 @@ void i915_gem_suspend(struct drm_i915_private *i915) * state. Fortunately, the kernel_context is disposable and we do * not rely on its state. */ - intel_gt_suspend_prepare(to_gt(i915)); + for_each_gt(gt, i915, i) + intel_gt_suspend_prepare(gt); i915_gem_drain_freed_objects(i915); } @@ -131,7 +135,9 @@ void i915_gem_suspend_late(struct drm_i915_private *i915) &i915->mm.purge_list, NULL }, **phase; + struct intel_gt *gt; unsigned long flags; + unsigned int i; bool flush = false; /* @@ -154,7 +160,8 @@ void i915_gem_suspend_late(struct drm_i915_private *i915) * machine in an unusable condition. */ - intel_gt_suspend_late(to_gt(i915)); + for_each_gt(gt, i915, i) + intel_gt_suspend_late(gt); spin_lock_irqsave(&i915->mm.obj_lock, flags); for (phase = phases; *phase; phase++) { @@ -212,7 +219,8 @@ int i915_gem_freeze_late(struct drm_i915_private *i915) void i915_gem_resume(struct drm_i915_private *i915) { - int ret; + struct intel_gt *gt; + int ret, i, j; GEM_TRACE("%s\n", dev_name(i915->drm.dev)); @@ -224,8 +232,25 @@ void i915_gem_resume(struct drm_i915_private *i915) * guarantee that the context image is complete. So let's just reset * it and start again. */ - intel_gt_resume(to_gt(i915)); + for_each_gt(gt, i915, i) + if (intel_gt_resume(gt)) + goto err_wedged; ret = lmem_restore(i915, I915_TTM_BACKUP_ALLOW_GPU); GEM_WARN_ON(ret); + + return; + +err_wedged: + for_each_gt(gt, i915, j) { + if (!intel_gt_is_wedged(gt)) { + dev_err(i915->drm.dev, + "Failed to re-initialize GPU[%u], declaring it wedged!\n", + j); + intel_gt_set_wedged(gt); + } + + if (j == i) + break; + } } diff --git a/drivers/gpu/drm/i915/gem/i915_gem_shmem.c b/drivers/gpu/drm/i915/gem/i915_gem_shmem.c index 2f7804492cd5..9c759df700ca 100644 --- a/drivers/gpu/drm/i915/gem/i915_gem_shmem.c +++ b/drivers/gpu/drm/i915/gem/i915_gem_shmem.c @@ -247,7 +247,7 @@ rebuild_st: if (i915_gem_object_can_bypass_llc(obj)) obj->cache_dirty = true; - __i915_gem_object_set_pages(obj, st, i915_sg_dma_sizes(st->sgl)); + __i915_gem_object_set_pages(obj, st); return 0; diff --git a/drivers/gpu/drm/i915/gem/i915_gem_stolen.c b/drivers/gpu/drm/i915/gem/i915_gem_stolen.c index acc561c0f0aa..bc9521078807 100644 --- a/drivers/gpu/drm/i915/gem/i915_gem_stolen.c +++ b/drivers/gpu/drm/i915/gem/i915_gem_stolen.c @@ -77,22 +77,26 @@ void i915_gem_stolen_remove_node(struct drm_i915_private *i915, mutex_unlock(&i915->mm.stolen_lock); } -static int i915_adjust_stolen(struct drm_i915_private *i915, - struct resource *dsm) +static bool valid_stolen_size(struct drm_i915_private *i915, struct resource *dsm) +{ + return (dsm->start != 0 || HAS_LMEMBAR_SMEM_STOLEN(i915)) && dsm->end > dsm->start; +} + +static int adjust_stolen(struct drm_i915_private *i915, + struct resource *dsm) { struct i915_ggtt *ggtt = to_gt(i915)->ggtt; struct intel_uncore *uncore = ggtt->vm.gt->uncore; - struct resource *r; - if (dsm->start == 0 || dsm->end <= dsm->start) + if (!valid_stolen_size(i915, dsm)) return -EINVAL; /* + * Make sure we don't clobber the GTT if it's within stolen memory + * * TODO: We have yet too encounter the case where the GTT wasn't at the * end of stolen. With that assumption we could simplify this. */ - - /* Make sure we don't clobber the GTT if it's within stolen memory */ if (GRAPHICS_VER(i915) <= 4 && !IS_G33(i915) && !IS_PINEVIEW(i915) && !IS_G4X(i915)) { struct resource stolen[2] = {*dsm, *dsm}; @@ -131,12 +135,25 @@ static int i915_adjust_stolen(struct drm_i915_private *i915, } } + if (!valid_stolen_size(i915, dsm)) + return -EINVAL; + + return 0; +} + +static int request_smem_stolen(struct drm_i915_private *i915, + struct resource *dsm) +{ + struct resource *r; + /* - * With stolen lmem, we don't need to check if the address range - * overlaps with the non-stolen system memory range, since lmem is local - * to the gpu. + * With stolen lmem, we don't need to request system memory for the + * address range since it's local to the gpu. + * + * Starting MTL, in IGFX devices the stolen memory is exposed via + * LMEMBAR and shall be considered similar to stolen lmem. */ - if (HAS_LMEM(i915)) + if (HAS_LMEM(i915) || HAS_LMEMBAR_SMEM_STOLEN(i915)) return 0; /* @@ -371,8 +388,6 @@ static void icl_get_stolen_reserved(struct drm_i915_private *i915, drm_dbg(&i915->drm, "GEN6_STOLEN_RESERVED = 0x%016llx\n", reg_val); - *base = reg_val & GEN11_STOLEN_RESERVED_ADDR_MASK; - switch (reg_val & GEN8_STOLEN_RESERVED_SIZE_MASK) { case GEN8_STOLEN_RESERVED_1M: *size = 1024 * 1024; @@ -390,41 +405,30 @@ static void icl_get_stolen_reserved(struct drm_i915_private *i915, *size = 8 * 1024 * 1024; MISSING_CASE(reg_val & GEN8_STOLEN_RESERVED_SIZE_MASK); } + + if (HAS_LMEMBAR_SMEM_STOLEN(i915)) + /* the base is initialized to stolen top so subtract size to get base */ + *base -= *size; + else + *base = reg_val & GEN11_STOLEN_RESERVED_ADDR_MASK; } -static int i915_gem_init_stolen(struct intel_memory_region *mem) +/* + * Initialize i915->dsm_reserved to contain the reserved space within the Data + * Stolen Memory. This is a range on the top of DSM that is reserved, not to + * be used by driver, so must be excluded from the region passed to the + * allocator later. In the spec this is also called as WOPCM. + * + * Our expectation is that the reserved space is at the top of the stolen + * region, as it has been the case for every platform, and *never* at the + * bottom, so the calculation here can be simplified. + */ +static int init_reserved_stolen(struct drm_i915_private *i915) { - struct drm_i915_private *i915 = mem->i915; struct intel_uncore *uncore = &i915->uncore; resource_size_t reserved_base, stolen_top; - resource_size_t reserved_total, reserved_size; - - mutex_init(&i915->mm.stolen_lock); - - if (intel_vgpu_active(i915)) { - drm_notice(&i915->drm, - "%s, disabling use of stolen memory\n", - "iGVT-g active"); - return 0; - } - - if (i915_vtd_active(i915) && GRAPHICS_VER(i915) < 8) { - drm_notice(&i915->drm, - "%s, disabling use of stolen memory\n", - "DMAR active"); - return 0; - } - - if (resource_size(&mem->region) == 0) - return 0; - - i915->dsm = mem->region; - - if (i915_adjust_stolen(i915, &i915->dsm)) - return 0; - - GEM_BUG_ON(i915->dsm.start == 0); - GEM_BUG_ON(i915->dsm.end <= i915->dsm.start); + resource_size_t reserved_size; + int ret = 0; stolen_top = i915->dsm.end + 1; reserved_base = stolen_top; @@ -455,17 +459,16 @@ static int i915_gem_init_stolen(struct intel_memory_region *mem) &reserved_base, &reserved_size); } - /* - * Our expectation is that the reserved space is at the top of the - * stolen region and *never* at the bottom. If we see !reserved_base, - * it likely means we failed to read the registers correctly. - */ + /* No reserved stolen */ + if (reserved_base == stolen_top) + goto bail_out; + if (!reserved_base) { drm_err(&i915->drm, "inconsistent reservation %pa + %pa; ignoring\n", &reserved_base, &reserved_size); - reserved_base = stolen_top; - reserved_size = 0; + ret = -EINVAL; + goto bail_out; } i915->dsm_reserved = @@ -475,19 +478,55 @@ static int i915_gem_init_stolen(struct intel_memory_region *mem) drm_err(&i915->drm, "Stolen reserved area %pR outside stolen memory %pR\n", &i915->dsm_reserved, &i915->dsm); - return 0; + ret = -EINVAL; + goto bail_out; } + return 0; + +bail_out: + i915->dsm_reserved = + (struct resource)DEFINE_RES_MEM(reserved_base, 0); + + return ret; +} + +static int i915_gem_init_stolen(struct intel_memory_region *mem) +{ + struct drm_i915_private *i915 = mem->i915; + + mutex_init(&i915->mm.stolen_lock); + + if (intel_vgpu_active(i915)) { + drm_notice(&i915->drm, + "%s, disabling use of stolen memory\n", + "iGVT-g active"); + return -ENOSPC; + } + + if (i915_vtd_active(i915) && GRAPHICS_VER(i915) < 8) { + drm_notice(&i915->drm, + "%s, disabling use of stolen memory\n", + "DMAR active"); + return -ENOSPC; + } + + if (adjust_stolen(i915, &mem->region)) + return -ENOSPC; + + if (request_smem_stolen(i915, &mem->region)) + return -ENOSPC; + + i915->dsm = mem->region; + + if (init_reserved_stolen(i915)) + return -ENOSPC; + /* Exclude the reserved region from driver use */ - mem->region.end = reserved_base - 1; + mem->region.end = i915->dsm_reserved.start - 1; mem->io_size = min(mem->io_size, resource_size(&mem->region)); - /* It is possible for the reserved area to end before the end of stolen - * memory, so just consider the start. */ - reserved_total = stolen_top - reserved_base; - - i915->stolen_usable_size = - resource_size(&i915->dsm) - reserved_total; + i915->stolen_usable_size = resource_size(&mem->region); drm_dbg(&i915->drm, "Memory reserved for graphics device: %lluK, usable: %lluK\n", @@ -495,7 +534,7 @@ static int i915_gem_init_stolen(struct intel_memory_region *mem) (u64)i915->stolen_usable_size >> 10); if (i915->stolen_usable_size == 0) - return 0; + return -ENOSPC; /* Basic memrange allocator for stolen space. */ drm_mm_init(&i915->mm.stolen, 0, i915->stolen_usable_size); @@ -589,7 +628,7 @@ static int i915_gem_object_get_pages_stolen(struct drm_i915_gem_object *obj) sg_dma_len(pages->sgl), POISON_INUSE); - __i915_gem_object_set_pages(obj, pages, obj->stolen->size); + __i915_gem_object_set_pages(obj, pages); return 0; } @@ -733,11 +772,17 @@ i915_gem_object_create_stolen(struct drm_i915_private *i915, static int init_stolen_smem(struct intel_memory_region *mem) { + int err; + /* * Initialise stolen early so that we may reserve preallocated * objects for the BIOS to KMS transition. */ - return i915_gem_init_stolen(mem); + err = i915_gem_init_stolen(mem); + if (err) + drm_dbg(&mem->i915->drm, "Skip stolen region: failed to setup\n"); + + return 0; } static int release_stolen_smem(struct intel_memory_region *mem) @@ -754,26 +799,25 @@ static const struct intel_memory_region_ops i915_region_stolen_smem_ops = { static int init_stolen_lmem(struct intel_memory_region *mem) { + struct drm_i915_private *i915 = mem->i915; int err; if (GEM_WARN_ON(resource_size(&mem->region) == 0)) - return -ENODEV; + return 0; - /* - * TODO: For stolen lmem we mostly just care about populating the dsm - * related bits and setting up the drm_mm allocator for the range. - * Perhaps split up i915_gem_init_stolen() for this. - */ err = i915_gem_init_stolen(mem); - if (err) - return err; + if (err) { + drm_dbg(&mem->i915->drm, "Skip stolen region: failed to setup\n"); + return 0; + } - if (mem->io_size && !io_mapping_init_wc(&mem->iomap, - mem->io_start, - mem->io_size)) { - err = -EIO; + if (mem->io_size && + !io_mapping_init_wc(&mem->iomap, mem->io_start, mem->io_size)) goto err_cleanup; - } + + drm_dbg(&i915->drm, "Stolen Local memory IO start: %pa\n", + &mem->io_start); + drm_dbg(&i915->drm, "Stolen Local DSM base: %pa\n", &mem->region.start); return 0; @@ -796,6 +840,29 @@ static const struct intel_memory_region_ops i915_region_stolen_lmem_ops = { .init_object = _i915_gem_object_stolen_init, }; +static int mtl_get_gms_size(struct intel_uncore *uncore) +{ + u16 ggc, gms; + + ggc = intel_uncore_read16(uncore, GGC); + + /* check GGMS, should be fixed 0x3 (8MB) */ + if ((ggc & GGMS_MASK) != GGMS_MASK) + return -EIO; + + /* return valid GMS value, -EIO if invalid */ + gms = REG_FIELD_GET(GMS_MASK, ggc); + switch (gms) { + case 0x0 ... 0x04: + return gms * 32; + case 0xf0 ... 0xfe: + return (gms - 0xf0 + 1) * 4; + default: + MISSING_CASE(gms); + return -EIO; + } +} + struct intel_memory_region * i915_gem_stolen_lmem_setup(struct drm_i915_private *i915, u16 type, u16 instance) @@ -806,6 +873,7 @@ i915_gem_stolen_lmem_setup(struct drm_i915_private *i915, u16 type, struct intel_memory_region *mem; resource_size_t io_start, io_size; resource_size_t min_page_size; + int ret; if (WARN_ON_ONCE(instance)) return ERR_PTR(-ENODEV); @@ -813,12 +881,8 @@ i915_gem_stolen_lmem_setup(struct drm_i915_private *i915, u16 type, if (!i915_pci_resource_valid(pdev, GEN12_LMEM_BAR)) return ERR_PTR(-ENXIO); - /* Use DSM base address instead for stolen memory */ - dsm_base = intel_uncore_read64(uncore, GEN12_DSMBASE); - if (IS_DG1(uncore->i915)) { + if (HAS_LMEMBAR_SMEM_STOLEN(i915) || IS_DG1(i915)) { lmem_size = pci_resource_len(pdev, GEN12_LMEM_BAR); - if (WARN_ON(lmem_size < dsm_base)) - return ERR_PTR(-ENODEV); } else { resource_size_t lmem_range; @@ -827,13 +891,39 @@ i915_gem_stolen_lmem_setup(struct drm_i915_private *i915, u16 type, lmem_size *= SZ_1G; } - dsm_size = lmem_size - dsm_base; - if (pci_resource_len(pdev, GEN12_LMEM_BAR) < lmem_size) { + if (HAS_LMEMBAR_SMEM_STOLEN(i915)) { + /* + * MTL dsm size is in GGC register. + * Also MTL uses offset to DSMBASE in ptes, so i915 + * uses dsm_base = 0 to setup stolen region. + */ + ret = mtl_get_gms_size(uncore); + if (ret < 0) { + drm_err(&i915->drm, "invalid MTL GGC register setting\n"); + return ERR_PTR(ret); + } + + dsm_base = 0; + dsm_size = (resource_size_t)(ret * SZ_1M); + + GEM_BUG_ON(pci_resource_len(pdev, GEN12_LMEM_BAR) != SZ_256M); + GEM_BUG_ON((dsm_size + SZ_8M) > lmem_size); + } else { + /* Use DSM base address instead for stolen memory */ + dsm_base = intel_uncore_read64(uncore, GEN12_DSMBASE) & GEN12_BDSM_MASK; + if (WARN_ON(lmem_size < dsm_base)) + return ERR_PTR(-ENODEV); + dsm_size = lmem_size - dsm_base; + } + + io_size = dsm_size; + if (HAS_LMEMBAR_SMEM_STOLEN(i915)) { + io_start = pci_resource_start(pdev, GEN12_LMEM_BAR) + SZ_8M; + } else if (pci_resource_len(pdev, GEN12_LMEM_BAR) < lmem_size) { io_start = 0; io_size = 0; } else { io_start = pci_resource_start(pdev, GEN12_LMEM_BAR) + dsm_base; - io_size = dsm_size; } min_page_size = HAS_64K_PAGES(i915) ? I915_GTT_PAGE_SIZE_64K : @@ -847,16 +937,6 @@ i915_gem_stolen_lmem_setup(struct drm_i915_private *i915, u16 type, if (IS_ERR(mem)) return mem; - /* - * TODO: consider creating common helper to just print all the - * interesting stuff from intel_memory_region, which we can use for all - * our probed regions. - */ - - drm_dbg(&i915->drm, "Stolen Local memory IO start: %pa\n", - &mem->io_start); - drm_dbg(&i915->drm, "Stolen Local DSM base: %pa\n", &dsm_base); - intel_memory_region_set_name(mem, "stolen-local"); mem->private = true; @@ -881,6 +961,7 @@ i915_gem_stolen_smem_setup(struct drm_i915_private *i915, u16 type, intel_memory_region_set_name(mem, "stolen-system"); mem->private = true; + return mem; } diff --git a/drivers/gpu/drm/i915/gem/i915_gem_ttm.c b/drivers/gpu/drm/i915/gem/i915_gem_ttm.c index 0d6d640225fc..1e50fb0d6bfc 100644 --- a/drivers/gpu/drm/i915/gem/i915_gem_ttm.c +++ b/drivers/gpu/drm/i915/gem/i915_gem_ttm.c @@ -279,7 +279,7 @@ static struct ttm_tt *i915_ttm_tt_create(struct ttm_buffer_object *bo, struct i915_ttm_tt *i915_tt; int ret; - if (!obj) + if (i915_ttm_is_ghost_object(bo)) return NULL; i915_tt = kzalloc(sizeof(*i915_tt), GFP_KERNEL); @@ -362,7 +362,7 @@ static bool i915_ttm_eviction_valuable(struct ttm_buffer_object *bo, { struct drm_i915_gem_object *obj = i915_ttm_to_gem(bo); - if (!obj) + if (i915_ttm_is_ghost_object(bo)) return false; /* @@ -509,18 +509,9 @@ static int i915_ttm_shrink(struct drm_i915_gem_object *obj, unsigned int flags) static void i915_ttm_delete_mem_notify(struct ttm_buffer_object *bo) { struct drm_i915_gem_object *obj = i915_ttm_to_gem(bo); - intel_wakeref_t wakeref = 0; - - if (bo->resource && likely(obj)) { - /* ttm_bo_release() already has dma_resv_lock */ - if (i915_ttm_cpu_maps_iomem(bo->resource)) - wakeref = intel_runtime_pm_get(&to_i915(obj->base.dev)->runtime_pm); + if (bo->resource && !i915_ttm_is_ghost_object(bo)) { __i915_gem_object_pages_fini(obj); - - if (wakeref) - intel_runtime_pm_put(&to_i915(obj->base.dev)->runtime_pm, wakeref); - i915_ttm_free_cached_io_rsgt(obj); } } @@ -628,7 +619,7 @@ static void i915_ttm_swap_notify(struct ttm_buffer_object *bo) struct drm_i915_gem_object *obj = i915_ttm_to_gem(bo); int ret; - if (!obj) + if (i915_ttm_is_ghost_object(bo)) return; ret = i915_ttm_move_notify(bo); @@ -653,7 +644,7 @@ bool i915_ttm_resource_mappable(struct ttm_resource *res) if (!i915_ttm_cpu_maps_iomem(res)) return true; - return bman_res->used_visible_size == bman_res->base.num_pages; + return bman_res->used_visible_size == PFN_UP(bman_res->base.size); } static int i915_ttm_io_mem_reserve(struct ttm_device *bdev, struct ttm_resource *mem) @@ -661,7 +652,7 @@ static int i915_ttm_io_mem_reserve(struct ttm_device *bdev, struct ttm_resource struct drm_i915_gem_object *obj = i915_ttm_to_gem(mem->bo); bool unknown_state; - if (!obj) + if (i915_ttm_is_ghost_object(mem->bo)) return -EINVAL; if (!kref_get_unless_zero(&obj->base.refcount)) @@ -694,7 +685,7 @@ static unsigned long i915_ttm_io_mem_pfn(struct ttm_buffer_object *bo, unsigned long base; unsigned int ofs; - GEM_BUG_ON(!obj); + GEM_BUG_ON(i915_ttm_is_ghost_object(bo)); GEM_WARN_ON(bo->ttm); base = obj->mm.region->iomap.base - obj->mm.region->region.start; @@ -703,6 +694,50 @@ static unsigned long i915_ttm_io_mem_pfn(struct ttm_buffer_object *bo, return ((base + sg_dma_address(sg)) >> PAGE_SHIFT) + ofs; } +static int i915_ttm_access_memory(struct ttm_buffer_object *bo, + unsigned long offset, void *buf, + int len, int write) +{ + struct drm_i915_gem_object *obj = i915_ttm_to_gem(bo); + resource_size_t iomap = obj->mm.region->iomap.base - + obj->mm.region->region.start; + unsigned long page = offset >> PAGE_SHIFT; + unsigned long bytes_left = len; + + /* + * TODO: For now just let it fail if the resource is non-mappable, + * otherwise we need to perform the memcpy from the gpu here, without + * interfering with the object (like moving the entire thing). + */ + if (!i915_ttm_resource_mappable(bo->resource)) + return -EIO; + + offset -= page << PAGE_SHIFT; + do { + unsigned long bytes = min(bytes_left, PAGE_SIZE - offset); + void __iomem *ptr; + dma_addr_t daddr; + + daddr = i915_gem_object_get_dma_address(obj, page); + ptr = ioremap_wc(iomap + daddr + offset, bytes); + if (!ptr) + return -EIO; + + if (write) + memcpy_toio(ptr, buf, bytes); + else + memcpy_fromio(buf, ptr, bytes); + iounmap(ptr); + + page++; + buf += bytes; + bytes_left -= bytes; + offset = 0; + } while (bytes_left); + + return len; +} + /* * All callbacks need to take care not to downcast a struct ttm_buffer_object * without checking its subclass, since it might be a TTM ghost object. @@ -719,6 +754,7 @@ static struct ttm_device_funcs i915_ttm_bo_driver = { .delete_mem_notify = i915_ttm_delete_mem_notify, .io_mem_reserve = i915_ttm_io_mem_reserve, .io_mem_pfn = i915_ttm_io_mem_pfn, + .access_memory = i915_ttm_access_memory, }; /** @@ -783,8 +819,7 @@ static int __i915_ttm_get_pages(struct drm_i915_gem_object *obj, GEM_BUG_ON(obj->mm.rsgt); obj->mm.rsgt = rsgt; - __i915_gem_object_set_pages(obj, &rsgt->table, - i915_sg_dma_sizes(rsgt->table.sgl)); + __i915_gem_object_set_pages(obj, &rsgt->table); } GEM_BUG_ON(bo->ttm && ((obj->base.size >> PAGE_SHIFT) < bo->ttm->num_pages)); @@ -994,15 +1029,11 @@ static vm_fault_t vm_fault_ttm(struct vm_fault *vmf) struct vm_area_struct *area = vmf->vma; struct ttm_buffer_object *bo = area->vm_private_data; struct drm_device *dev = bo->base.dev; - struct drm_i915_gem_object *obj; + struct drm_i915_gem_object *obj = i915_ttm_to_gem(bo); intel_wakeref_t wakeref = 0; vm_fault_t ret; int idx; - obj = i915_ttm_to_gem(bo); - if (!obj) - return VM_FAULT_SIGBUS; - /* Sanity check that we allow writing into this object */ if (unlikely(i915_gem_object_is_readonly(obj) && area->vm_flags & VM_WRITE)) @@ -1036,7 +1067,8 @@ static vm_fault_t vm_fault_ttm(struct vm_fault *vmf) } if (err) { - drm_dbg(dev, "Unable to make resource CPU accessible\n"); + drm_dbg(dev, "Unable to make resource CPU accessible(err = %pe)\n", + ERR_PTR(err)); dma_resv_unlock(bo->base.resv); ret = VM_FAULT_SIGBUS; goto out_rpm; @@ -1057,16 +1089,21 @@ static vm_fault_t vm_fault_ttm(struct vm_fault *vmf) if (ret == VM_FAULT_RETRY && !(vmf->flags & FAULT_FLAG_RETRY_NOWAIT)) goto out_rpm; - /* ttm_bo_vm_reserve() already has dma_resv_lock */ + /* + * ttm_bo_vm_reserve() already has dma_resv_lock. + * userfault_count is protected by dma_resv lock and rpm wakeref. + */ if (ret == VM_FAULT_NOPAGE && wakeref && !obj->userfault_count) { obj->userfault_count = 1; - mutex_lock(&to_gt(to_i915(obj->base.dev))->lmem_userfault_lock); - list_add(&obj->userfault_link, &to_gt(to_i915(obj->base.dev))->lmem_userfault_list); - mutex_unlock(&to_gt(to_i915(obj->base.dev))->lmem_userfault_lock); + spin_lock(&to_i915(obj->base.dev)->runtime_pm.lmem_userfault_lock); + list_add(&obj->userfault_link, &to_i915(obj->base.dev)->runtime_pm.lmem_userfault_list); + spin_unlock(&to_i915(obj->base.dev)->runtime_pm.lmem_userfault_lock); + + GEM_WARN_ON(!i915_ttm_cpu_maps_iomem(bo->resource)); } if (wakeref & CONFIG_DRM_I915_USERFAULT_AUTOSUSPEND) - intel_wakeref_auto(&to_gt(to_i915(obj->base.dev))->userfault_wakeref, + intel_wakeref_auto(&to_i915(obj->base.dev)->runtime_pm.userfault_wakeref, msecs_to_jiffies_timeout(CONFIG_DRM_I915_USERFAULT_AUTOSUSPEND)); i915_ttm_adjust_lru(obj); @@ -1098,7 +1135,7 @@ static void ttm_vm_open(struct vm_area_struct *vma) struct drm_i915_gem_object *obj = i915_ttm_to_gem(vma->vm_private_data); - GEM_BUG_ON(!obj); + GEM_BUG_ON(i915_ttm_is_ghost_object(vma->vm_private_data)); i915_gem_object_get(obj); } @@ -1107,7 +1144,7 @@ static void ttm_vm_close(struct vm_area_struct *vma) struct drm_i915_gem_object *obj = i915_ttm_to_gem(vma->vm_private_data); - GEM_BUG_ON(!obj); + GEM_BUG_ON(i915_ttm_is_ghost_object(vma->vm_private_data)); i915_gem_object_put(obj); } @@ -1128,7 +1165,29 @@ static u64 i915_ttm_mmap_offset(struct drm_i915_gem_object *obj) static void i915_ttm_unmap_virtual(struct drm_i915_gem_object *obj) { + struct ttm_buffer_object *bo = i915_gem_to_ttm(obj); + intel_wakeref_t wakeref = 0; + + assert_object_held_shared(obj); + + if (i915_ttm_cpu_maps_iomem(bo->resource)) { + wakeref = intel_runtime_pm_get(&to_i915(obj->base.dev)->runtime_pm); + + /* userfault_count is protected by obj lock and rpm wakeref. */ + if (obj->userfault_count) { + spin_lock(&to_i915(obj->base.dev)->runtime_pm.lmem_userfault_lock); + list_del(&obj->userfault_link); + spin_unlock(&to_i915(obj->base.dev)->runtime_pm.lmem_userfault_lock); + obj->userfault_count = 0; + } + } + + GEM_WARN_ON(obj->userfault_count); + ttm_bo_unmap_virtual(i915_gem_to_ttm(obj)); + + if (wakeref) + intel_runtime_pm_put(&to_i915(obj->base.dev)->runtime_pm, wakeref); } static const struct drm_i915_gem_object_ops i915_gem_ttm_obj_ops = { diff --git a/drivers/gpu/drm/i915/gem/i915_gem_ttm.h b/drivers/gpu/drm/i915/gem/i915_gem_ttm.h index e4842b4296fc..2a94a99ef76b 100644 --- a/drivers/gpu/drm/i915/gem/i915_gem_ttm.h +++ b/drivers/gpu/drm/i915/gem/i915_gem_ttm.h @@ -28,18 +28,26 @@ i915_gem_to_ttm(struct drm_i915_gem_object *obj) void i915_ttm_bo_destroy(struct ttm_buffer_object *bo); /** + * i915_ttm_is_ghost_object - Check if the ttm bo is a ghost object. + * @bo: Pointer to the ttm buffer object + * + * Return: True if the ttm bo is not a i915 object but a ghost ttm object, + * False otherwise. + */ +static inline bool i915_ttm_is_ghost_object(struct ttm_buffer_object *bo) +{ + return bo->destroy != i915_ttm_bo_destroy; +} + +/** * i915_ttm_to_gem - Convert a struct ttm_buffer_object to an embedding * struct drm_i915_gem_object. * - * Return: Pointer to the embedding struct ttm_buffer_object, or NULL - * if the object was not an i915 ttm object. + * Return: Pointer to the embedding struct ttm_buffer_object. */ static inline struct drm_i915_gem_object * i915_ttm_to_gem(struct ttm_buffer_object *bo) { - if (bo->destroy != i915_ttm_bo_destroy) - return NULL; - return container_of(bo, struct drm_i915_gem_object, __do_not_access); } diff --git a/drivers/gpu/drm/i915/gem/i915_gem_ttm_move.c b/drivers/gpu/drm/i915/gem/i915_gem_ttm_move.c index 9a7e50534b84..f59f812dc6d2 100644 --- a/drivers/gpu/drm/i915/gem/i915_gem_ttm_move.c +++ b/drivers/gpu/drm/i915/gem/i915_gem_ttm_move.c @@ -560,7 +560,7 @@ int i915_ttm_move(struct ttm_buffer_object *bo, bool evict, bool clear; int ret; - if (GEM_WARN_ON(!obj)) { + if (GEM_WARN_ON(i915_ttm_is_ghost_object(bo))) { ttm_bo_move_null(bo, dst_mem); return 0; } diff --git a/drivers/gpu/drm/i915/gem/i915_gem_userptr.c b/drivers/gpu/drm/i915/gem/i915_gem_userptr.c index ba14b18d65f3..9348b1804d53 100644 --- a/drivers/gpu/drm/i915/gem/i915_gem_userptr.c +++ b/drivers/gpu/drm/i915/gem/i915_gem_userptr.c @@ -131,7 +131,6 @@ static int i915_gem_userptr_get_pages(struct drm_i915_gem_object *obj) const unsigned long num_pages = obj->base.size >> PAGE_SHIFT; unsigned int max_segment = i915_sg_segment_size(obj->base.dev->dev); struct sg_table *st; - unsigned int sg_page_sizes; struct page **pvec; int ret; @@ -170,8 +169,7 @@ alloc_table: if (i915_gem_object_can_bypass_llc(obj)) obj->cache_dirty = true; - sg_page_sizes = i915_sg_dma_sizes(st->sgl); - __i915_gem_object_set_pages(obj, st, sg_page_sizes); + __i915_gem_object_set_pages(obj, st); return 0; @@ -292,7 +290,7 @@ int i915_gem_object_userptr_submit_init(struct drm_i915_gem_object *obj) if (!i915_gem_object_is_readonly(obj)) gup_flags |= FOLL_WRITE; - pinned = ret = 0; + pinned = 0; while (pinned < num_pages) { ret = pin_user_pages_fast(obj->userptr.ptr + pinned * PAGE_SIZE, num_pages - pinned, gup_flags, @@ -302,7 +300,6 @@ int i915_gem_object_userptr_submit_init(struct drm_i915_gem_object *obj) pinned += ret; } - ret = 0; ret = i915_gem_object_lock_interruptible(obj, NULL); if (ret) diff --git a/drivers/gpu/drm/i915/gem/selftests/huge_gem_object.c b/drivers/gpu/drm/i915/gem/selftests/huge_gem_object.c index f963b8e1e37b..cbd9b624a788 100644 --- a/drivers/gpu/drm/i915/gem/selftests/huge_gem_object.c +++ b/drivers/gpu/drm/i915/gem/selftests/huge_gem_object.c @@ -68,7 +68,7 @@ static int huge_get_pages(struct drm_i915_gem_object *obj) if (i915_gem_gtt_prepare_pages(obj, pages)) goto err; - __i915_gem_object_set_pages(obj, pages, PAGE_SIZE); + __i915_gem_object_set_pages(obj, pages); return 0; diff --git a/drivers/gpu/drm/i915/gem/selftests/huge_pages.c b/drivers/gpu/drm/i915/gem/selftests/huge_pages.c index c570cf780079..beaf27e09e8a 100644 --- a/drivers/gpu/drm/i915/gem/selftests/huge_pages.c +++ b/drivers/gpu/drm/i915/gem/selftests/huge_pages.c @@ -136,7 +136,7 @@ static int get_huge_pages(struct drm_i915_gem_object *obj) goto err; GEM_BUG_ON(sg_page_sizes != obj->mm.page_mask); - __i915_gem_object_set_pages(obj, st, sg_page_sizes); + __i915_gem_object_set_pages(obj, st); return 0; @@ -210,7 +210,6 @@ static int fake_get_huge_pages(struct drm_i915_gem_object *obj) const u64 max_len = rounddown_pow_of_two(UINT_MAX); struct sg_table *st; struct scatterlist *sg; - unsigned int sg_page_sizes; u64 rem; st = kmalloc(sizeof(*st), GFP); @@ -226,7 +225,6 @@ static int fake_get_huge_pages(struct drm_i915_gem_object *obj) rem = obj->base.size; sg = st->sgl; st->nents = 0; - sg_page_sizes = 0; do { unsigned int page_size = get_largest_page_size(i915, rem); unsigned int len = min(page_size * div_u64(rem, page_size), @@ -239,8 +237,6 @@ static int fake_get_huge_pages(struct drm_i915_gem_object *obj) sg_dma_len(sg) = len; sg_dma_address(sg) = page_size; - sg_page_sizes |= len; - st->nents++; rem -= len; @@ -254,7 +250,7 @@ static int fake_get_huge_pages(struct drm_i915_gem_object *obj) i915_sg_trim(st); - __i915_gem_object_set_pages(obj, st, sg_page_sizes); + __i915_gem_object_set_pages(obj, st); return 0; } @@ -286,7 +282,7 @@ static int fake_get_huge_pages_single(struct drm_i915_gem_object *obj) sg_dma_len(sg) = obj->base.size; sg_dma_address(sg) = page_size; - __i915_gem_object_set_pages(obj, st, sg->length); + __i915_gem_object_set_pages(obj, st); return 0; #undef GFP @@ -1161,7 +1157,8 @@ static int igt_write_huge(struct drm_i915_private *i915, GEM_BUG_ON(!i915_gem_object_has_pinned_pages(obj)); size = obj->base.size; - if (obj->mm.page_sizes.sg & I915_GTT_PAGE_SIZE_64K) + if (obj->mm.page_sizes.sg & I915_GTT_PAGE_SIZE_64K && + !HAS_64K_PAGES(i915)) size = round_up(size, I915_GTT_PAGE_SIZE_2M); n = 0; @@ -1214,6 +1211,10 @@ static int igt_write_huge(struct drm_i915_private *i915, * size and ensure the vma offset is at the start of the pt * boundary, however to improve coverage we opt for testing both * aligned and unaligned offsets. + * + * With PS64 this is no longer the case, but to ensure we + * sometimes get the compact layout for smaller objects, apply + * the round_up anyway. */ if (obj->mm.page_sizes.sg & I915_GTT_PAGE_SIZE_64K) offset_low = round_down(offset_low, @@ -1411,6 +1412,7 @@ static int igt_ppgtt_sanity_check(void *arg) { SZ_2M + SZ_4K, SZ_64K | SZ_4K }, { SZ_2M + SZ_4K, SZ_2M | SZ_4K }, { SZ_2M + SZ_64K, SZ_2M | SZ_64K }, + { SZ_2M + SZ_64K, SZ_64K }, }; int i, j; int err; @@ -1540,6 +1542,154 @@ out_put: return err; } +static int igt_ppgtt_mixed(void *arg) +{ + struct drm_i915_private *i915 = arg; + const unsigned long flags = PIN_OFFSET_FIXED | PIN_USER; + struct drm_i915_gem_object *obj, *on; + struct i915_gem_engines *engines; + struct i915_gem_engines_iter it; + struct i915_address_space *vm; + struct i915_gem_context *ctx; + struct intel_context *ce; + struct file *file; + I915_RND_STATE(prng); + LIST_HEAD(objects); + struct intel_memory_region *mr; + struct i915_vma *vma; + unsigned int count; + u32 i, addr; + int *order; + int n, err; + + /* + * Sanity check mixing 4K and 64K pages within the same page-table via + * the new PS64 TLB hint. + */ + + if (!HAS_64K_PAGES(i915)) { + pr_info("device lacks PS64, skipping\n"); + return 0; + } + + file = mock_file(i915); + if (IS_ERR(file)) + return PTR_ERR(file); + + ctx = hugepage_ctx(i915, file); + if (IS_ERR(ctx)) { + err = PTR_ERR(ctx); + goto out; + } + vm = i915_gem_context_get_eb_vm(ctx); + + i = 0; + addr = 0; + do { + u32 sz; + + sz = i915_prandom_u32_max_state(SZ_4M, &prng); + sz = max_t(u32, sz, SZ_4K); + + mr = i915->mm.regions[INTEL_REGION_LMEM_0]; + if (i & 1) + mr = i915->mm.regions[INTEL_REGION_SMEM]; + + obj = i915_gem_object_create_region(mr, sz, 0, 0); + if (IS_ERR(obj)) { + err = PTR_ERR(obj); + goto out_vm; + } + + list_add_tail(&obj->st_link, &objects); + + vma = i915_vma_instance(obj, vm, NULL); + if (IS_ERR(vma)) { + err = PTR_ERR(vma); + goto err_put; + } + + addr = round_up(addr, mr->min_page_size); + err = i915_vma_pin(vma, 0, 0, addr | flags); + if (err) + goto err_put; + + if (mr->type == INTEL_MEMORY_LOCAL && + (vma->resource->page_sizes_gtt & I915_GTT_PAGE_SIZE_4K)) { + err = -EINVAL; + goto err_put; + } + + addr += obj->base.size; + i++; + } while (addr <= SZ_16M); + + n = 0; + count = 0; + for_each_gem_engine(ce, i915_gem_context_lock_engines(ctx), it) { + count++; + if (!intel_engine_can_store_dword(ce->engine)) + continue; + + n++; + } + i915_gem_context_unlock_engines(ctx); + if (!n) + goto err_put; + + order = i915_random_order(count * count, &prng); + if (!order) { + err = -ENOMEM; + goto err_put; + } + + i = 0; + addr = 0; + engines = i915_gem_context_lock_engines(ctx); + list_for_each_entry(obj, &objects, st_link) { + u32 rnd = i915_prandom_u32_max_state(UINT_MAX, &prng); + + addr = round_up(addr, obj->mm.region->min_page_size); + + ce = engines->engines[order[i] % engines->num_engines]; + i = (i + 1) % (count * count); + if (!ce || !intel_engine_can_store_dword(ce->engine)) + continue; + + err = __igt_write_huge(ce, obj, obj->base.size, addr, 0, rnd); + if (err) + break; + + err = __igt_write_huge(ce, obj, obj->base.size, addr, + offset_in_page(rnd) / sizeof(u32), rnd + 1); + if (err) + break; + + err = __igt_write_huge(ce, obj, obj->base.size, addr, + (PAGE_SIZE / sizeof(u32)) - 1, + rnd + 2); + if (err) + break; + + addr += obj->base.size; + + cond_resched(); + } + + i915_gem_context_unlock_engines(ctx); + kfree(order); +err_put: + list_for_each_entry_safe(obj, on, &objects, st_link) { + list_del(&obj->st_link); + i915_gem_object_put(obj); + } +out_vm: + i915_vm_put(vm); +out: + fput(file); + return err; +} + static int igt_tmpfs_fallback(void *arg) { struct drm_i915_private *i915 = arg; @@ -1803,6 +1953,7 @@ int i915_gem_huge_page_live_selftests(struct drm_i915_private *i915) SUBTEST(igt_ppgtt_smoke_huge), SUBTEST(igt_ppgtt_sanity_check), SUBTEST(igt_ppgtt_compact), + SUBTEST(igt_ppgtt_mixed), }; if (!HAS_PPGTT(i915)) { diff --git a/drivers/gpu/drm/i915/gem/selftests/i915_gem_client_blt.c b/drivers/gpu/drm/i915/gem/selftests/i915_gem_client_blt.c index 9a6a6b5b722b..692a16914ca0 100644 --- a/drivers/gpu/drm/i915/gem/selftests/i915_gem_client_blt.c +++ b/drivers/gpu/drm/i915/gem/selftests/i915_gem_client_blt.c @@ -13,6 +13,7 @@ #include "gt/intel_gt_regs.h" #include "gem/i915_gem_lmem.h" +#include "gem/selftests/igt_gem_utils.h" #include "selftests/igt_flush_test.h" #include "selftests/mock_drm.h" #include "selftests/i915_random.h" @@ -457,21 +458,6 @@ static int verify_buffer(const struct tiled_blits *t, return ret; } -static int move_to_active(struct i915_vma *vma, - struct i915_request *rq, - unsigned int flags) -{ - int err; - - i915_vma_lock(vma); - err = i915_request_await_object(rq, vma->obj, false); - if (err == 0) - err = i915_vma_move_to_active(vma, rq, flags); - i915_vma_unlock(vma); - - return err; -} - static int pin_buffer(struct i915_vma *vma, u64 addr) { int err; @@ -525,11 +511,11 @@ tiled_blit(struct tiled_blits *t, goto err_bb; } - err = move_to_active(t->batch, rq, 0); + err = igt_vma_move_to_active_unlocked(t->batch, rq, 0); if (!err) - err = move_to_active(src->vma, rq, 0); + err = igt_vma_move_to_active_unlocked(src->vma, rq, 0); if (!err) - err = move_to_active(dst->vma, rq, 0); + err = igt_vma_move_to_active_unlocked(dst->vma, rq, 0); if (!err) err = rq->engine->emit_bb_start(rq, t->batch->node.start, diff --git a/drivers/gpu/drm/i915/gem/selftests/i915_gem_coherency.c b/drivers/gpu/drm/i915/gem/selftests/i915_gem_coherency.c index a666d7e610f5..c228fe4aba50 100644 --- a/drivers/gpu/drm/i915/gem/selftests/i915_gem_coherency.c +++ b/drivers/gpu/drm/i915/gem/selftests/i915_gem_coherency.c @@ -239,9 +239,7 @@ static int gpu_set(struct context *ctx, unsigned long offset, u32 v) } intel_ring_advance(rq, cs); - err = i915_request_await_object(rq, vma->obj, true); - if (err == 0) - err = i915_vma_move_to_active(vma, rq, EXEC_OBJECT_WRITE); + err = i915_vma_move_to_active(vma, rq, EXEC_OBJECT_WRITE); out_rq: i915_request_add(rq); diff --git a/drivers/gpu/drm/i915/gem/selftests/i915_gem_context.c b/drivers/gpu/drm/i915/gem/selftests/i915_gem_context.c index c6ad67b90e8a..a0ff51d71d07 100644 --- a/drivers/gpu/drm/i915/gem/selftests/i915_gem_context.c +++ b/drivers/gpu/drm/i915/gem/selftests/i915_gem_context.c @@ -179,97 +179,108 @@ out_file: } struct parallel_switch { - struct task_struct *tsk; + struct kthread_worker *worker; + struct kthread_work work; struct intel_context *ce[2]; + int result; }; -static int __live_parallel_switch1(void *data) +static void __live_parallel_switch1(struct kthread_work *work) { - struct parallel_switch *arg = data; + struct parallel_switch *arg = + container_of(work, typeof(*arg), work); IGT_TIMEOUT(end_time); unsigned long count; count = 0; + arg->result = 0; do { struct i915_request *rq = NULL; - int err, n; + int n; - err = 0; - for (n = 0; !err && n < ARRAY_SIZE(arg->ce); n++) { + for (n = 0; !arg->result && n < ARRAY_SIZE(arg->ce); n++) { struct i915_request *prev = rq; rq = i915_request_create(arg->ce[n]); if (IS_ERR(rq)) { i915_request_put(prev); - return PTR_ERR(rq); + arg->result = PTR_ERR(rq); + break; } i915_request_get(rq); if (prev) { - err = i915_request_await_dma_fence(rq, &prev->fence); + arg->result = + i915_request_await_dma_fence(rq, + &prev->fence); i915_request_put(prev); } i915_request_add(rq); } + + if (IS_ERR_OR_NULL(rq)) + break; + if (i915_request_wait(rq, 0, HZ) < 0) - err = -ETIME; + arg->result = -ETIME; + i915_request_put(rq); - if (err) - return err; count++; - } while (!__igt_timeout(end_time, NULL)); + } while (!arg->result && !__igt_timeout(end_time, NULL)); - pr_info("%s: %lu switches (sync)\n", arg->ce[0]->engine->name, count); - return 0; + pr_info("%s: %lu switches (sync) <%d>\n", + arg->ce[0]->engine->name, count, arg->result); } -static int __live_parallel_switchN(void *data) +static void __live_parallel_switchN(struct kthread_work *work) { - struct parallel_switch *arg = data; + struct parallel_switch *arg = + container_of(work, typeof(*arg), work); struct i915_request *rq = NULL; IGT_TIMEOUT(end_time); unsigned long count; int n; count = 0; + arg->result = 0; do { - for (n = 0; n < ARRAY_SIZE(arg->ce); n++) { + for (n = 0; !arg->result && n < ARRAY_SIZE(arg->ce); n++) { struct i915_request *prev = rq; - int err = 0; rq = i915_request_create(arg->ce[n]); if (IS_ERR(rq)) { i915_request_put(prev); - return PTR_ERR(rq); + arg->result = PTR_ERR(rq); + break; } i915_request_get(rq); if (prev) { - err = i915_request_await_dma_fence(rq, &prev->fence); + arg->result = + i915_request_await_dma_fence(rq, + &prev->fence); i915_request_put(prev); } i915_request_add(rq); - if (err) { - i915_request_put(rq); - return err; - } } count++; - } while (!__igt_timeout(end_time, NULL)); - i915_request_put(rq); + } while (!arg->result && !__igt_timeout(end_time, NULL)); - pr_info("%s: %lu switches (many)\n", arg->ce[0]->engine->name, count); - return 0; + if (!IS_ERR_OR_NULL(rq)) + i915_request_put(rq); + + pr_info("%s: %lu switches (many) <%d>\n", + arg->ce[0]->engine->name, count, arg->result); } static int live_parallel_switch(void *arg) { struct drm_i915_private *i915 = arg; - static int (* const func[])(void *arg) = { + static void (* const func[])(struct kthread_work *) = { __live_parallel_switch1, __live_parallel_switchN, NULL, @@ -277,7 +288,7 @@ static int live_parallel_switch(void *arg) struct parallel_switch *data = NULL; struct i915_gem_engines *engines; struct i915_gem_engines_iter it; - int (* const *fn)(void *arg); + void (* const *fn)(struct kthread_work *); struct i915_gem_context *ctx; struct intel_context *ce; struct file *file; @@ -348,9 +359,22 @@ static int live_parallel_switch(void *arg) } } + for (n = 0; n < count; n++) { + struct kthread_worker *worker; + + if (!data[n].ce[0]) + continue; + + worker = kthread_create_worker(0, "igt/parallel:%s", + data[n].ce[0]->engine->name); + if (IS_ERR(worker)) + goto out; + + data[n].worker = worker; + } + for (fn = func; !err && *fn; fn++) { struct igt_live_test t; - int n; err = igt_live_test_begin(&t, i915, __func__, ""); if (err) @@ -360,30 +384,17 @@ static int live_parallel_switch(void *arg) if (!data[n].ce[0]) continue; - data[n].tsk = kthread_run(*fn, &data[n], - "igt/parallel:%s", - data[n].ce[0]->engine->name); - if (IS_ERR(data[n].tsk)) { - err = PTR_ERR(data[n].tsk); - break; - } - get_task_struct(data[n].tsk); + data[n].result = 0; + kthread_init_work(&data[n].work, *fn); + kthread_queue_work(data[n].worker, &data[n].work); } - yield(); /* start all threads before we kthread_stop() */ - for (n = 0; n < count; n++) { - int status; - - if (IS_ERR_OR_NULL(data[n].tsk)) - continue; - - status = kthread_stop(data[n].tsk); - if (status && !err) - err = status; - - put_task_struct(data[n].tsk); - data[n].tsk = NULL; + if (data[n].ce[0]) { + kthread_flush_work(&data[n].work); + if (data[n].result && !err) + err = data[n].result; + } } if (igt_live_test_end(&t)) @@ -399,6 +410,9 @@ out: intel_context_unpin(data[n].ce[m]); intel_context_put(data[n].ce[m]); } + + if (data[n].worker) + kthread_destroy_worker(data[n].worker); } kfree(data); out_file: @@ -970,15 +984,11 @@ retry: goto err_batch; } - err = i915_request_await_object(rq, batch->obj, false); - if (err == 0) - err = i915_vma_move_to_active(batch, rq, 0); + err = i915_vma_move_to_active(batch, rq, 0); if (err) goto skip_request; - err = i915_request_await_object(rq, vma->obj, true); - if (err == 0) - err = i915_vma_move_to_active(vma, rq, EXEC_OBJECT_WRITE); + err = i915_vma_move_to_active(vma, rq, EXEC_OBJECT_WRITE); if (err) goto skip_request; @@ -1539,9 +1549,7 @@ static int write_to_scratch(struct i915_gem_context *ctx, } i915_vma_lock(vma); - err = i915_request_await_object(rq, vma->obj, false); - if (err == 0) - err = i915_vma_move_to_active(vma, rq, 0); + err = i915_vma_move_to_active(vma, rq, 0); i915_vma_unlock(vma); if (err) goto skip_request; @@ -1675,9 +1683,7 @@ static int read_from_scratch(struct i915_gem_context *ctx, } i915_vma_lock(vma); - err = i915_request_await_object(rq, vma->obj, true); - if (err == 0) - err = i915_vma_move_to_active(vma, rq, EXEC_OBJECT_WRITE); + err = i915_vma_move_to_active(vma, rq, EXEC_OBJECT_WRITE); i915_vma_unlock(vma); if (err) goto skip_request; diff --git a/drivers/gpu/drm/i915/gem/selftests/i915_gem_dmabuf.c b/drivers/gpu/drm/i915/gem/selftests/i915_gem_dmabuf.c index 51ed824b020c..e57f9390076c 100644 --- a/drivers/gpu/drm/i915/gem/selftests/i915_gem_dmabuf.c +++ b/drivers/gpu/drm/i915/gem/selftests/i915_gem_dmabuf.c @@ -6,8 +6,12 @@ #include "i915_drv.h" #include "i915_selftest.h" +#include "gem/i915_gem_context.h" +#include "mock_context.h" #include "mock_dmabuf.h" +#include "igt_gem_utils.h" +#include "selftests/mock_drm.h" #include "selftests/mock_gem_device.h" static int igt_dmabuf_export(void *arg) @@ -140,6 +144,75 @@ out_ret: return err; } +static int verify_access(struct drm_i915_private *i915, + struct drm_i915_gem_object *native_obj, + struct drm_i915_gem_object *import_obj) +{ + struct i915_gem_engines_iter it; + struct i915_gem_context *ctx; + struct intel_context *ce; + struct i915_vma *vma; + struct file *file; + u32 *vaddr; + int err = 0, i; + + file = mock_file(i915); + if (IS_ERR(file)) + return PTR_ERR(file); + + ctx = live_context(i915, file); + if (IS_ERR(ctx)) { + err = PTR_ERR(ctx); + goto out_file; + } + + for_each_gem_engine(ce, i915_gem_context_lock_engines(ctx), it) { + if (intel_engine_can_store_dword(ce->engine)) + break; + } + i915_gem_context_unlock_engines(ctx); + if (!ce) + goto out_file; + + vma = i915_vma_instance(import_obj, ce->vm, NULL); + if (IS_ERR(vma)) { + err = PTR_ERR(vma); + goto out_file; + } + + err = i915_vma_pin(vma, 0, 0, PIN_USER); + if (err) + goto out_file; + + err = igt_gpu_fill_dw(ce, vma, 0, + vma->size >> PAGE_SHIFT, 0xdeadbeaf); + i915_vma_unpin(vma); + if (err) + goto out_file; + + err = i915_gem_object_wait(import_obj, 0, MAX_SCHEDULE_TIMEOUT); + if (err) + goto out_file; + + vaddr = i915_gem_object_pin_map_unlocked(native_obj, I915_MAP_WB); + if (IS_ERR(vaddr)) { + err = PTR_ERR(vaddr); + goto out_file; + } + + for (i = 0; i < native_obj->base.size / sizeof(u32); i += PAGE_SIZE / sizeof(u32)) { + if (vaddr[i] != 0xdeadbeaf) { + pr_err("Data mismatch [%d]=%u\n", i, vaddr[i]); + err = -EINVAL; + goto out_file; + } + } + +out_file: + fput(file); + return err; +} + static int igt_dmabuf_import_same_driver(struct drm_i915_private *i915, struct intel_memory_region **regions, unsigned int num_regions) @@ -154,7 +227,7 @@ static int igt_dmabuf_import_same_driver(struct drm_i915_private *i915, force_different_devices = true; - obj = __i915_gem_object_create_user(i915, PAGE_SIZE, + obj = __i915_gem_object_create_user(i915, SZ_8M, regions, num_regions); if (IS_ERR(obj)) { pr_err("__i915_gem_object_create_user failed with err=%ld\n", @@ -206,6 +279,10 @@ static int igt_dmabuf_import_same_driver(struct drm_i915_private *i915, i915_gem_object_unlock(import_obj); + err = verify_access(i915, obj, import_obj); + if (err) + goto out_import; + /* Now try a fake an importer */ import_attach = dma_buf_attach(dmabuf, obj->base.dev->dev); if (IS_ERR(import_attach)) { @@ -213,7 +290,7 @@ static int igt_dmabuf_import_same_driver(struct drm_i915_private *i915, goto out_import; } - st = dma_buf_map_attachment(import_attach, DMA_BIDIRECTIONAL); + st = dma_buf_map_attachment_unlocked(import_attach, DMA_BIDIRECTIONAL); if (IS_ERR(st)) { err = PTR_ERR(st); goto out_detach; @@ -226,7 +303,7 @@ static int igt_dmabuf_import_same_driver(struct drm_i915_private *i915, timeout = -ETIME; } err = timeout > 0 ? 0 : timeout; - dma_buf_unmap_attachment(import_attach, st, DMA_BIDIRECTIONAL); + dma_buf_unmap_attachment_unlocked(import_attach, st, DMA_BIDIRECTIONAL); out_detach: dma_buf_detach(dmabuf, import_attach); out_import: @@ -296,7 +373,7 @@ static int igt_dmabuf_import(void *arg) goto out_obj; } - err = dma_buf_vmap(dmabuf, &map); + err = dma_buf_vmap_unlocked(dmabuf, &map); dma_map = err ? NULL : map.vaddr; if (!dma_map) { pr_err("dma_buf_vmap failed\n"); @@ -337,7 +414,7 @@ static int igt_dmabuf_import(void *arg) err = 0; out_dma_map: - dma_buf_vunmap(dmabuf, &map); + dma_buf_vunmap_unlocked(dmabuf, &map); out_obj: i915_gem_object_put(obj); out_dmabuf: @@ -358,7 +435,7 @@ static int igt_dmabuf_import_ownership(void *arg) if (IS_ERR(dmabuf)) return PTR_ERR(dmabuf); - err = dma_buf_vmap(dmabuf, &map); + err = dma_buf_vmap_unlocked(dmabuf, &map); ptr = err ? NULL : map.vaddr; if (!ptr) { pr_err("dma_buf_vmap failed\n"); @@ -367,7 +444,7 @@ static int igt_dmabuf_import_ownership(void *arg) } memset(ptr, 0xc5, PAGE_SIZE); - dma_buf_vunmap(dmabuf, &map); + dma_buf_vunmap_unlocked(dmabuf, &map); obj = to_intel_bo(i915_gem_prime_import(&i915->drm, dmabuf)); if (IS_ERR(obj)) { @@ -418,7 +495,7 @@ static int igt_dmabuf_export_vmap(void *arg) } i915_gem_object_put(obj); - err = dma_buf_vmap(dmabuf, &map); + err = dma_buf_vmap_unlocked(dmabuf, &map); ptr = err ? NULL : map.vaddr; if (!ptr) { pr_err("dma_buf_vmap failed\n"); @@ -435,7 +512,7 @@ static int igt_dmabuf_export_vmap(void *arg) memset(ptr, 0xc5, dmabuf->size); err = 0; - dma_buf_vunmap(dmabuf, &map); + dma_buf_vunmap_unlocked(dmabuf, &map); out: dma_buf_put(dmabuf); return err; diff --git a/drivers/gpu/drm/i915/gem/selftests/i915_gem_mman.c b/drivers/gpu/drm/i915/gem/selftests/i915_gem_mman.c index b73c91aa5450..3f658d5717d8 100644 --- a/drivers/gpu/drm/i915/gem/selftests/i915_gem_mman.c +++ b/drivers/gpu/drm/i915/gem/selftests/i915_gem_mman.c @@ -8,6 +8,7 @@ #include <linux/prime_numbers.h> #include "gem/i915_gem_internal.h" +#include "gem/i915_gem_lmem.h" #include "gem/i915_gem_region.h" #include "gem/i915_gem_ttm.h" #include "gem/i915_gem_ttm_move.h" @@ -16,6 +17,7 @@ #include "gt/intel_gt.h" #include "gt/intel_gt_pm.h" #include "gt/intel_migrate.h" +#include "i915_reg.h" #include "i915_ttm_buddy_manager.h" #include "huge_gem_object.h" @@ -564,10 +566,8 @@ retry: goto err_unpin; } - err = i915_request_await_object(rq, vma->obj, true); - if (err == 0) - err = i915_vma_move_to_active(vma, rq, - EXEC_OBJECT_WRITE); + err = i915_vma_move_to_active(vma, rq, + EXEC_OBJECT_WRITE); i915_request_add(rq); err_unpin: @@ -1607,9 +1607,7 @@ retry: goto out_unpin; } - err = i915_request_await_object(rq, vma->obj, false); - if (err == 0) - err = i915_vma_move_to_active(vma, rq, 0); + err = i915_vma_move_to_active(vma, rq, 0); err = engine->emit_bb_start(rq, vma->node.start, 0, 0); i915_request_get(rq); diff --git a/drivers/gpu/drm/i915/gem/selftests/igt_gem_utils.c b/drivers/gpu/drm/i915/gem/selftests/igt_gem_utils.c index 3c55e77b0f1b..374b10ac430e 100644 --- a/drivers/gpu/drm/i915/gem/selftests/igt_gem_utils.c +++ b/drivers/gpu/drm/i915/gem/selftests/igt_gem_utils.c @@ -131,17 +131,13 @@ int igt_gpu_fill_dw(struct intel_context *ce, } i915_vma_lock(batch); - err = i915_request_await_object(rq, batch->obj, false); - if (err == 0) - err = i915_vma_move_to_active(batch, rq, 0); + err = i915_vma_move_to_active(batch, rq, 0); i915_vma_unlock(batch); if (err) goto skip_request; i915_vma_lock(vma); - err = i915_request_await_object(rq, vma->obj, true); - if (err == 0) - err = i915_vma_move_to_active(vma, rq, EXEC_OBJECT_WRITE); + err = i915_vma_move_to_active(vma, rq, EXEC_OBJECT_WRITE); i915_vma_unlock(vma); if (err) goto skip_request; diff --git a/drivers/gpu/drm/i915/gem/selftests/igt_gem_utils.h b/drivers/gpu/drm/i915/gem/selftests/igt_gem_utils.h index 4221cf84d175..1379fbc14431 100644 --- a/drivers/gpu/drm/i915/gem/selftests/igt_gem_utils.h +++ b/drivers/gpu/drm/i915/gem/selftests/igt_gem_utils.h @@ -9,6 +9,8 @@ #include <linux/types.h> +#include "i915_vma.h" + struct i915_request; struct i915_gem_context; struct i915_vma; @@ -29,4 +31,16 @@ int igt_gpu_fill_dw(struct intel_context *ce, struct i915_vma *vma, u64 offset, unsigned long count, u32 val); +static inline int __must_check +igt_vma_move_to_active_unlocked(struct i915_vma *vma, struct i915_request *rq, + unsigned int flags) +{ + int err; + + i915_vma_lock(vma); + err = _i915_vma_move_to_active(vma, rq, &rq->fence, flags); + i915_vma_unlock(vma); + return err; +} + #endif /* __IGT_GEM_UTILS_H__ */ diff --git a/drivers/gpu/drm/i915/gt/gen8_engine_cs.c b/drivers/gpu/drm/i915/gt/gen8_engine_cs.c index e49fa6fa6aee..e1c76e5bfa82 100644 --- a/drivers/gpu/drm/i915/gt/gen8_engine_cs.c +++ b/drivers/gpu/drm/i915/gt/gen8_engine_cs.c @@ -396,15 +396,17 @@ int gen8_emit_init_breadcrumb(struct i915_request *rq) return 0; } -static int __gen125_emit_bb_start(struct i915_request *rq, - u64 offset, u32 len, - const unsigned int flags, - u32 arb) +static int __xehp_emit_bb_start(struct i915_request *rq, + u64 offset, u32 len, + const unsigned int flags, + u32 arb) { struct intel_context *ce = rq->context; u32 wa_offset = lrc_indirect_bb(ce); u32 *cs; + GEM_BUG_ON(!ce->wa_bb_page); + cs = intel_ring_begin(rq, 12); if (IS_ERR(cs)) return PTR_ERR(cs); @@ -435,18 +437,18 @@ static int __gen125_emit_bb_start(struct i915_request *rq, return 0; } -int gen125_emit_bb_start_noarb(struct i915_request *rq, - u64 offset, u32 len, - const unsigned int flags) +int xehp_emit_bb_start_noarb(struct i915_request *rq, + u64 offset, u32 len, + const unsigned int flags) { - return __gen125_emit_bb_start(rq, offset, len, flags, MI_ARB_DISABLE); + return __xehp_emit_bb_start(rq, offset, len, flags, MI_ARB_DISABLE); } -int gen125_emit_bb_start(struct i915_request *rq, - u64 offset, u32 len, - const unsigned int flags) +int xehp_emit_bb_start(struct i915_request *rq, + u64 offset, u32 len, + const unsigned int flags) { - return __gen125_emit_bb_start(rq, offset, len, flags, MI_ARB_ENABLE); + return __xehp_emit_bb_start(rq, offset, len, flags, MI_ARB_ENABLE); } int gen8_emit_bb_start_noarb(struct i915_request *rq, @@ -583,6 +585,8 @@ u32 *gen8_emit_fini_breadcrumb_xcs(struct i915_request *rq, u32 *cs) u32 *gen8_emit_fini_breadcrumb_rcs(struct i915_request *rq, u32 *cs) { cs = gen8_emit_pipe_control(cs, + PIPE_CONTROL_CS_STALL | + PIPE_CONTROL_TLB_INVALIDATE | PIPE_CONTROL_RENDER_TARGET_CACHE_FLUSH | PIPE_CONTROL_DEPTH_CACHE_FLUSH | PIPE_CONTROL_DC_FLUSH_ENABLE, @@ -600,15 +604,21 @@ u32 *gen8_emit_fini_breadcrumb_rcs(struct i915_request *rq, u32 *cs) u32 *gen11_emit_fini_breadcrumb_rcs(struct i915_request *rq, u32 *cs) { + cs = gen8_emit_pipe_control(cs, + PIPE_CONTROL_CS_STALL | + PIPE_CONTROL_TLB_INVALIDATE | + PIPE_CONTROL_TILE_CACHE_FLUSH | + PIPE_CONTROL_RENDER_TARGET_CACHE_FLUSH | + PIPE_CONTROL_DEPTH_CACHE_FLUSH | + PIPE_CONTROL_DC_FLUSH_ENABLE, + 0); + + /*XXX: Look at gen8_emit_fini_breadcrumb_rcs */ cs = gen8_emit_ggtt_write_rcs(cs, rq->fence.seqno, hwsp_offset(rq), - PIPE_CONTROL_CS_STALL | - PIPE_CONTROL_TILE_CACHE_FLUSH | - PIPE_CONTROL_RENDER_TARGET_CACHE_FLUSH | - PIPE_CONTROL_DEPTH_CACHE_FLUSH | - PIPE_CONTROL_DC_FLUSH_ENABLE | - PIPE_CONTROL_FLUSH_ENABLE); + PIPE_CONTROL_FLUSH_ENABLE | + PIPE_CONTROL_CS_STALL); return gen8_emit_fini_breadcrumb_tail(rq, cs); } @@ -715,6 +725,7 @@ u32 *gen12_emit_fini_breadcrumb_rcs(struct i915_request *rq, u32 *cs) { struct drm_i915_private *i915 = rq->engine->i915; u32 flags = (PIPE_CONTROL_CS_STALL | + PIPE_CONTROL_TLB_INVALIDATE | PIPE_CONTROL_TILE_CACHE_FLUSH | PIPE_CONTROL_FLUSH_L3 | PIPE_CONTROL_RENDER_TARGET_CACHE_FLUSH | @@ -731,11 +742,15 @@ u32 *gen12_emit_fini_breadcrumb_rcs(struct i915_request *rq, u32 *cs) else if (rq->engine->class == COMPUTE_CLASS) flags &= ~PIPE_CONTROL_3D_ENGINE_FLAGS; + cs = gen12_emit_pipe_control(cs, PIPE_CONTROL0_HDC_PIPELINE_FLUSH, flags, 0); + + /*XXX: Look at gen8_emit_fini_breadcrumb_rcs */ cs = gen12_emit_ggtt_write_rcs(cs, rq->fence.seqno, hwsp_offset(rq), - PIPE_CONTROL0_HDC_PIPELINE_FLUSH, - flags); + 0, + PIPE_CONTROL_FLUSH_ENABLE | + PIPE_CONTROL_CS_STALL); return gen12_emit_fini_breadcrumb_tail(rq, cs); } diff --git a/drivers/gpu/drm/i915/gt/gen8_engine_cs.h b/drivers/gpu/drm/i915/gt/gen8_engine_cs.h index e4d24c811dd6..655e5c00ddc2 100644 --- a/drivers/gpu/drm/i915/gt/gen8_engine_cs.h +++ b/drivers/gpu/drm/i915/gt/gen8_engine_cs.h @@ -32,12 +32,12 @@ int gen8_emit_bb_start(struct i915_request *rq, u64 offset, u32 len, const unsigned int flags); -int gen125_emit_bb_start_noarb(struct i915_request *rq, - u64 offset, u32 len, - const unsigned int flags); -int gen125_emit_bb_start(struct i915_request *rq, - u64 offset, u32 len, - const unsigned int flags); +int xehp_emit_bb_start_noarb(struct i915_request *rq, + u64 offset, u32 len, + const unsigned int flags); +int xehp_emit_bb_start(struct i915_request *rq, + u64 offset, u32 len, + const unsigned int flags); u32 *gen8_emit_fini_breadcrumb_xcs(struct i915_request *rq, u32 *cs); u32 *gen12_emit_fini_breadcrumb_xcs(struct i915_request *rq, u32 *cs); diff --git a/drivers/gpu/drm/i915/gt/gen8_ppgtt.c b/drivers/gpu/drm/i915/gt/gen8_ppgtt.c index 2128b7a72a25..4daaa6f55668 100644 --- a/drivers/gpu/drm/i915/gt/gen8_ppgtt.c +++ b/drivers/gpu/drm/i915/gt/gen8_ppgtt.c @@ -476,6 +476,7 @@ xehpsdv_ppgtt_insert_huge(struct i915_address_space *vm, const gen8_pte_t pte_encode = vm->pte_encode(0, cache_level, flags); unsigned int rem = sg_dma_len(iter->sg); u64 start = vma_res->start; + u64 end = start + vma_res->vma_size; GEM_BUG_ON(!i915_vm_is_4lvl(vm)); @@ -489,9 +490,10 @@ xehpsdv_ppgtt_insert_huge(struct i915_address_space *vm, gen8_pte_t encode = pte_encode; unsigned int page_size; gen8_pte_t *vaddr; - u16 index, max; + u16 index, max, nent, i; max = I915_PDES; + nent = 1; if (vma_res->bi.page_sizes.sg & I915_GTT_PAGE_SIZE_2M && IS_ALIGNED(iter->dma, I915_GTT_PAGE_SIZE_2M) && @@ -503,25 +505,37 @@ xehpsdv_ppgtt_insert_huge(struct i915_address_space *vm, vaddr = px_vaddr(pd); } else { - if (encode & GEN12_PPGTT_PTE_LM) { - GEM_BUG_ON(__gen8_pte_index(start, 0) % 16); - GEM_BUG_ON(rem < I915_GTT_PAGE_SIZE_64K); - GEM_BUG_ON(!IS_ALIGNED(iter->dma, - I915_GTT_PAGE_SIZE_64K)); - - index = __gen8_pte_index(start, 0) / 16; - page_size = I915_GTT_PAGE_SIZE_64K; - - max /= 16; - - vaddr = px_vaddr(pd); - vaddr[__gen8_pte_index(start, 1)] |= GEN12_PDE_64K; + index = __gen8_pte_index(start, 0); + page_size = I915_GTT_PAGE_SIZE; - pt->is_compact = true; - } else { - GEM_BUG_ON(pt->is_compact); - index = __gen8_pte_index(start, 0); - page_size = I915_GTT_PAGE_SIZE; + if (vma_res->bi.page_sizes.sg & I915_GTT_PAGE_SIZE_64K) { + /* + * Device local-memory on these platforms should + * always use 64K pages or larger (including GTT + * alignment), therefore if we know the whole + * page-table needs to be filled we can always + * safely use the compact-layout. Otherwise fall + * back to the TLB hint with PS64. If this is + * system memory we only bother with PS64. + */ + if ((encode & GEN12_PPGTT_PTE_LM) && + end - start >= SZ_2M && !index) { + index = __gen8_pte_index(start, 0) / 16; + page_size = I915_GTT_PAGE_SIZE_64K; + + max /= 16; + + vaddr = px_vaddr(pd); + vaddr[__gen8_pte_index(start, 1)] |= GEN12_PDE_64K; + + pt->is_compact = true; + } else if (IS_ALIGNED(iter->dma, I915_GTT_PAGE_SIZE_64K) && + rem >= I915_GTT_PAGE_SIZE_64K && + !(index % 16)) { + encode |= GEN12_PTE_PS64; + page_size = I915_GTT_PAGE_SIZE_64K; + nent = 16; + } } vaddr = px_vaddr(pt); @@ -529,7 +543,12 @@ xehpsdv_ppgtt_insert_huge(struct i915_address_space *vm, do { GEM_BUG_ON(rem < page_size); - vaddr[index++] = encode | iter->dma; + + for (i = 0; i < nent; i++) { + vaddr[index++] = + encode | (iter->dma + i * + I915_GTT_PAGE_SIZE); + } start += page_size; iter->dma += page_size; @@ -745,6 +764,8 @@ static void __xehpsdv_ppgtt_insert_entry_lm(struct i915_address_space *vm, GEM_BUG_ON(!IS_ALIGNED(addr, SZ_64K)); GEM_BUG_ON(!IS_ALIGNED(offset, SZ_64K)); + /* XXX: we don't strictly need to use this layout */ + if (!pt->is_compact) { vaddr = px_vaddr(pd); vaddr[gen8_pd_index(idx, 1)] |= GEN12_PDE_64K; @@ -929,29 +950,18 @@ struct i915_ppgtt *gen8_ppgtt_create(struct intel_gt *gt, */ ppgtt->vm.has_read_only = !IS_GRAPHICS_VER(gt->i915, 11, 12); - if (HAS_LMEM(gt->i915)) { + if (HAS_LMEM(gt->i915)) ppgtt->vm.alloc_pt_dma = alloc_pt_lmem; - - /* - * On some platforms the hw has dropped support for 4K GTT pages - * when dealing with LMEM, and due to the design of 64K GTT - * pages in the hw, we can only mark the *entire* page-table as - * operating in 64K GTT mode, since the enable bit is still on - * the pde, and not the pte. And since we still need to allow - * 4K GTT pages for SMEM objects, we can't have a "normal" 4K - * page-table with scratch pointing to LMEM, since that's - * undefined from the hw pov. The simplest solution is to just - * move the 64K scratch page to SMEM on such platforms and call - * it a day, since that should work for all configurations. - */ - if (HAS_64K_PAGES(gt->i915)) - ppgtt->vm.alloc_scratch_dma = alloc_pt_dma; - else - ppgtt->vm.alloc_scratch_dma = alloc_pt_lmem; - } else { + else ppgtt->vm.alloc_pt_dma = alloc_pt_dma; - ppgtt->vm.alloc_scratch_dma = alloc_pt_dma; - } + + /* + * Using SMEM here instead of LMEM has the advantage of not reserving + * high performance memory for a "never" used filler page. It also + * removes the device access that would be required to initialise the + * scratch page, reducing pressure on an even scarcer resource. + */ + ppgtt->vm.alloc_scratch_dma = alloc_pt_dma; ppgtt->vm.pte_encode = gen8_pte_encode; diff --git a/drivers/gpu/drm/i915/gt/intel_context.h b/drivers/gpu/drm/i915/gt/intel_context.h index be09fb2e883a..fb62b7b8cbcd 100644 --- a/drivers/gpu/drm/i915/gt/intel_context.h +++ b/drivers/gpu/drm/i915/gt/intel_context.h @@ -276,6 +276,14 @@ static inline bool intel_context_is_barrier(const struct intel_context *ce) return test_bit(CONTEXT_BARRIER_BIT, &ce->flags); } +static inline void intel_context_close(struct intel_context *ce) +{ + set_bit(CONTEXT_CLOSED_BIT, &ce->flags); + + if (ce->ops->close) + ce->ops->close(ce); +} + static inline bool intel_context_is_closed(const struct intel_context *ce) { return test_bit(CONTEXT_CLOSED_BIT, &ce->flags); diff --git a/drivers/gpu/drm/i915/gt/intel_context_types.h b/drivers/gpu/drm/i915/gt/intel_context_types.h index 04eacae1aca5..e36670f2e626 100644 --- a/drivers/gpu/drm/i915/gt/intel_context_types.h +++ b/drivers/gpu/drm/i915/gt/intel_context_types.h @@ -43,6 +43,8 @@ struct intel_context_ops { void (*revoke)(struct intel_context *ce, struct i915_request *rq, unsigned int preempt_timeout_ms); + void (*close)(struct intel_context *ce); + int (*pre_pin)(struct intel_context *ce, struct i915_gem_ww_ctx *ww, void **vaddr); int (*pin)(struct intel_context *ce, void *vaddr); void (*unpin)(struct intel_context *ce); @@ -197,8 +199,6 @@ struct intel_context { * context's submissions is complete. */ struct i915_sw_fence blocked; - /** @number_committed_requests: number of committed requests */ - int number_committed_requests; /** @requests: list of active requests on this context */ struct list_head requests; /** @prio: the context's current guc priority */ @@ -208,6 +208,11 @@ struct intel_context { * each priority bucket */ u32 prio_count[GUC_CLIENT_PRIORITY_NUM]; + /** + * @sched_disable_delay_work: worker to disable scheduling on this + * context + */ + struct delayed_work sched_disable_delay_work; } guc_state; struct { diff --git a/drivers/gpu/drm/i915/gt/intel_engine.h b/drivers/gpu/drm/i915/gt/intel_engine.h index 04e435bce79b..cbc8b857d5f7 100644 --- a/drivers/gpu/drm/i915/gt/intel_engine.h +++ b/drivers/gpu/drm/i915/gt/intel_engine.h @@ -348,4 +348,10 @@ intel_engine_get_hung_context(struct intel_engine_cs *engine) return engine->hung_ce; } +u64 intel_clamp_heartbeat_interval_ms(struct intel_engine_cs *engine, u64 value); +u64 intel_clamp_max_busywait_duration_ns(struct intel_engine_cs *engine, u64 value); +u64 intel_clamp_preempt_timeout_ms(struct intel_engine_cs *engine, u64 value); +u64 intel_clamp_stop_timeout_ms(struct intel_engine_cs *engine, u64 value); +u64 intel_clamp_timeslice_duration_ms(struct intel_engine_cs *engine, u64 value); + #endif /* _INTEL_RINGBUFFER_H_ */ diff --git a/drivers/gpu/drm/i915/gt/intel_engine_cs.c b/drivers/gpu/drm/i915/gt/intel_engine_cs.c index 1f7188129cd1..c33e0d72d670 100644 --- a/drivers/gpu/drm/i915/gt/intel_engine_cs.c +++ b/drivers/gpu/drm/i915/gt/intel_engine_cs.c @@ -13,6 +13,8 @@ #include "i915_cmd_parser.h" #include "i915_drv.h" +#include "i915_irq.h" +#include "i915_reg.h" #include "intel_breadcrumbs.h" #include "intel_context.h" #include "intel_engine.h" @@ -244,6 +246,13 @@ static const struct engine_info intel_engines[] = { { .graphics_ver = 12, .base = GEN12_COMPUTE3_RING_BASE } } }, + [GSC0] = { + .class = OTHER_CLASS, + .instance = OTHER_GSC_INSTANCE, + .mmio_bases = { + { .graphics_ver = 12, .base = MTL_GSC_RING_BASE } + } + }, }; /** @@ -324,6 +333,7 @@ u32 intel_engine_context_size(struct intel_gt *gt, u8 class) case VIDEO_DECODE_CLASS: case VIDEO_ENHANCEMENT_CLASS: case COPY_ENGINE_CLASS: + case OTHER_CLASS: if (GRAPHICS_VER(gt->i915) < 8) return 0; return GEN8_LR_CONTEXT_OTHER_SIZE; @@ -415,6 +425,7 @@ static u32 get_reset_domain(u8 ver, enum intel_engine_id id) [CCS1] = GEN11_GRDOM_RENDER, [CCS2] = GEN11_GRDOM_RENDER, [CCS3] = GEN11_GRDOM_RENDER, + [GSC0] = GEN12_GRDOM_GSC, }; GEM_BUG_ON(id >= ARRAY_SIZE(engine_reset_domains) || !engine_reset_domains[id]); @@ -486,6 +497,17 @@ static int intel_engine_setup(struct intel_gt *gt, enum intel_engine_id id, engine->logical_mask = BIT(logical_instance); __sprint_engine_name(engine); + if ((engine->class == COMPUTE_CLASS && !RCS_MASK(engine->gt) && + __ffs(CCS_MASK(engine->gt)) == engine->instance) || + engine->class == RENDER_CLASS) + engine->flags |= I915_ENGINE_FIRST_RENDER_COMPUTE; + + /* features common between engines sharing EUs */ + if (engine->class == RENDER_CLASS || engine->class == COMPUTE_CLASS) { + engine->flags |= I915_ENGINE_HAS_RCS_REG_STATE; + engine->flags |= I915_ENGINE_HAS_EU_PRIORITY; + } + engine->props.heartbeat_interval_ms = CONFIG_DRM_I915_HEARTBEAT_INTERVAL; engine->props.max_busywait_duration_ns = @@ -497,20 +519,34 @@ static int intel_engine_setup(struct intel_gt *gt, enum intel_engine_id id, engine->props.timeslice_duration_ms = CONFIG_DRM_I915_TIMESLICE_DURATION; - /* Override to uninterruptible for OpenCL workloads. */ - if (GRAPHICS_VER(i915) == 12 && engine->class == RENDER_CLASS) - engine->props.preempt_timeout_ms = 0; - - if ((engine->class == COMPUTE_CLASS && !RCS_MASK(engine->gt) && - __ffs(CCS_MASK(engine->gt)) == engine->instance) || - engine->class == RENDER_CLASS) - engine->flags |= I915_ENGINE_FIRST_RENDER_COMPUTE; - - /* features common between engines sharing EUs */ - if (engine->class == RENDER_CLASS || engine->class == COMPUTE_CLASS) { - engine->flags |= I915_ENGINE_HAS_RCS_REG_STATE; - engine->flags |= I915_ENGINE_HAS_EU_PRIORITY; - } + /* + * Mid-thread pre-emption is not available in Gen12. Unfortunately, + * some compute workloads run quite long threads. That means they get + * reset due to not pre-empting in a timely manner. So, bump the + * pre-emption timeout value to be much higher for compute engines. + */ + if (GRAPHICS_VER(i915) == 12 && (engine->flags & I915_ENGINE_HAS_RCS_REG_STATE)) + engine->props.preempt_timeout_ms = CONFIG_DRM_I915_PREEMPT_TIMEOUT_COMPUTE; + + /* Cap properties according to any system limits */ +#define CLAMP_PROP(field) \ + do { \ + u64 clamp = intel_clamp_##field(engine, engine->props.field); \ + if (clamp != engine->props.field) { \ + drm_notice(&engine->i915->drm, \ + "Warning, clamping %s to %lld to prevent overflow\n", \ + #field, clamp); \ + engine->props.field = clamp; \ + } \ + } while (0) + + CLAMP_PROP(heartbeat_interval_ms); + CLAMP_PROP(max_busywait_duration_ns); + CLAMP_PROP(preempt_timeout_ms); + CLAMP_PROP(stop_timeout_ms); + CLAMP_PROP(timeslice_duration_ms); + +#undef CLAMP_PROP engine->defaults = engine->props; /* never to change again */ @@ -534,6 +570,55 @@ static int intel_engine_setup(struct intel_gt *gt, enum intel_engine_id id, return 0; } +u64 intel_clamp_heartbeat_interval_ms(struct intel_engine_cs *engine, u64 value) +{ + value = min_t(u64, value, jiffies_to_msecs(MAX_SCHEDULE_TIMEOUT)); + + return value; +} + +u64 intel_clamp_max_busywait_duration_ns(struct intel_engine_cs *engine, u64 value) +{ + value = min(value, jiffies_to_nsecs(2)); + + return value; +} + +u64 intel_clamp_preempt_timeout_ms(struct intel_engine_cs *engine, u64 value) +{ + /* + * NB: The GuC API only supports 32bit values. However, the limit is further + * reduced due to internal calculations which would otherwise overflow. + */ + if (intel_guc_submission_is_wanted(&engine->gt->uc.guc)) + value = min_t(u64, value, guc_policy_max_preempt_timeout_ms()); + + value = min_t(u64, value, jiffies_to_msecs(MAX_SCHEDULE_TIMEOUT)); + + return value; +} + +u64 intel_clamp_stop_timeout_ms(struct intel_engine_cs *engine, u64 value) +{ + value = min_t(u64, value, jiffies_to_msecs(MAX_SCHEDULE_TIMEOUT)); + + return value; +} + +u64 intel_clamp_timeslice_duration_ms(struct intel_engine_cs *engine, u64 value) +{ + /* + * NB: The GuC API only supports 32bit values. However, the limit is further + * reduced due to internal calculations which would otherwise overflow. + */ + if (intel_guc_submission_is_wanted(&engine->gt->uc.guc)) + value = min_t(u64, value, guc_policy_max_exec_quantum_ms()); + + value = min_t(u64, value, jiffies_to_msecs(MAX_SCHEDULE_TIMEOUT)); + + return value; +} + static void __setup_engine_capabilities(struct intel_engine_cs *engine) { struct drm_i915_private *i915 = engine->i915; @@ -1274,8 +1359,13 @@ int intel_engines_init(struct intel_gt *gt) return err; err = setup(engine); - if (err) + if (err) { + intel_engine_cleanup_common(engine); return err; + } + + /* The backend should now be responsible for cleanup */ + GEM_BUG_ON(engine->release == NULL); err = engine_init_common(engine); if (err) @@ -1554,11 +1644,11 @@ void intel_engine_get_instdone(const struct intel_engine_cs *engine, for_each_ss_steering(iter, engine->gt, slice, subslice) { instdone->sampler[slice][subslice] = intel_gt_mcr_read(engine->gt, - GEN7_SAMPLER_INSTDONE, + GEN8_SAMPLER_INSTDONE, slice, subslice); instdone->row[slice][subslice] = intel_gt_mcr_read(engine->gt, - GEN7_ROW_INSTDONE, + GEN8_ROW_INSTDONE, slice, subslice); } diff --git a/drivers/gpu/drm/i915/gt/intel_engine_heartbeat.c b/drivers/gpu/drm/i915/gt/intel_engine_heartbeat.c index a3698f611f45..9a527e1f5be6 100644 --- a/drivers/gpu/drm/i915/gt/intel_engine_heartbeat.c +++ b/drivers/gpu/drm/i915/gt/intel_engine_heartbeat.c @@ -22,9 +22,37 @@ static bool next_heartbeat(struct intel_engine_cs *engine) { + struct i915_request *rq; long delay; delay = READ_ONCE(engine->props.heartbeat_interval_ms); + + rq = engine->heartbeat.systole; + + /* + * FIXME: The final period extension is disabled if the period has been + * modified from the default. This is to prevent issues with certain + * selftests which override the value and expect specific behaviour. + * Once the selftests have been updated to either cope with variable + * heartbeat periods (or to override the pre-emption timeout as well, + * or just to add a selftest specific override of the extension), the + * generic override can be removed. + */ + if (rq && rq->sched.attr.priority >= I915_PRIORITY_BARRIER && + delay == engine->defaults.heartbeat_interval_ms) { + long longer; + + /* + * The final try is at the highest priority possible. Up until now + * a pre-emption might not even have been attempted. So make sure + * this last attempt allows enough time for a pre-emption to occur. + */ + longer = READ_ONCE(engine->props.preempt_timeout_ms) * 2; + longer = intel_clamp_heartbeat_interval_ms(engine, longer); + if (longer > delay) + delay = longer; + } + if (!delay) return false; @@ -288,6 +316,17 @@ int intel_engine_set_heartbeat(struct intel_engine_cs *engine, if (!delay && !intel_engine_has_preempt_reset(engine)) return -ENODEV; + /* FIXME: Remove together with equally marked hack in next_heartbeat. */ + if (delay != engine->defaults.heartbeat_interval_ms && + delay < 2 * engine->props.preempt_timeout_ms) { + if (intel_engine_uses_guc(engine)) + drm_notice(&engine->i915->drm, "%s heartbeat interval adjusted to a non-default value which may downgrade individual engine resets to full GPU resets!\n", + engine->name); + else + drm_notice(&engine->i915->drm, "%s heartbeat interval adjusted to a non-default value which may cause engine resets to target innocent contexts!\n", + engine->name); + } + intel_engine_pm_get(engine); err = mutex_lock_interruptible(&ce->timeline->mutex); diff --git a/drivers/gpu/drm/i915/gt/intel_engine_regs.h b/drivers/gpu/drm/i915/gt/intel_engine_regs.h index fe1a0d5fd4b1..ee3efd06ee54 100644 --- a/drivers/gpu/drm/i915/gt/intel_engine_regs.h +++ b/drivers/gpu/drm/i915/gt/intel_engine_regs.h @@ -201,6 +201,7 @@ #define RING_CONTEXT_STATUS_PTR(base) _MMIO((base) + 0x3a0) #define RING_CTX_TIMESTAMP(base) _MMIO((base) + 0x3a8) /* gen8+ */ #define RING_PREDICATE_RESULT(base) _MMIO((base) + 0x3b8) +#define MI_PREDICATE_RESULT_2_ENGINE(base) _MMIO((base) + 0x3bc) #define RING_FORCE_TO_NONPRIV(base, i) _MMIO(((base) + 0x4D0) + (i) * 4) #define RING_FORCE_TO_NONPRIV_DENY REG_BIT(30) #define RING_FORCE_TO_NONPRIV_ADDRESS_MASK REG_GENMASK(25, 2) diff --git a/drivers/gpu/drm/i915/gt/intel_engine_types.h b/drivers/gpu/drm/i915/gt/intel_engine_types.h index 6b5d4ea22b67..4fd54fb8810f 100644 --- a/drivers/gpu/drm/i915/gt/intel_engine_types.h +++ b/drivers/gpu/drm/i915/gt/intel_engine_types.h @@ -136,6 +136,7 @@ enum intel_engine_id { CCS2, CCS3, #define _CCS(n) (CCS0 + (n)) + GSC0, I915_NUM_ENGINES #define INVALID_ENGINE ((enum intel_engine_id)-1) }; diff --git a/drivers/gpu/drm/i915/gt/intel_engine_user.c b/drivers/gpu/drm/i915/gt/intel_engine_user.c index 46a174f8aa00..cd4f1b126f75 100644 --- a/drivers/gpu/drm/i915/gt/intel_engine_user.c +++ b/drivers/gpu/drm/i915/gt/intel_engine_user.c @@ -140,6 +140,7 @@ const char *intel_engine_class_repr(u8 class) [COPY_ENGINE_CLASS] = "bcs", [VIDEO_DECODE_CLASS] = "vcs", [VIDEO_ENHANCEMENT_CLASS] = "vecs", + [OTHER_CLASS] = "other", [COMPUTE_CLASS] = "ccs", }; @@ -190,6 +191,15 @@ static void add_legacy_ring(struct legacy_ring *ring, ring->instance++; } +static void engine_rename(struct intel_engine_cs *engine, const char *name, u16 instance) +{ + char old[sizeof(engine->name)]; + + memcpy(old, engine->name, sizeof(engine->name)); + scnprintf(engine->name, sizeof(engine->name), "%s%u", name, instance); + drm_dbg(&engine->i915->drm, "renamed %s to %s\n", old, engine->name); +} + void intel_engines_driver_register(struct drm_i915_private *i915) { struct legacy_ring ring = {}; @@ -205,11 +215,19 @@ void intel_engines_driver_register(struct drm_i915_private *i915) struct intel_engine_cs *engine = container_of((struct rb_node *)it, typeof(*engine), uabi_node); - char old[sizeof(engine->name)]; if (intel_gt_has_unrecoverable_error(engine->gt)) continue; /* ignore incomplete engines */ + /* + * We don't want to expose the GSC engine to the users, but we + * still rename it so it is easier to identify in the debug logs + */ + if (engine->id == GSC0) { + engine_rename(engine, "gsc", 0); + continue; + } + GEM_BUG_ON(engine->class >= ARRAY_SIZE(uabi_classes)); engine->uabi_class = uabi_classes[engine->class]; @@ -219,11 +237,9 @@ void intel_engines_driver_register(struct drm_i915_private *i915) i915->engine_uabi_class_count[engine->uabi_class]++; /* Replace the internal name with the final user facing name */ - memcpy(old, engine->name, sizeof(engine->name)); - scnprintf(engine->name, sizeof(engine->name), "%s%u", - intel_engine_class_repr(engine->class), - engine->uabi_instance); - DRM_DEBUG_DRIVER("renamed %s to %s\n", old, engine->name); + engine_rename(engine, + intel_engine_class_repr(engine->class), + engine->uabi_instance); rb_link_node(&engine->uabi_node, prev, p); rb_insert_color(&engine->uabi_node, &i915->uabi_engines); diff --git a/drivers/gpu/drm/i915/gt/intel_execlists_submission.c b/drivers/gpu/drm/i915/gt/intel_execlists_submission.c index 45b605e32c87..2daffa7c7dfd 100644 --- a/drivers/gpu/drm/i915/gt/intel_execlists_submission.c +++ b/drivers/gpu/drm/i915/gt/intel_execlists_submission.c @@ -110,6 +110,7 @@ #include <linux/string_helpers.h> #include "i915_drv.h" +#include "i915_reg.h" #include "i915_trace.h" #include "i915_vgpu.h" #include "gen8_engine_cs.h" @@ -3471,9 +3472,9 @@ logical_ring_default_vfuncs(struct intel_engine_cs *engine) if (GRAPHICS_VER_FULL(engine->i915) >= IP_VER(12, 50)) { if (intel_engine_has_preemption(engine)) - engine->emit_bb_start = gen125_emit_bb_start; + engine->emit_bb_start = xehp_emit_bb_start; else - engine->emit_bb_start = gen125_emit_bb_start_noarb; + engine->emit_bb_start = xehp_emit_bb_start_noarb; } else { if (intel_engine_has_preemption(engine)) engine->emit_bb_start = gen8_emit_bb_start; @@ -3921,6 +3922,7 @@ static struct intel_context * execlists_create_virtual(struct intel_engine_cs **siblings, unsigned int count, unsigned long flags) { + struct drm_i915_private *i915 = siblings[0]->i915; struct virtual_engine *ve; unsigned int n; int err; @@ -3929,7 +3931,7 @@ execlists_create_virtual(struct intel_engine_cs **siblings, unsigned int count, if (!ve) return ERR_PTR(-ENOMEM); - ve->base.i915 = siblings[0]->i915; + ve->base.i915 = i915; ve->base.gt = siblings[0]->gt; ve->base.uncore = siblings[0]->uncore; ve->base.id = -1; @@ -3988,8 +3990,9 @@ execlists_create_virtual(struct intel_engine_cs **siblings, unsigned int count, GEM_BUG_ON(!is_power_of_2(sibling->mask)); if (sibling->mask & ve->base.mask) { - DRM_DEBUG("duplicate %s entry in load balancer\n", - sibling->name); + drm_dbg(&i915->drm, + "duplicate %s entry in load balancer\n", + sibling->name); err = -EINVAL; goto err_put; } @@ -4023,8 +4026,9 @@ execlists_create_virtual(struct intel_engine_cs **siblings, unsigned int count, */ if (ve->base.class != OTHER_CLASS) { if (ve->base.class != sibling->class) { - DRM_DEBUG("invalid mixing of engine class, sibling %d, already %d\n", - sibling->class, ve->base.class); + drm_dbg(&i915->drm, + "invalid mixing of engine class, sibling %d, already %d\n", + sibling->class, ve->base.class); err = -EINVAL; goto err_put; } diff --git a/drivers/gpu/drm/i915/gt/intel_ggtt.c b/drivers/gpu/drm/i915/gt/intel_ggtt.c index 2049a00417af..8145851ad23d 100644 --- a/drivers/gpu/drm/i915/gt/intel_ggtt.c +++ b/drivers/gpu/drm/i915/gt/intel_ggtt.c @@ -560,7 +560,7 @@ static int init_ggtt(struct i915_ggtt *ggtt) * why. */ ggtt->pin_bias = max_t(u32, I915_GTT_PAGE_SIZE, - intel_wopcm_guc_size(&ggtt->vm.i915->wopcm)); + intel_wopcm_guc_size(&ggtt->vm.gt->wopcm)); ret = intel_vgt_balloon(ggtt); if (ret) @@ -871,8 +871,8 @@ static int ggtt_probe_common(struct i915_ggtt *ggtt, u64 size) u32 pte_flags; int ret; - GEM_WARN_ON(pci_resource_len(pdev, GTTMMADR_BAR) != gen6_gttmmadr_size(i915)); - phys_addr = pci_resource_start(pdev, GTTMMADR_BAR) + gen6_gttadr_offset(i915); + GEM_WARN_ON(pci_resource_len(pdev, GEN4_GTTMMADR_BAR) != gen6_gttmmadr_size(i915)); + phys_addr = pci_resource_start(pdev, GEN4_GTTMMADR_BAR) + gen6_gttadr_offset(i915); /* * On BXT+/ICL+ writes larger than 64 bit to the GTT pagetable range @@ -931,11 +931,11 @@ static int gen8_gmch_probe(struct i915_ggtt *ggtt) unsigned int size; u16 snb_gmch_ctl; - if (!HAS_LMEM(i915)) { - if (!i915_pci_resource_valid(pdev, GTT_APERTURE_BAR)) + if (!HAS_LMEM(i915) && !HAS_LMEMBAR_SMEM_STOLEN(i915)) { + if (!i915_pci_resource_valid(pdev, GEN4_GMADR_BAR)) return -ENXIO; - ggtt->gmadr = pci_resource(pdev, GTT_APERTURE_BAR); + ggtt->gmadr = pci_resource(pdev, GEN4_GMADR_BAR); ggtt->mappable_end = resource_size(&ggtt->gmadr); } @@ -986,7 +986,7 @@ static int gen8_gmch_probe(struct i915_ggtt *ggtt) ggtt->vm.pte_encode = gen8_ggtt_pte_encode; - setup_private_pat(ggtt->vm.gt->uncore); + setup_private_pat(ggtt->vm.gt); return ggtt_probe_common(ggtt, size); } @@ -1089,10 +1089,10 @@ static int gen6_gmch_probe(struct i915_ggtt *ggtt) unsigned int size; u16 snb_gmch_ctl; - if (!i915_pci_resource_valid(pdev, GTT_APERTURE_BAR)) + if (!i915_pci_resource_valid(pdev, GEN4_GMADR_BAR)) return -ENXIO; - ggtt->gmadr = pci_resource(pdev, GTT_APERTURE_BAR); + ggtt->gmadr = pci_resource(pdev, GEN4_GMADR_BAR); ggtt->mappable_end = resource_size(&ggtt->gmadr); /* @@ -1308,7 +1308,7 @@ void i915_ggtt_resume(struct i915_ggtt *ggtt) wbinvd_on_all_cpus(); if (GRAPHICS_VER(ggtt->vm.i915) >= 8) - setup_private_pat(ggtt->vm.gt->uncore); + setup_private_pat(ggtt->vm.gt); intel_ggtt_restore_fences(ggtt); } diff --git a/drivers/gpu/drm/i915/gt/intel_ggtt_fencing.c b/drivers/gpu/drm/i915/gt/intel_ggtt_fencing.c index ea775e601686..995082d45cb2 100644 --- a/drivers/gpu/drm/i915/gt/intel_ggtt_fencing.c +++ b/drivers/gpu/drm/i915/gt/intel_ggtt_fencing.c @@ -816,8 +816,8 @@ i915_gem_object_save_bit_17_swizzle(struct drm_i915_gem_object *obj, if (obj->bit_17 == NULL) { obj->bit_17 = bitmap_zalloc(page_count, GFP_KERNEL); if (obj->bit_17 == NULL) { - DRM_ERROR("Failed to allocate memory for bit 17 " - "record\n"); + drm_err(&to_i915(obj->base.dev)->drm, + "Failed to allocate memory for bit 17 record\n"); return; } } diff --git a/drivers/gpu/drm/i915/gt/intel_gpu_commands.h b/drivers/gpu/drm/i915/gt/intel_gpu_commands.h index d4e9702d3c8e..f50ea92910d9 100644 --- a/drivers/gpu/drm/i915/gt/intel_gpu_commands.h +++ b/drivers/gpu/drm/i915/gt/intel_gpu_commands.h @@ -187,6 +187,10 @@ #define MI_BATCH_RESOURCE_STREAMER REG_BIT(10) #define MI_BATCH_PREDICATE REG_BIT(15) /* HSW+ on RCS only*/ +#define MI_OPCODE(x) (((x) >> 23) & 0x3f) +#define IS_MI_LRI_CMD(x) (MI_OPCODE(x) == MI_OPCODE(MI_INSTR(0x22, 0))) +#define MI_LRI_LEN(x) (((x) & 0xff) + 1) + /* * 3D instructions used by the kernel */ diff --git a/drivers/gpu/drm/i915/gt/intel_gsc.c b/drivers/gpu/drm/i915/gt/intel_gsc.c index 7af6db3194dd..976fdf27e790 100644 --- a/drivers/gpu/drm/i915/gt/intel_gsc.c +++ b/drivers/gpu/drm/i915/gt/intel_gsc.c @@ -7,6 +7,7 @@ #include <linux/mei_aux.h> #include "i915_drv.h" #include "i915_reg.h" +#include "gem/i915_gem_lmem.h" #include "gem/i915_gem_region.h" #include "gt/intel_gsc.h" #include "gt/intel_gt.h" @@ -142,8 +143,14 @@ static void gsc_destroy_one(struct drm_i915_private *i915, struct intel_gsc_intf *intf = &gsc->intf[intf_id]; if (intf->adev) { - auxiliary_device_delete(&intf->adev->aux_dev); - auxiliary_device_uninit(&intf->adev->aux_dev); + struct auxiliary_device *aux_dev = &intf->adev->aux_dev; + + if (intf_id == 0) + intel_huc_unregister_gsc_notifier(&gsc_to_gt(gsc)->uc.huc, + aux_dev->dev.bus); + + auxiliary_device_delete(aux_dev); + auxiliary_device_uninit(aux_dev); intf->adev = NULL; } @@ -242,14 +249,24 @@ add_device: goto fail; } + intf->adev = adev; /* needed by the notifier */ + + if (intf_id == 0) + intel_huc_register_gsc_notifier(&gsc_to_gt(gsc)->uc.huc, + aux_dev->dev.bus); + ret = auxiliary_device_add(aux_dev); if (ret < 0) { drm_err(&i915->drm, "gsc aux add failed %d\n", ret); + if (intf_id == 0) + intel_huc_unregister_gsc_notifier(&gsc_to_gt(gsc)->uc.huc, + aux_dev->dev.bus); + intf->adev = NULL; + /* adev will be freed with the put_device() and .release sequence */ auxiliary_device_uninit(aux_dev); goto fail; } - intf->adev = adev; return; fail: diff --git a/drivers/gpu/drm/i915/gt/intel_gt.c b/drivers/gpu/drm/i915/gt/intel_gt.c index 7caa3412a244..767e329e1cc5 100644 --- a/drivers/gpu/drm/i915/gt/intel_gt.c +++ b/drivers/gpu/drm/i915/gt/intel_gt.c @@ -12,6 +12,7 @@ #include "i915_drv.h" #include "i915_perf_oa_regs.h" +#include "i915_reg.h" #include "intel_context.h" #include "intel_engine_pm.h" #include "intel_engine_regs.h" @@ -40,8 +41,6 @@ void intel_gt_common_init_early(struct intel_gt *gt) { spin_lock_init(gt->irq_lock); - INIT_LIST_HEAD(>->lmem_userfault_list); - mutex_init(>->lmem_userfault_lock); INIT_LIST_HEAD(>->closed_vma); spin_lock_init(>->closed_lock); @@ -56,6 +55,7 @@ void intel_gt_common_init_early(struct intel_gt *gt) seqcount_mutex_init(>->tlb.seqno, >->tlb.invalidate_lock); intel_gt_pm_init_early(gt); + intel_wopcm_init_early(>->wopcm); intel_uc_init_early(>->uc); intel_rps_init_early(>->rps); } @@ -192,7 +192,7 @@ int intel_gt_init_hw(struct intel_gt *gt) ret = i915_ppgtt_init_hw(gt); if (ret) { - DRM_ERROR("Enabling PPGTT failed (%d)\n", ret); + drm_err(&i915->drm, "Enabling PPGTT failed (%d)\n", ret); goto out; } @@ -231,6 +231,16 @@ static void gen6_clear_engine_error_register(struct intel_engine_cs *engine) GEN6_RING_FAULT_REG_POSTING_READ(engine); } +i915_reg_t intel_gt_perf_limit_reasons_reg(struct intel_gt *gt) +{ + /* GT0_PERF_LIMIT_REASONS is available only for Gen11+ */ + if (GRAPHICS_VER(gt->i915) < 11) + return INVALID_MMIO_REG; + + return gt->type == GT_MEDIA ? + MTL_MEDIA_PERF_LIMIT_REASONS : GT0_PERF_LIMIT_REASONS; +} + void intel_gt_clear_error_registers(struct intel_gt *gt, intel_engine_mask_t engine_mask) @@ -254,13 +264,17 @@ intel_gt_clear_error_registers(struct intel_gt *gt, * some errors might have become stuck, * mask them. */ - DRM_DEBUG_DRIVER("EIR stuck: 0x%08x, masking\n", eir); + drm_dbg(>->i915->drm, "EIR stuck: 0x%08x, masking\n", eir); rmw_set(uncore, EMR, eir); intel_uncore_write(uncore, GEN2_IIR, I915_MASTER_ERROR_INTERRUPT); } - if (GRAPHICS_VER(i915) >= 12) { + if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) { + intel_gt_mcr_multicast_rmw(gt, XEHP_RING_FAULT_REG, + RING_FAULT_VALID, 0); + intel_gt_mcr_read_any(gt, XEHP_RING_FAULT_REG); + } else if (GRAPHICS_VER(i915) >= 12) { rmw_clear(uncore, GEN12_RING_FAULT_REG, RING_FAULT_VALID); intel_uncore_posting_read(uncore, GEN12_RING_FAULT_REG); } else if (GRAPHICS_VER(i915) >= 8) { @@ -298,6 +312,42 @@ static void gen6_check_faults(struct intel_gt *gt) } } +static void xehp_check_faults(struct intel_gt *gt) +{ + u32 fault; + + /* + * Although the fault register now lives in an MCR register range, + * the GAM registers are special and we only truly need to read + * the "primary" GAM instance rather than handling each instance + * individually. intel_gt_mcr_read_any() will automatically steer + * toward the primary instance. + */ + fault = intel_gt_mcr_read_any(gt, XEHP_RING_FAULT_REG); + if (fault & RING_FAULT_VALID) { + u32 fault_data0, fault_data1; + u64 fault_addr; + + fault_data0 = intel_gt_mcr_read_any(gt, XEHP_FAULT_TLB_DATA0); + fault_data1 = intel_gt_mcr_read_any(gt, XEHP_FAULT_TLB_DATA1); + + fault_addr = ((u64)(fault_data1 & FAULT_VA_HIGH_BITS) << 44) | + ((u64)fault_data0 << 12); + + drm_dbg(>->i915->drm, "Unexpected fault\n" + "\tAddr: 0x%08x_%08x\n" + "\tAddress space: %s\n" + "\tEngine ID: %d\n" + "\tSource ID: %d\n" + "\tType: %d\n", + upper_32_bits(fault_addr), lower_32_bits(fault_addr), + fault_data1 & FAULT_GTT_SEL ? "GGTT" : "PPGTT", + GEN8_RING_FAULT_ENGINE_ID(fault), + RING_FAULT_SRCID(fault), + RING_FAULT_FAULT_TYPE(fault)); + } +} + static void gen8_check_faults(struct intel_gt *gt) { struct intel_uncore *uncore = gt->uncore; @@ -344,7 +394,9 @@ void intel_gt_check_and_clear_faults(struct intel_gt *gt) struct drm_i915_private *i915 = gt->i915; /* From GEN8 onwards we only have one 'All Engine Fault Register' */ - if (GRAPHICS_VER(i915) >= 8) + if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) + xehp_check_faults(gt); + else if (GRAPHICS_VER(i915) >= 8) gen8_check_faults(gt); else if (GRAPHICS_VER(i915) >= 6) gen6_check_faults(gt); @@ -812,7 +864,6 @@ static int intel_gt_tile_setup(struct intel_gt *gt, phys_addr_t phys_addr) } intel_uncore_init_early(gt->uncore, gt); - intel_wakeref_auto_init(>->userfault_wakeref, gt->uncore->rpm); ret = intel_uncore_setup_mmio(gt->uncore, phys_addr); if (ret) @@ -833,7 +884,7 @@ int intel_gt_probe_all(struct drm_i915_private *i915) unsigned int i; int ret; - mmio_bar = GRAPHICS_VER(i915) == 2 ? GEN2_GTTMMADR_BAR : GTTMMADR_BAR; + mmio_bar = intel_mmio_bar(GRAPHICS_VER(i915)); phys_addr = pci_resource_start(pdev, mmio_bar); /* @@ -944,7 +995,10 @@ void intel_gt_info_print(const struct intel_gt_info *info, } struct reg_and_bit { - i915_reg_t reg; + union { + i915_reg_t reg; + i915_mcr_reg_t mcr_reg; + }; u32 bit; }; @@ -970,6 +1024,32 @@ get_reg_and_bit(const struct intel_engine_cs *engine, const bool gen8, return rb; } +/* + * HW architecture suggest typical invalidation time at 40us, + * with pessimistic cases up to 100us and a recommendation to + * cap at 1ms. We go a bit higher just in case. + */ +#define TLB_INVAL_TIMEOUT_US 100 +#define TLB_INVAL_TIMEOUT_MS 4 + +/* + * On Xe_HP the TLB invalidation registers are located at the same MMIO offsets + * but are now considered MCR registers. Since they exist within a GAM range, + * the primary instance of the register rolls up the status from each unit. + */ +static int wait_for_invalidate(struct intel_gt *gt, struct reg_and_bit rb) +{ + if (GRAPHICS_VER_FULL(gt->i915) >= IP_VER(12, 50)) + return intel_gt_mcr_wait_for_reg(gt, rb.mcr_reg, rb.bit, 0, + TLB_INVAL_TIMEOUT_US, + TLB_INVAL_TIMEOUT_MS); + else + return __intel_wait_for_register_fw(gt->uncore, rb.reg, rb.bit, 0, + TLB_INVAL_TIMEOUT_US, + TLB_INVAL_TIMEOUT_MS, + NULL); +} + static void mmio_invalidate_full(struct intel_gt *gt) { static const i915_reg_t gen8_regs[] = { @@ -985,6 +1065,13 @@ static void mmio_invalidate_full(struct intel_gt *gt) [COPY_ENGINE_CLASS] = GEN12_BLT_TLB_INV_CR, [COMPUTE_CLASS] = GEN12_COMPCTX_TLB_INV_CR, }; + static const i915_mcr_reg_t xehp_regs[] = { + [RENDER_CLASS] = XEHP_GFX_TLB_INV_CR, + [VIDEO_DECODE_CLASS] = XEHP_VD_TLB_INV_CR, + [VIDEO_ENHANCEMENT_CLASS] = XEHP_VE_TLB_INV_CR, + [COPY_ENGINE_CLASS] = XEHP_BLT_TLB_INV_CR, + [COMPUTE_CLASS] = XEHP_COMPCTX_TLB_INV_CR, + }; struct drm_i915_private *i915 = gt->i915; struct intel_uncore *uncore = gt->uncore; struct intel_engine_cs *engine; @@ -993,7 +1080,10 @@ static void mmio_invalidate_full(struct intel_gt *gt) const i915_reg_t *regs; unsigned int num = 0; - if (GRAPHICS_VER(i915) == 12) { + if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) { + regs = NULL; + num = ARRAY_SIZE(xehp_regs); + } else if (GRAPHICS_VER(i915) == 12) { regs = gen12_regs; num = ARRAY_SIZE(gen12_regs); } else if (GRAPHICS_VER(i915) >= 8 && GRAPHICS_VER(i915) <= 11) { @@ -1018,16 +1108,22 @@ static void mmio_invalidate_full(struct intel_gt *gt) if (!intel_engine_pm_is_awake(engine)) continue; - rb = get_reg_and_bit(engine, regs == gen8_regs, regs, num); - if (!i915_mmio_reg_offset(rb.reg)) - continue; - - if (GRAPHICS_VER(i915) == 12 && (engine->class == VIDEO_DECODE_CLASS || - engine->class == VIDEO_ENHANCEMENT_CLASS || - engine->class == COMPUTE_CLASS)) - rb.bit = _MASKED_BIT_ENABLE(rb.bit); - - intel_uncore_write_fw(uncore, rb.reg, rb.bit); + if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) { + intel_gt_mcr_multicast_write_fw(gt, + xehp_regs[engine->class], + BIT(engine->instance)); + } else { + rb = get_reg_and_bit(engine, regs == gen8_regs, regs, num); + if (!i915_mmio_reg_offset(rb.reg)) + continue; + + if (GRAPHICS_VER(i915) == 12 && (engine->class == VIDEO_DECODE_CLASS || + engine->class == VIDEO_ENHANCEMENT_CLASS || + engine->class == COMPUTE_CLASS)) + rb.bit = _MASKED_BIT_ENABLE(rb.bit); + + intel_uncore_write_fw(uncore, rb.reg, rb.bit); + } awake |= engine->mask; } @@ -1047,22 +1143,17 @@ static void mmio_invalidate_full(struct intel_gt *gt) for_each_engine_masked(engine, gt, awake, tmp) { struct reg_and_bit rb; - /* - * HW architecture suggest typical invalidation time at 40us, - * with pessimistic cases up to 100us and a recommendation to - * cap at 1ms. We go a bit higher just in case. - */ - const unsigned int timeout_us = 100; - const unsigned int timeout_ms = 4; - - rb = get_reg_and_bit(engine, regs == gen8_regs, regs, num); - if (__intel_wait_for_register_fw(uncore, - rb.reg, rb.bit, 0, - timeout_us, timeout_ms, - NULL)) + if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) { + rb.mcr_reg = xehp_regs[engine->class]; + rb.bit = BIT(engine->instance); + } else { + rb = get_reg_and_bit(engine, regs == gen8_regs, regs, num); + } + + if (wait_for_invalidate(gt, rb)) drm_err_ratelimited(>->i915->drm, "%s TLB invalidation did not complete in %ums!\n", - engine->name, timeout_ms); + engine->name, TLB_INVAL_TIMEOUT_MS); } /* diff --git a/drivers/gpu/drm/i915/gt/intel_gt.h b/drivers/gpu/drm/i915/gt/intel_gt.h index 2ee582e287c8..e0365d556248 100644 --- a/drivers/gpu/drm/i915/gt/intel_gt.h +++ b/drivers/gpu/drm/i915/gt/intel_gt.h @@ -60,6 +60,7 @@ void intel_gt_driver_late_release_all(struct drm_i915_private *i915); int intel_gt_wait_for_idle(struct intel_gt *gt, long timeout); void intel_gt_check_and_clear_faults(struct intel_gt *gt); +i915_reg_t intel_gt_perf_limit_reasons_reg(struct intel_gt *gt); void intel_gt_clear_error_registers(struct intel_gt *gt, intel_engine_mask_t engine_mask); diff --git a/drivers/gpu/drm/i915/gt/intel_gt_clock_utils.c b/drivers/gpu/drm/i915/gt/intel_gt_clock_utils.c index 3f656d3dba9a..2a6a4ca7fdad 100644 --- a/drivers/gpu/drm/i915/gt/intel_gt_clock_utils.c +++ b/drivers/gpu/drm/i915/gt/intel_gt_clock_utils.c @@ -107,7 +107,7 @@ static u32 gen9_read_clock_frequency(struct intel_uncore *uncore) return freq; } -static u32 gen5_read_clock_frequency(struct intel_uncore *uncore) +static u32 gen6_read_clock_frequency(struct intel_uncore *uncore) { /* * PRMs say: @@ -119,7 +119,27 @@ static u32 gen5_read_clock_frequency(struct intel_uncore *uncore) return 12500000; } -static u32 gen2_read_clock_frequency(struct intel_uncore *uncore) +static u32 gen5_read_clock_frequency(struct intel_uncore *uncore) +{ + /* + * 63:32 increments every 1000 ns + * 31:0 mbz + */ + return 1000000000 / 1000; +} + +static u32 g4x_read_clock_frequency(struct intel_uncore *uncore) +{ + /* + * 63:20 increments every 1/4 ns + * 19:0 mbz + * + * -> 63:32 increments every 1024 ns + */ + return 1000000000 / 1024; +} + +static u32 gen4_read_clock_frequency(struct intel_uncore *uncore) { /* * PRMs say: @@ -127,8 +147,10 @@ static u32 gen2_read_clock_frequency(struct intel_uncore *uncore) * "The value in this register increments once every 16 * hclks." (through the “Clocking Configuration” * (“CLKCFG”) MCHBAR register) + * + * Testing on actual hardware has shown there is no /16. */ - return RUNTIME_INFO(uncore->i915)->rawclk_freq * 1000 / 16; + return RUNTIME_INFO(uncore->i915)->rawclk_freq * 1000; } static u32 read_clock_frequency(struct intel_uncore *uncore) @@ -137,10 +159,16 @@ static u32 read_clock_frequency(struct intel_uncore *uncore) return gen11_read_clock_frequency(uncore); else if (GRAPHICS_VER(uncore->i915) >= 9) return gen9_read_clock_frequency(uncore); - else if (GRAPHICS_VER(uncore->i915) >= 5) + else if (GRAPHICS_VER(uncore->i915) >= 6) + return gen6_read_clock_frequency(uncore); + else if (GRAPHICS_VER(uncore->i915) == 5) return gen5_read_clock_frequency(uncore); + else if (IS_G4X(uncore->i915)) + return g4x_read_clock_frequency(uncore); + else if (GRAPHICS_VER(uncore->i915) == 4) + return gen4_read_clock_frequency(uncore); else - return gen2_read_clock_frequency(uncore); + return 0; } void intel_gt_init_clock_frequency(struct intel_gt *gt) diff --git a/drivers/gpu/drm/i915/gt/intel_gt_irq.c b/drivers/gpu/drm/i915/gt/intel_gt_irq.c index f26882fdc24c..6f6b9e04d916 100644 --- a/drivers/gpu/drm/i915/gt/intel_gt_irq.c +++ b/drivers/gpu/drm/i915/gt/intel_gt_irq.c @@ -17,6 +17,9 @@ static void guc_irq_handler(struct intel_guc *guc, u16 iir) { + if (unlikely(!guc->interrupts.enabled)) + return; + if (iir & GUC_INTR_GUC2HOST) intel_guc_to_host_event_handler(guc); } @@ -44,8 +47,9 @@ gen11_gt_engine_identity(struct intel_gt *gt, !time_after32(local_clock() >> 10, timeout_ts)); if (unlikely(!(ident & GEN11_INTR_DATA_VALID))) { - DRM_ERROR("INTR_IDENTITY_REG%u:%u 0x%08x not valid!\n", - bank, bit, ident); + drm_err(>->i915->drm, + "INTR_IDENTITY_REG%u:%u 0x%08x not valid!\n", + bank, bit, ident); return 0; } @@ -81,35 +85,27 @@ gen11_other_irq_handler(struct intel_gt *gt, const u8 instance, instance, iir); } -static void -gen11_engine_irq_handler(struct intel_gt *gt, const u8 class, - const u8 instance, const u16 iir) +static struct intel_gt *pick_gt(struct intel_gt *gt, u8 class, u8 instance) { - struct intel_engine_cs *engine; - - /* - * Platforms with standalone media have their media engines in another - * GT. - */ - if (MEDIA_VER(gt->i915) >= 13 && - (class == VIDEO_DECODE_CLASS || class == VIDEO_ENHANCEMENT_CLASS)) { - if (!gt->i915->media_gt) - goto err; + struct intel_gt *media_gt = gt->i915->media_gt; - gt = gt->i915->media_gt; + /* we expect the non-media gt to be passed in */ + GEM_BUG_ON(gt == media_gt); + + if (!media_gt) + return gt; + + switch (class) { + case VIDEO_DECODE_CLASS: + case VIDEO_ENHANCEMENT_CLASS: + return media_gt; + case OTHER_CLASS: + if (instance == OTHER_GSC_INSTANCE && HAS_ENGINE(media_gt, GSC0)) + return media_gt; + fallthrough; + default: + return gt; } - - if (instance <= MAX_ENGINE_INSTANCE) - engine = gt->engine_class[class][instance]; - else - engine = NULL; - - if (likely(engine)) - return intel_engine_cs_irq(engine, iir); - -err: - WARN_ONCE(1, "unhandled engine interrupt class=0x%x, instance=0x%x\n", - class, instance); } static void @@ -122,8 +118,17 @@ gen11_gt_identity_handler(struct intel_gt *gt, const u32 identity) if (unlikely(!intr)) return; - if (class <= COPY_ENGINE_CLASS || class == COMPUTE_CLASS) - return gen11_engine_irq_handler(gt, class, instance, intr); + /* + * Platforms with standalone media have the media and GSC engines in + * another GT. + */ + gt = pick_gt(gt, class, instance); + + if (class <= MAX_ENGINE_CLASS && instance <= MAX_ENGINE_INSTANCE) { + struct intel_engine_cs *engine = gt->engine_class[class][instance]; + if (engine) + return intel_engine_cs_irq(engine, intr); + } if (class == OTHER_CLASS) return gen11_other_irq_handler(gt, instance, intr); @@ -206,7 +211,7 @@ void gen11_gt_irq_reset(struct intel_gt *gt) intel_uncore_write(uncore, GEN11_VCS_VECS_INTR_ENABLE, 0); if (CCS_MASK(gt)) intel_uncore_write(uncore, GEN12_CCS_RSVD_INTR_ENABLE, 0); - if (HAS_HECI_GSC(gt->i915)) + if (HAS_HECI_GSC(gt->i915) || HAS_ENGINE(gt, GSC0)) intel_uncore_write(uncore, GEN11_GUNIT_CSME_INTR_ENABLE, 0); /* Restore masks irqs on RCS, BCS, VCS and VECS engines. */ @@ -233,7 +238,7 @@ void gen11_gt_irq_reset(struct intel_gt *gt) intel_uncore_write(uncore, GEN12_CCS0_CCS1_INTR_MASK, ~0); if (HAS_ENGINE(gt, CCS2) || HAS_ENGINE(gt, CCS3)) intel_uncore_write(uncore, GEN12_CCS2_CCS3_INTR_MASK, ~0); - if (HAS_HECI_GSC(gt->i915)) + if (HAS_HECI_GSC(gt->i915) || HAS_ENGINE(gt, GSC0)) intel_uncore_write(uncore, GEN11_GUNIT_CSME_INTR_MASK, ~0); intel_uncore_write(uncore, GEN11_GPM_WGBOXPERF_INTR_ENABLE, 0); @@ -249,7 +254,8 @@ void gen11_gt_irq_postinstall(struct intel_gt *gt) { struct intel_uncore *uncore = gt->uncore; u32 irqs = GT_RENDER_USER_INTERRUPT; - const u32 gsc_mask = GSC_IRQ_INTF(0) | GSC_IRQ_INTF(1); + u32 guc_mask = intel_uc_wants_guc(>->uc) ? GUC_INTR_GUC2HOST : 0; + u32 gsc_mask = 0; u32 dmask; u32 smask; @@ -261,6 +267,11 @@ void gen11_gt_irq_postinstall(struct intel_gt *gt) dmask = irqs << 16 | irqs; smask = irqs << 16; + if (HAS_ENGINE(gt, GSC0)) + gsc_mask = irqs; + else if (HAS_HECI_GSC(gt->i915)) + gsc_mask = GSC_IRQ_INTF(0) | GSC_IRQ_INTF(1); + BUILD_BUG_ON(irqs & 0xffff0000); /* Enable RCS, BCS, VCS and VECS class interrupts. */ @@ -268,9 +279,8 @@ void gen11_gt_irq_postinstall(struct intel_gt *gt) intel_uncore_write(uncore, GEN11_VCS_VECS_INTR_ENABLE, dmask); if (CCS_MASK(gt)) intel_uncore_write(uncore, GEN12_CCS_RSVD_INTR_ENABLE, smask); - if (HAS_HECI_GSC(gt->i915)) - intel_uncore_write(uncore, GEN11_GUNIT_CSME_INTR_ENABLE, - gsc_mask); + if (gsc_mask) + intel_uncore_write(uncore, GEN11_GUNIT_CSME_INTR_ENABLE, gsc_mask); /* Unmask irqs on RCS, BCS, VCS and VECS engines. */ intel_uncore_write(uncore, GEN11_RCS0_RSVD_INTR_MASK, ~smask); @@ -296,9 +306,22 @@ void gen11_gt_irq_postinstall(struct intel_gt *gt) intel_uncore_write(uncore, GEN12_CCS0_CCS1_INTR_MASK, ~dmask); if (HAS_ENGINE(gt, CCS2) || HAS_ENGINE(gt, CCS3)) intel_uncore_write(uncore, GEN12_CCS2_CCS3_INTR_MASK, ~dmask); - if (HAS_HECI_GSC(gt->i915)) + if (gsc_mask) intel_uncore_write(uncore, GEN11_GUNIT_CSME_INTR_MASK, ~gsc_mask); + if (guc_mask) { + /* the enable bit is common for both GTs but the masks are separate */ + u32 mask = gt->type == GT_MEDIA ? + REG_FIELD_PREP(ENGINE0_MASK, guc_mask) : + REG_FIELD_PREP(ENGINE1_MASK, guc_mask); + + intel_uncore_write(uncore, GEN11_GUC_SG_INTR_ENABLE, + REG_FIELD_PREP(ENGINE1_MASK, guc_mask)); + + /* we might not be the first GT to write this reg */ + intel_uncore_rmw(uncore, MTL_GUC_MGUC_INTR_MASK, mask, 0); + } + /* * RPS interrupts will get enabled/disabled on demand when RPS itself * is enabled/disabled. @@ -307,10 +330,6 @@ void gen11_gt_irq_postinstall(struct intel_gt *gt) gt->pm_imr = ~gt->pm_ier; intel_uncore_write(uncore, GEN11_GPM_WGBOXPERF_INTR_ENABLE, 0); intel_uncore_write(uncore, GEN11_GPM_WGBOXPERF_INTR_MASK, ~0); - - /* Same thing for GuC interrupts */ - intel_uncore_write(uncore, GEN11_GUC_SG_INTR_ENABLE, 0); - intel_uncore_write(uncore, GEN11_GUC_SG_INTR_MASK, ~0); } void gen5_gt_irq_handler(struct intel_gt *gt, u32 gt_iir) @@ -359,7 +378,8 @@ void gen6_gt_irq_handler(struct intel_gt *gt, u32 gt_iir) if (gt_iir & (GT_BLT_CS_ERROR_INTERRUPT | GT_BSD_CS_ERROR_INTERRUPT | GT_CS_MASTER_ERROR_INTERRUPT)) - DRM_DEBUG("Command parser error, gt_iir 0x%08x\n", gt_iir); + drm_dbg(>->i915->drm, "Command parser error, gt_iir 0x%08x\n", + gt_iir); if (gt_iir & GT_PARITY_ERROR(gt->i915)) gen7_parity_error_irq_handler(gt, gt_iir); diff --git a/drivers/gpu/drm/i915/gt/intel_gt_mcr.c b/drivers/gpu/drm/i915/gt/intel_gt_mcr.c index e79405a45312..d9a8ff9e5e57 100644 --- a/drivers/gpu/drm/i915/gt/intel_gt_mcr.c +++ b/drivers/gpu/drm/i915/gt/intel_gt_mcr.c @@ -40,6 +40,9 @@ static const char * const intel_steering_types[] = { "L3BANK", "MSLICE", "LNCF", + "GAM", + "DSS", + "OADDRM", "INSTANCE 0", }; @@ -48,14 +51,23 @@ static const struct intel_mmio_range icl_l3bank_steering_table[] = { {}, }; +/* + * Although the bspec lists more "MSLICE" ranges than shown here, some of those + * are of a "GAM" subclass that has special rules. Thus we use a separate + * GAM table farther down for those. + */ static const struct intel_mmio_range xehpsdv_mslice_steering_table[] = { - { 0x004000, 0x004AFF }, - { 0x00C800, 0x00CFFF }, { 0x00DD00, 0x00DDFF }, { 0x00E900, 0x00FFFF }, /* 0xEA00 - OxEFFF is unused */ {}, }; +static const struct intel_mmio_range xehpsdv_gam_steering_table[] = { + { 0x004000, 0x004AFF }, + { 0x00C800, 0x00CFFF }, + {}, +}; + static const struct intel_mmio_range xehpsdv_lncf_steering_table[] = { { 0x00B000, 0x00B0FF }, { 0x00D800, 0x00D8FF }, @@ -89,9 +101,47 @@ static const struct intel_mmio_range pvc_instance0_steering_table[] = { {}, }; +static const struct intel_mmio_range xelpg_instance0_steering_table[] = { + { 0x000B00, 0x000BFF }, /* SQIDI */ + { 0x001000, 0x001FFF }, /* SQIDI */ + { 0x004000, 0x0048FF }, /* GAM */ + { 0x008700, 0x0087FF }, /* SQIDI */ + { 0x00B000, 0x00B0FF }, /* NODE */ + { 0x00C800, 0x00CFFF }, /* GAM */ + { 0x00D880, 0x00D8FF }, /* NODE */ + { 0x00DD00, 0x00DDFF }, /* OAAL2 */ + {}, +}; + +static const struct intel_mmio_range xelpg_l3bank_steering_table[] = { + { 0x00B100, 0x00B3FF }, + {}, +}; + +/* DSS steering is used for SLICE ranges as well */ +static const struct intel_mmio_range xelpg_dss_steering_table[] = { + { 0x005200, 0x0052FF }, /* SLICE */ + { 0x005500, 0x007FFF }, /* SLICE */ + { 0x008140, 0x00815F }, /* SLICE (0x8140-0x814F), DSS (0x8150-0x815F) */ + { 0x0094D0, 0x00955F }, /* SLICE (0x94D0-0x951F), DSS (0x9520-0x955F) */ + { 0x009680, 0x0096FF }, /* DSS */ + { 0x00D800, 0x00D87F }, /* SLICE */ + { 0x00DC00, 0x00DCFF }, /* SLICE */ + { 0x00DE80, 0x00E8FF }, /* DSS (0xE000-0xE0FF reserved) */ + {}, +}; + +static const struct intel_mmio_range xelpmp_oaddrm_steering_table[] = { + { 0x393200, 0x39323F }, + { 0x393400, 0x3934FF }, + {}, +}; + void intel_gt_mcr_init(struct intel_gt *gt) { struct drm_i915_private *i915 = gt->i915; + unsigned long fuse; + int i; /* * An mslice is unavailable only if both the meml3 for the slice is @@ -109,14 +159,36 @@ void intel_gt_mcr_init(struct intel_gt *gt) drm_warn(&i915->drm, "mslice mask all zero!\n"); } - if (IS_PONTEVECCHIO(i915)) { + if (MEDIA_VER(i915) >= 13 && gt->type == GT_MEDIA) { + gt->steering_table[OADDRM] = xelpmp_oaddrm_steering_table; + } else if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 70)) { + fuse = REG_FIELD_GET(GT_L3_EXC_MASK, + intel_uncore_read(gt->uncore, XEHP_FUSE4)); + + /* + * Despite the register field being named "exclude mask" the + * bits actually represent enabled banks (two banks per bit). + */ + for_each_set_bit(i, &fuse, 3) + gt->info.l3bank_mask |= 0x3 << 2 * i; + + gt->steering_table[INSTANCE0] = xelpg_instance0_steering_table; + gt->steering_table[L3BANK] = xelpg_l3bank_steering_table; + gt->steering_table[DSS] = xelpg_dss_steering_table; + } else if (IS_PONTEVECCHIO(i915)) { gt->steering_table[INSTANCE0] = pvc_instance0_steering_table; } else if (IS_DG2(i915)) { gt->steering_table[MSLICE] = xehpsdv_mslice_steering_table; gt->steering_table[LNCF] = dg2_lncf_steering_table; + /* + * No need to hook up the GAM table since it has a dedicated + * steering control register on DG2 and can use implicit + * steering. + */ } else if (IS_XEHPSDV(i915)) { gt->steering_table[MSLICE] = xehpsdv_mslice_steering_table; gt->steering_table[LNCF] = xehpsdv_lncf_steering_table; + gt->steering_table[GAM] = xehpsdv_gam_steering_table; } else if (GRAPHICS_VER(i915) >= 11 && GRAPHICS_VER_FULL(i915) < IP_VER(12, 50)) { gt->steering_table[L3BANK] = icl_l3bank_steering_table; @@ -135,6 +207,19 @@ void intel_gt_mcr_init(struct intel_gt *gt) } /* + * Although the rest of the driver should use MCR-specific functions to + * read/write MCR registers, we still use the regular intel_uncore_* functions + * internally to implement those, so we need a way for the functions in this + * file to "cast" an i915_mcr_reg_t into an i915_reg_t. + */ +static i915_reg_t mcr_reg_cast(const i915_mcr_reg_t mcr) +{ + i915_reg_t r = { .reg = mcr.reg }; + + return r; +} + +/* * rw_with_mcr_steering_fw - Access a register with specific MCR steering * @uncore: pointer to struct intel_uncore * @reg: register being accessed @@ -148,14 +233,26 @@ void intel_gt_mcr_init(struct intel_gt *gt) * Caller needs to make sure the relevant forcewake wells are up. */ static u32 rw_with_mcr_steering_fw(struct intel_uncore *uncore, - i915_reg_t reg, u8 rw_flag, + i915_mcr_reg_t reg, u8 rw_flag, int group, int instance, u32 value) { u32 mcr_mask, mcr_ss, mcr, old_mcr, val = 0; lockdep_assert_held(&uncore->lock); - if (GRAPHICS_VER(uncore->i915) >= 11) { + if (GRAPHICS_VER_FULL(uncore->i915) >= IP_VER(12, 70)) { + /* + * Always leave the hardware in multicast mode when doing reads + * (see comment about Wa_22013088509 below) and only change it + * to unicast mode when doing writes of a specific instance. + * + * No need to save old steering reg value. + */ + intel_uncore_write_fw(uncore, MTL_MCR_SELECTOR, + REG_FIELD_PREP(MTL_MCR_GROUPID, group) | + REG_FIELD_PREP(MTL_MCR_INSTANCEID, instance) | + (rw_flag == FW_REG_READ ? GEN11_MCR_MULTICAST : 0)); + } else if (GRAPHICS_VER(uncore->i915) >= 11) { mcr_mask = GEN11_MCR_SLICE_MASK | GEN11_MCR_SUBSLICE_MASK; mcr_ss = GEN11_MCR_SLICE(group) | GEN11_MCR_SUBSLICE(instance); @@ -173,39 +270,53 @@ static u32 rw_with_mcr_steering_fw(struct intel_uncore *uncore, */ if (rw_flag == FW_REG_WRITE) mcr_mask |= GEN11_MCR_MULTICAST; + + mcr = intel_uncore_read_fw(uncore, GEN8_MCR_SELECTOR); + old_mcr = mcr; + + mcr &= ~mcr_mask; + mcr |= mcr_ss; + intel_uncore_write_fw(uncore, GEN8_MCR_SELECTOR, mcr); } else { mcr_mask = GEN8_MCR_SLICE_MASK | GEN8_MCR_SUBSLICE_MASK; mcr_ss = GEN8_MCR_SLICE(group) | GEN8_MCR_SUBSLICE(instance); - } - old_mcr = mcr = intel_uncore_read_fw(uncore, GEN8_MCR_SELECTOR); + mcr = intel_uncore_read_fw(uncore, GEN8_MCR_SELECTOR); + old_mcr = mcr; - mcr &= ~mcr_mask; - mcr |= mcr_ss; - intel_uncore_write_fw(uncore, GEN8_MCR_SELECTOR, mcr); + mcr &= ~mcr_mask; + mcr |= mcr_ss; + intel_uncore_write_fw(uncore, GEN8_MCR_SELECTOR, mcr); + } if (rw_flag == FW_REG_READ) - val = intel_uncore_read_fw(uncore, reg); + val = intel_uncore_read_fw(uncore, mcr_reg_cast(reg)); else - intel_uncore_write_fw(uncore, reg, value); - - mcr &= ~mcr_mask; - mcr |= old_mcr & mcr_mask; + intel_uncore_write_fw(uncore, mcr_reg_cast(reg), value); - intel_uncore_write_fw(uncore, GEN8_MCR_SELECTOR, mcr); + /* + * For pre-MTL platforms, we need to restore the old value of the + * steering control register to ensure that implicit steering continues + * to behave as expected. For MTL and beyond, we need only reinstate + * the 'multicast' bit (and only if we did a write that cleared it). + */ + if (GRAPHICS_VER_FULL(uncore->i915) >= IP_VER(12, 70) && rw_flag == FW_REG_WRITE) + intel_uncore_write_fw(uncore, MTL_MCR_SELECTOR, GEN11_MCR_MULTICAST); + else if (GRAPHICS_VER_FULL(uncore->i915) < IP_VER(12, 70)) + intel_uncore_write_fw(uncore, GEN8_MCR_SELECTOR, old_mcr); return val; } static u32 rw_with_mcr_steering(struct intel_uncore *uncore, - i915_reg_t reg, u8 rw_flag, + i915_mcr_reg_t reg, u8 rw_flag, int group, int instance, u32 value) { enum forcewake_domains fw_domains; u32 val; - fw_domains = intel_uncore_forcewake_for_reg(uncore, reg, + fw_domains = intel_uncore_forcewake_for_reg(uncore, mcr_reg_cast(reg), rw_flag); fw_domains |= intel_uncore_forcewake_for_reg(uncore, GEN8_MCR_SELECTOR, @@ -233,7 +344,7 @@ static u32 rw_with_mcr_steering(struct intel_uncore *uncore, * group/instance. */ u32 intel_gt_mcr_read(struct intel_gt *gt, - i915_reg_t reg, + i915_mcr_reg_t reg, int group, int instance) { return rw_with_mcr_steering(gt->uncore, reg, FW_REG_READ, group, instance, 0); @@ -250,7 +361,7 @@ u32 intel_gt_mcr_read(struct intel_gt *gt, * Write an MCR register in unicast mode after steering toward a specific * group/instance. */ -void intel_gt_mcr_unicast_write(struct intel_gt *gt, i915_reg_t reg, u32 value, +void intel_gt_mcr_unicast_write(struct intel_gt *gt, i915_mcr_reg_t reg, u32 value, int group, int instance) { rw_with_mcr_steering(gt->uncore, reg, FW_REG_WRITE, group, instance, value); @@ -265,9 +376,16 @@ void intel_gt_mcr_unicast_write(struct intel_gt *gt, i915_reg_t reg, u32 value, * Write an MCR register in multicast mode to update all instances. */ void intel_gt_mcr_multicast_write(struct intel_gt *gt, - i915_reg_t reg, u32 value) + i915_mcr_reg_t reg, u32 value) { - intel_uncore_write(gt->uncore, reg, value); + /* + * Ensure we have multicast behavior, just in case some non-i915 agent + * left the hardware in unicast mode. + */ + if (GRAPHICS_VER_FULL(gt->i915) >= IP_VER(12, 70)) + intel_uncore_write_fw(gt->uncore, MTL_MCR_SELECTOR, GEN11_MCR_MULTICAST); + + intel_uncore_write(gt->uncore, mcr_reg_cast(reg), value); } /** @@ -281,9 +399,44 @@ void intel_gt_mcr_multicast_write(struct intel_gt *gt, * domains; use intel_gt_mcr_multicast_write() in cases where forcewake should * be obtained automatically. */ -void intel_gt_mcr_multicast_write_fw(struct intel_gt *gt, i915_reg_t reg, u32 value) +void intel_gt_mcr_multicast_write_fw(struct intel_gt *gt, i915_mcr_reg_t reg, u32 value) { - intel_uncore_write_fw(gt->uncore, reg, value); + /* + * Ensure we have multicast behavior, just in case some non-i915 agent + * left the hardware in unicast mode. + */ + if (GRAPHICS_VER_FULL(gt->i915) >= IP_VER(12, 70)) + intel_uncore_write_fw(gt->uncore, MTL_MCR_SELECTOR, GEN11_MCR_MULTICAST); + + intel_uncore_write_fw(gt->uncore, mcr_reg_cast(reg), value); +} + +/** + * intel_gt_mcr_multicast_rmw - Performs a multicast RMW operations + * @gt: GT structure + * @reg: the MCR register to read and write + * @clear: bits to clear during RMW + * @set: bits to set during RMW + * + * Performs a read-modify-write on an MCR register in a multicast manner. + * This operation only makes sense on MCR registers where all instances are + * expected to have the same value. The read will target any non-terminated + * instance and the write will be applied to all instances. + * + * This function assumes the caller is already holding any necessary forcewake + * domains; use intel_gt_mcr_multicast_rmw() in cases where forcewake should + * be obtained automatically. + * + * Returns the old (unmodified) value read. + */ +u32 intel_gt_mcr_multicast_rmw(struct intel_gt *gt, i915_mcr_reg_t reg, + u32 clear, u32 set) +{ + u32 val = intel_gt_mcr_read_any(gt, reg); + + intel_gt_mcr_multicast_write(gt, reg, (val & ~clear) | set); + + return val; } /* @@ -301,7 +454,7 @@ void intel_gt_mcr_multicast_write_fw(struct intel_gt *gt, i915_reg_t reg, u32 va * for @type steering too. */ static bool reg_needs_read_steering(struct intel_gt *gt, - i915_reg_t reg, + i915_mcr_reg_t reg, enum intel_steering_type type) { const u32 offset = i915_mmio_reg_offset(reg); @@ -332,6 +485,8 @@ static void get_nonterminated_steering(struct intel_gt *gt, enum intel_steering_type type, u8 *group, u8 *instance) { + u32 dss; + switch (type) { case L3BANK: *group = 0; /* unused */ @@ -351,6 +506,15 @@ static void get_nonterminated_steering(struct intel_gt *gt, *group = __ffs(gt->info.mslice_mask) << 1; *instance = 0; /* unused */ break; + case GAM: + *group = IS_DG2(gt->i915) ? 1 : 0; + *instance = 0; + break; + case DSS: + dss = intel_sseu_find_first_xehp_dss(>->info.sseu, 0, 0); + *group = dss / GEN_DSS_PER_GSLICE; + *instance = dss % GEN_DSS_PER_GSLICE; + break; case INSTANCE0: /* * There are a lot of MCR types for which instance (0, 0) @@ -359,6 +523,13 @@ static void get_nonterminated_steering(struct intel_gt *gt, *group = 0; *instance = 0; break; + case OADDRM: + if ((VDBOX_MASK(gt) | VEBOX_MASK(gt) | gt->info.sfc_mask) & BIT(0)) + *group = 0; + else + *group = 1; + *instance = 0; + break; default: MISSING_CASE(type); *group = 0; @@ -380,7 +551,7 @@ static void get_nonterminated_steering(struct intel_gt *gt, * steering. */ void intel_gt_mcr_get_nonterminated_steering(struct intel_gt *gt, - i915_reg_t reg, + i915_mcr_reg_t reg, u8 *group, u8 *instance) { int type; @@ -409,7 +580,7 @@ void intel_gt_mcr_get_nonterminated_steering(struct intel_gt *gt, * * Returns the value from a non-terminated instance of @reg. */ -u32 intel_gt_mcr_read_any_fw(struct intel_gt *gt, i915_reg_t reg) +u32 intel_gt_mcr_read_any_fw(struct intel_gt *gt, i915_mcr_reg_t reg) { int type; u8 group, instance; @@ -423,7 +594,7 @@ u32 intel_gt_mcr_read_any_fw(struct intel_gt *gt, i915_reg_t reg) } } - return intel_uncore_read_fw(gt->uncore, reg); + return intel_uncore_read_fw(gt->uncore, mcr_reg_cast(reg)); } /** @@ -436,7 +607,7 @@ u32 intel_gt_mcr_read_any_fw(struct intel_gt *gt, i915_reg_t reg) * * Returns the value from a non-terminated instance of @reg. */ -u32 intel_gt_mcr_read_any(struct intel_gt *gt, i915_reg_t reg) +u32 intel_gt_mcr_read_any(struct intel_gt *gt, i915_mcr_reg_t reg) { int type; u8 group, instance; @@ -450,7 +621,7 @@ u32 intel_gt_mcr_read_any(struct intel_gt *gt, i915_reg_t reg) } } - return intel_uncore_read(gt->uncore, reg); + return intel_uncore_read(gt->uncore, mcr_reg_cast(reg)); } static void report_steering_type(struct drm_printer *p, @@ -483,11 +654,20 @@ static void report_steering_type(struct drm_printer *p, void intel_gt_mcr_report_steering(struct drm_printer *p, struct intel_gt *gt, bool dump_table) { - drm_printf(p, "Default steering: group=0x%x, instance=0x%x\n", - gt->default_steering.groupid, - gt->default_steering.instanceid); - - if (IS_PONTEVECCHIO(gt->i915)) { + /* + * Starting with MTL we no longer have default steering; + * all ranges are explicitly steered. + */ + if (GRAPHICS_VER_FULL(gt->i915) < IP_VER(12, 70)) + drm_printf(p, "Default steering: group=0x%x, instance=0x%x\n", + gt->default_steering.groupid, + gt->default_steering.instanceid); + + if (GRAPHICS_VER_FULL(gt->i915) >= IP_VER(12, 70)) { + for (int i = 0; i < NUM_STEERING_TYPES; i++) + if (gt->steering_table[i]) + report_steering_type(p, gt, i, dump_table); + } else if (IS_PONTEVECCHIO(gt->i915)) { report_steering_type(p, gt, INSTANCE0, dump_table); } else if (HAS_MSLICE_STEERING(gt->i915)) { report_steering_type(p, gt, MSLICE, dump_table); @@ -520,3 +700,60 @@ void intel_gt_mcr_get_ss_steering(struct intel_gt *gt, unsigned int dss, return; } } + +/** + * intel_gt_mcr_wait_for_reg_fw - wait until MCR register matches expected state + * @gt: GT structure + * @reg: the register to read + * @mask: mask to apply to register value + * @value: value to wait for + * @fast_timeout_us: fast timeout in microsecond for atomic/tight wait + * @slow_timeout_ms: slow timeout in millisecond + * + * This routine waits until the target register @reg contains the expected + * @value after applying the @mask, i.e. it waits until :: + * + * (intel_gt_mcr_read_any_fw(gt, reg) & mask) == value + * + * Otherwise, the wait will timeout after @slow_timeout_ms milliseconds. + * For atomic context @slow_timeout_ms must be zero and @fast_timeout_us + * must be not larger than 20,0000 microseconds. + * + * This function is basically an MCR-friendly version of + * __intel_wait_for_register_fw(). Generally this function will only be used + * on GAM registers which are a bit special --- although they're MCR registers, + * reads (e.g., waiting for status updates) are always directed to the primary + * instance. + * + * Note that this routine assumes the caller holds forcewake asserted, it is + * not suitable for very long waits. + * + * Return: 0 if the register matches the desired condition, or -ETIMEDOUT. + */ +int intel_gt_mcr_wait_for_reg(struct intel_gt *gt, + i915_mcr_reg_t reg, + u32 mask, + u32 value, + unsigned int fast_timeout_us, + unsigned int slow_timeout_ms) +{ + int ret; + + lockdep_assert_not_held(>->uncore->lock); + +#define done ((intel_gt_mcr_read_any(gt, reg) & mask) == value) + + /* Catch any overuse of this function */ + might_sleep_if(slow_timeout_ms); + GEM_BUG_ON(fast_timeout_us > 20000); + GEM_BUG_ON(!fast_timeout_us && !slow_timeout_ms); + + ret = -ETIMEDOUT; + if (fast_timeout_us && fast_timeout_us <= 20000) + ret = _wait_for_atomic(done, fast_timeout_us, 0); + if (ret && slow_timeout_ms) + ret = wait_for(done, slow_timeout_ms); + + return ret; +#undef done +} diff --git a/drivers/gpu/drm/i915/gt/intel_gt_mcr.h b/drivers/gpu/drm/i915/gt/intel_gt_mcr.h index 77a8b11c287d..ae93b20e1c17 100644 --- a/drivers/gpu/drm/i915/gt/intel_gt_mcr.h +++ b/drivers/gpu/drm/i915/gt/intel_gt_mcr.h @@ -11,21 +11,24 @@ void intel_gt_mcr_init(struct intel_gt *gt); u32 intel_gt_mcr_read(struct intel_gt *gt, - i915_reg_t reg, + i915_mcr_reg_t reg, int group, int instance); -u32 intel_gt_mcr_read_any_fw(struct intel_gt *gt, i915_reg_t reg); -u32 intel_gt_mcr_read_any(struct intel_gt *gt, i915_reg_t reg); +u32 intel_gt_mcr_read_any_fw(struct intel_gt *gt, i915_mcr_reg_t reg); +u32 intel_gt_mcr_read_any(struct intel_gt *gt, i915_mcr_reg_t reg); void intel_gt_mcr_unicast_write(struct intel_gt *gt, - i915_reg_t reg, u32 value, + i915_mcr_reg_t reg, u32 value, int group, int instance); void intel_gt_mcr_multicast_write(struct intel_gt *gt, - i915_reg_t reg, u32 value); + i915_mcr_reg_t reg, u32 value); void intel_gt_mcr_multicast_write_fw(struct intel_gt *gt, - i915_reg_t reg, u32 value); + i915_mcr_reg_t reg, u32 value); + +u32 intel_gt_mcr_multicast_rmw(struct intel_gt *gt, i915_mcr_reg_t reg, + u32 clear, u32 set); void intel_gt_mcr_get_nonterminated_steering(struct intel_gt *gt, - i915_reg_t reg, + i915_mcr_reg_t reg, u8 *group, u8 *instance); void intel_gt_mcr_report_steering(struct drm_printer *p, struct intel_gt *gt, @@ -34,6 +37,13 @@ void intel_gt_mcr_report_steering(struct drm_printer *p, struct intel_gt *gt, void intel_gt_mcr_get_ss_steering(struct intel_gt *gt, unsigned int dss, unsigned int *group, unsigned int *instance); +int intel_gt_mcr_wait_for_reg(struct intel_gt *gt, + i915_mcr_reg_t reg, + u32 mask, + u32 value, + unsigned int fast_timeout_us, + unsigned int slow_timeout_ms); + /* * Helper for for_each_ss_steering loop. On pre-Xe_HP platforms, subslice * presence is determined by using the group/instance as direct lookups in the diff --git a/drivers/gpu/drm/i915/gt/intel_gt_pm.c b/drivers/gpu/drm/i915/gt/intel_gt_pm.c index f553e2173bda..16db85fab0b1 100644 --- a/drivers/gpu/drm/i915/gt/intel_gt_pm.c +++ b/drivers/gpu/drm/i915/gt/intel_gt_pm.c @@ -7,6 +7,7 @@ #include <linux/suspend.h> #include "i915_drv.h" +#include "i915_irq.h" #include "i915_params.h" #include "intel_context.h" #include "intel_engine_pm.h" @@ -19,10 +20,31 @@ #include "intel_rc6.h" #include "intel_rps.h" #include "intel_wakeref.h" +#include "intel_pcode.h" #include "pxp/intel_pxp_pm.h" #define I915_GT_SUSPEND_IDLE_TIMEOUT (HZ / 2) +static void mtl_media_busy(struct intel_gt *gt) +{ + /* Wa_14017073508: mtl */ + if (IS_MTL_GRAPHICS_STEP(gt->i915, P, STEP_A0, STEP_B0) && + gt->type == GT_MEDIA) + snb_pcode_write_p(gt->uncore, PCODE_MBOX_GT_STATE, + PCODE_MBOX_GT_STATE_MEDIA_BUSY, + PCODE_MBOX_GT_STATE_DOMAIN_MEDIA, 0); +} + +static void mtl_media_idle(struct intel_gt *gt) +{ + /* Wa_14017073508: mtl */ + if (IS_MTL_GRAPHICS_STEP(gt->i915, P, STEP_A0, STEP_B0) && + gt->type == GT_MEDIA) + snb_pcode_write_p(gt->uncore, PCODE_MBOX_GT_STATE, + PCODE_MBOX_GT_STATE_MEDIA_NOT_BUSY, + PCODE_MBOX_GT_STATE_DOMAIN_MEDIA, 0); +} + static void user_forcewake(struct intel_gt *gt, bool suspend) { int count = atomic_read(>->user_wakeref); @@ -70,6 +92,9 @@ static int __gt_unpark(struct intel_wakeref *wf) GT_TRACE(gt, "\n"); + /* Wa_14017073508: mtl */ + mtl_media_busy(gt); + /* * It seems that the DMC likes to transition between the DC states a lot * when there are no connected displays (no active power domains) during @@ -119,6 +144,9 @@ static int __gt_park(struct intel_wakeref *wf) GEM_BUG_ON(!wakeref); intel_display_power_put_async(i915, POWER_DOMAIN_GT_IRQ, wakeref); + /* Wa_14017073508: mtl */ + mtl_media_idle(gt); + return 0; } diff --git a/drivers/gpu/drm/i915/gt/intel_gt_pm_debugfs.c b/drivers/gpu/drm/i915/gt/intel_gt_pm_debugfs.c index 108b9e76c32e..83df4cd5e06c 100644 --- a/drivers/gpu/drm/i915/gt/intel_gt_pm_debugfs.c +++ b/drivers/gpu/drm/i915/gt/intel_gt_pm_debugfs.c @@ -83,19 +83,6 @@ static int fw_domains_show(struct seq_file *m, void *data) } DEFINE_INTEL_GT_DEBUGFS_ATTRIBUTE(fw_domains); -static void print_rc6_res(struct seq_file *m, - const char *title, - const i915_reg_t reg) -{ - struct intel_gt *gt = m->private; - intel_wakeref_t wakeref; - - with_intel_runtime_pm(gt->uncore->rpm, wakeref) - seq_printf(m, "%s %u (%llu us)\n", title, - intel_uncore_read(gt->uncore, reg), - intel_rc6_residency_us(>->rc6, reg)); -} - static int vlv_drpc(struct seq_file *m) { struct intel_gt *gt = m->private; @@ -115,8 +102,8 @@ static int vlv_drpc(struct seq_file *m) seq_printf(m, "Media Power Well: %s\n", (pw_status & VLV_GTLC_PW_MEDIA_STATUS_MASK) ? "Up" : "Down"); - print_rc6_res(m, "Render RC6 residency since boot:", GEN6_GT_GFX_RC6); - print_rc6_res(m, "Media RC6 residency since boot:", VLV_GT_MEDIA_RC6); + intel_rc6_print_residency(m, "Render RC6 residency since boot:", INTEL_RC6_RES_RC6); + intel_rc6_print_residency(m, "Media RC6 residency since boot:", INTEL_RC6_RES_VLV_MEDIA); return fw_domains_show(m, NULL); } @@ -192,11 +179,11 @@ static int gen6_drpc(struct seq_file *m) } /* Not exactly sure what this is */ - print_rc6_res(m, "RC6 \"Locked to RPn\" residency since boot:", - GEN6_GT_GFX_RC6_LOCKED); - print_rc6_res(m, "RC6 residency since boot:", GEN6_GT_GFX_RC6); - print_rc6_res(m, "RC6+ residency since boot:", GEN6_GT_GFX_RC6p); - print_rc6_res(m, "RC6++ residency since boot:", GEN6_GT_GFX_RC6pp); + intel_rc6_print_residency(m, "RC6 \"Locked to RPn\" residency since boot:", + INTEL_RC6_RES_RC6_LOCKED); + intel_rc6_print_residency(m, "RC6 residency since boot:", INTEL_RC6_RES_RC6); + intel_rc6_print_residency(m, "RC6+ residency since boot:", INTEL_RC6_RES_RC6p); + intel_rc6_print_residency(m, "RC6++ residency since boot:", INTEL_RC6_RES_RC6pp); if (GRAPHICS_VER(i915) <= 7) { seq_printf(m, "RC6 voltage: %dmV\n", @@ -269,6 +256,61 @@ static int ilk_drpc(struct seq_file *m) return 0; } +static int mtl_drpc(struct seq_file *m) +{ + struct intel_gt *gt = m->private; + struct intel_uncore *uncore = gt->uncore; + u32 gt_core_status, rcctl1, mt_fwake_req; + u32 mtl_powergate_enable = 0, mtl_powergate_status = 0; + + mt_fwake_req = intel_uncore_read_fw(uncore, FORCEWAKE_MT); + gt_core_status = intel_uncore_read(uncore, MTL_MIRROR_TARGET_WP1); + + rcctl1 = intel_uncore_read(uncore, GEN6_RC_CONTROL); + mtl_powergate_enable = intel_uncore_read(uncore, GEN9_PG_ENABLE); + mtl_powergate_status = intel_uncore_read(uncore, + GEN9_PWRGT_DOMAIN_STATUS); + + seq_printf(m, "RC6 Enabled: %s\n", + str_yes_no(rcctl1 & GEN6_RC_CTL_RC6_ENABLE)); + if (gt->type == GT_MEDIA) { + seq_printf(m, "Media Well Gating Enabled: %s\n", + str_yes_no(mtl_powergate_enable & GEN9_MEDIA_PG_ENABLE)); + } else { + seq_printf(m, "Render Well Gating Enabled: %s\n", + str_yes_no(mtl_powergate_enable & GEN9_RENDER_PG_ENABLE)); + } + + seq_puts(m, "Current RC state: "); + switch (REG_FIELD_GET(MTL_CC_MASK, gt_core_status)) { + case MTL_CC0: + seq_puts(m, "RC0\n"); + break; + case MTL_CC6: + seq_puts(m, "RC6\n"); + break; + default: + MISSING_CASE(REG_FIELD_GET(MTL_CC_MASK, gt_core_status)); + seq_puts(m, "Unknown\n"); + break; + } + + seq_printf(m, "Multi-threaded Forcewake Request: 0x%x\n", mt_fwake_req); + if (gt->type == GT_MEDIA) + seq_printf(m, "Media Power Well: %s\n", + (mtl_powergate_status & + GEN9_PWRGT_MEDIA_STATUS_MASK) ? "Up" : "Down"); + else + seq_printf(m, "Render Power Well: %s\n", + (mtl_powergate_status & + GEN9_PWRGT_RENDER_STATUS_MASK) ? "Up" : "Down"); + + /* Works for both render and media gt's */ + intel_rc6_print_residency(m, "RC6 residency since boot:", INTEL_RC6_RES_RC6); + + return fw_domains_show(m, NULL); +} + static int drpc_show(struct seq_file *m, void *unused) { struct intel_gt *gt = m->private; @@ -277,7 +319,9 @@ static int drpc_show(struct seq_file *m, void *unused) int err = -ENODEV; with_intel_runtime_pm(gt->uncore->rpm, wakeref) { - if (IS_VALLEYVIEW(i915) || IS_CHERRYVIEW(i915)) + if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 70)) + err = mtl_drpc(m); + else if (IS_VALLEYVIEW(i915) || IS_CHERRYVIEW(i915)) err = vlv_drpc(m); else if (GRAPHICS_VER(i915) >= 6) err = gen6_drpc(m); @@ -307,7 +351,7 @@ void intel_gt_pm_frequency_dump(struct intel_gt *gt, struct drm_printer *p) drm_printf(p, "Current VID: %d\n", (rgvstat & MEMSTAT_VID_MASK) >> MEMSTAT_VID_SHIFT); drm_printf(p, "Current P-state: %d\n", - (rgvstat & MEMSTAT_PSTATE_MASK) >> MEMSTAT_PSTATE_SHIFT); + REG_FIELD_GET(MEMSTAT_PSTATE_MASK, rgvstat)); } else if (IS_VALLEYVIEW(i915) || IS_CHERRYVIEW(i915)) { u32 rpmodectl, freq_sts; @@ -344,162 +388,7 @@ void intel_gt_pm_frequency_dump(struct intel_gt *gt, struct drm_printer *p) drm_printf(p, "efficient (RPe) frequency: %d MHz\n", intel_gpu_freq(rps, rps->efficient_freq)); } else if (GRAPHICS_VER(i915) >= 6) { - u32 rp_state_limits; - u32 gt_perf_status; - struct intel_rps_freq_caps caps; - u32 rpmodectl, rpinclimit, rpdeclimit; - u32 rpstat, cagf, reqf; - u32 rpcurupei, rpcurup, rpprevup; - u32 rpcurdownei, rpcurdown, rpprevdown; - u32 rpupei, rpupt, rpdownei, rpdownt; - u32 pm_ier, pm_imr, pm_isr, pm_iir, pm_mask; - - rp_state_limits = intel_uncore_read(uncore, GEN6_RP_STATE_LIMITS); - gen6_rps_get_freq_caps(rps, &caps); - if (IS_GEN9_LP(i915)) - gt_perf_status = intel_uncore_read(uncore, BXT_GT_PERF_STATUS); - else - gt_perf_status = intel_uncore_read(uncore, GEN6_GT_PERF_STATUS); - - /* RPSTAT1 is in the GT power well */ - intel_uncore_forcewake_get(uncore, FORCEWAKE_ALL); - - reqf = intel_uncore_read(uncore, GEN6_RPNSWREQ); - if (GRAPHICS_VER(i915) >= 9) { - reqf >>= 23; - } else { - reqf &= ~GEN6_TURBO_DISABLE; - if (IS_HASWELL(i915) || IS_BROADWELL(i915)) - reqf >>= 24; - else - reqf >>= 25; - } - reqf = intel_gpu_freq(rps, reqf); - - rpmodectl = intel_uncore_read(uncore, GEN6_RP_CONTROL); - rpinclimit = intel_uncore_read(uncore, GEN6_RP_UP_THRESHOLD); - rpdeclimit = intel_uncore_read(uncore, GEN6_RP_DOWN_THRESHOLD); - - rpstat = intel_uncore_read(uncore, GEN6_RPSTAT1); - rpcurupei = intel_uncore_read(uncore, GEN6_RP_CUR_UP_EI) & GEN6_CURICONT_MASK; - rpcurup = intel_uncore_read(uncore, GEN6_RP_CUR_UP) & GEN6_CURBSYTAVG_MASK; - rpprevup = intel_uncore_read(uncore, GEN6_RP_PREV_UP) & GEN6_CURBSYTAVG_MASK; - rpcurdownei = intel_uncore_read(uncore, GEN6_RP_CUR_DOWN_EI) & GEN6_CURIAVG_MASK; - rpcurdown = intel_uncore_read(uncore, GEN6_RP_CUR_DOWN) & GEN6_CURBSYTAVG_MASK; - rpprevdown = intel_uncore_read(uncore, GEN6_RP_PREV_DOWN) & GEN6_CURBSYTAVG_MASK; - - rpupei = intel_uncore_read(uncore, GEN6_RP_UP_EI); - rpupt = intel_uncore_read(uncore, GEN6_RP_UP_THRESHOLD); - - rpdownei = intel_uncore_read(uncore, GEN6_RP_DOWN_EI); - rpdownt = intel_uncore_read(uncore, GEN6_RP_DOWN_THRESHOLD); - - cagf = intel_rps_read_actual_frequency(rps); - - intel_uncore_forcewake_put(uncore, FORCEWAKE_ALL); - - if (GRAPHICS_VER(i915) >= 11) { - pm_ier = intel_uncore_read(uncore, GEN11_GPM_WGBOXPERF_INTR_ENABLE); - pm_imr = intel_uncore_read(uncore, GEN11_GPM_WGBOXPERF_INTR_MASK); - /* - * The equivalent to the PM ISR & IIR cannot be read - * without affecting the current state of the system - */ - pm_isr = 0; - pm_iir = 0; - } else if (GRAPHICS_VER(i915) >= 8) { - pm_ier = intel_uncore_read(uncore, GEN8_GT_IER(2)); - pm_imr = intel_uncore_read(uncore, GEN8_GT_IMR(2)); - pm_isr = intel_uncore_read(uncore, GEN8_GT_ISR(2)); - pm_iir = intel_uncore_read(uncore, GEN8_GT_IIR(2)); - } else { - pm_ier = intel_uncore_read(uncore, GEN6_PMIER); - pm_imr = intel_uncore_read(uncore, GEN6_PMIMR); - pm_isr = intel_uncore_read(uncore, GEN6_PMISR); - pm_iir = intel_uncore_read(uncore, GEN6_PMIIR); - } - pm_mask = intel_uncore_read(uncore, GEN6_PMINTRMSK); - - drm_printf(p, "Video Turbo Mode: %s\n", - str_yes_no(rpmodectl & GEN6_RP_MEDIA_TURBO)); - drm_printf(p, "HW control enabled: %s\n", - str_yes_no(rpmodectl & GEN6_RP_ENABLE)); - drm_printf(p, "SW control enabled: %s\n", - str_yes_no((rpmodectl & GEN6_RP_MEDIA_MODE_MASK) == GEN6_RP_MEDIA_SW_MODE)); - - drm_printf(p, "PM IER=0x%08x IMR=0x%08x, MASK=0x%08x\n", - pm_ier, pm_imr, pm_mask); - if (GRAPHICS_VER(i915) <= 10) - drm_printf(p, "PM ISR=0x%08x IIR=0x%08x\n", - pm_isr, pm_iir); - drm_printf(p, "pm_intrmsk_mbz: 0x%08x\n", - rps->pm_intrmsk_mbz); - drm_printf(p, "GT_PERF_STATUS: 0x%08x\n", gt_perf_status); - drm_printf(p, "Render p-state ratio: %d\n", - (gt_perf_status & (GRAPHICS_VER(i915) >= 9 ? 0x1ff00 : 0xff00)) >> 8); - drm_printf(p, "Render p-state VID: %d\n", - gt_perf_status & 0xff); - drm_printf(p, "Render p-state limit: %d\n", - rp_state_limits & 0xff); - drm_printf(p, "RPSTAT1: 0x%08x\n", rpstat); - drm_printf(p, "RPMODECTL: 0x%08x\n", rpmodectl); - drm_printf(p, "RPINCLIMIT: 0x%08x\n", rpinclimit); - drm_printf(p, "RPDECLIMIT: 0x%08x\n", rpdeclimit); - drm_printf(p, "RPNSWREQ: %dMHz\n", reqf); - drm_printf(p, "CAGF: %dMHz\n", cagf); - drm_printf(p, "RP CUR UP EI: %d (%lldns)\n", - rpcurupei, - intel_gt_pm_interval_to_ns(gt, rpcurupei)); - drm_printf(p, "RP CUR UP: %d (%lldns)\n", - rpcurup, intel_gt_pm_interval_to_ns(gt, rpcurup)); - drm_printf(p, "RP PREV UP: %d (%lldns)\n", - rpprevup, intel_gt_pm_interval_to_ns(gt, rpprevup)); - drm_printf(p, "Up threshold: %d%%\n", - rps->power.up_threshold); - drm_printf(p, "RP UP EI: %d (%lldns)\n", - rpupei, intel_gt_pm_interval_to_ns(gt, rpupei)); - drm_printf(p, "RP UP THRESHOLD: %d (%lldns)\n", - rpupt, intel_gt_pm_interval_to_ns(gt, rpupt)); - - drm_printf(p, "RP CUR DOWN EI: %d (%lldns)\n", - rpcurdownei, - intel_gt_pm_interval_to_ns(gt, rpcurdownei)); - drm_printf(p, "RP CUR DOWN: %d (%lldns)\n", - rpcurdown, - intel_gt_pm_interval_to_ns(gt, rpcurdown)); - drm_printf(p, "RP PREV DOWN: %d (%lldns)\n", - rpprevdown, - intel_gt_pm_interval_to_ns(gt, rpprevdown)); - drm_printf(p, "Down threshold: %d%%\n", - rps->power.down_threshold); - drm_printf(p, "RP DOWN EI: %d (%lldns)\n", - rpdownei, intel_gt_pm_interval_to_ns(gt, rpdownei)); - drm_printf(p, "RP DOWN THRESHOLD: %d (%lldns)\n", - rpdownt, intel_gt_pm_interval_to_ns(gt, rpdownt)); - - drm_printf(p, "Lowest (RPN) frequency: %dMHz\n", - intel_gpu_freq(rps, caps.min_freq)); - drm_printf(p, "Nominal (RP1) frequency: %dMHz\n", - intel_gpu_freq(rps, caps.rp1_freq)); - drm_printf(p, "Max non-overclocked (RP0) frequency: %dMHz\n", - intel_gpu_freq(rps, caps.rp0_freq)); - drm_printf(p, "Max overclocked frequency: %dMHz\n", - intel_gpu_freq(rps, rps->max_freq)); - - drm_printf(p, "Current freq: %d MHz\n", - intel_gpu_freq(rps, rps->cur_freq)); - drm_printf(p, "Actual freq: %d MHz\n", cagf); - drm_printf(p, "Idle freq: %d MHz\n", - intel_gpu_freq(rps, rps->idle_freq)); - drm_printf(p, "Min freq: %d MHz\n", - intel_gpu_freq(rps, rps->min_freq)); - drm_printf(p, "Boost freq: %d MHz\n", - intel_gpu_freq(rps, rps->boost_freq)); - drm_printf(p, "Max freq: %d MHz\n", - intel_gpu_freq(rps, rps->max_freq)); - drm_printf(p, - "efficient (RPe) frequency: %d MHz\n", - intel_gpu_freq(rps, rps->efficient_freq)); + gen6_rps_frequency_dump(rps, p); } else { drm_puts(p, "no P-state info available\n"); } @@ -655,6 +544,44 @@ static bool rps_eval(void *data) DEFINE_INTEL_GT_DEBUGFS_ATTRIBUTE(rps_boost); +static int perf_limit_reasons_get(void *data, u64 *val) +{ + struct intel_gt *gt = data; + intel_wakeref_t wakeref; + + with_intel_runtime_pm(gt->uncore->rpm, wakeref) + *val = intel_uncore_read(gt->uncore, intel_gt_perf_limit_reasons_reg(gt)); + + return 0; +} + +static int perf_limit_reasons_clear(void *data, u64 val) +{ + struct intel_gt *gt = data; + intel_wakeref_t wakeref; + + /* + * Clear the upper 16 "log" bits, the lower 16 "status" bits are + * read-only. The upper 16 "log" bits are identical to the lower 16 + * "status" bits except that the "log" bits remain set until cleared. + */ + with_intel_runtime_pm(gt->uncore->rpm, wakeref) + intel_uncore_rmw(gt->uncore, intel_gt_perf_limit_reasons_reg(gt), + GT0_PERF_LIMIT_REASONS_LOG_MASK, 0); + + return 0; +} + +static bool perf_limit_reasons_eval(void *data) +{ + struct intel_gt *gt = data; + + return i915_mmio_reg_valid(intel_gt_perf_limit_reasons_reg(gt)); +} + +DEFINE_SIMPLE_ATTRIBUTE(perf_limit_reasons_fops, perf_limit_reasons_get, + perf_limit_reasons_clear, "%llu\n"); + void intel_gt_pm_debugfs_register(struct intel_gt *gt, struct dentry *root) { static const struct intel_gt_debugfs_file files[] = { @@ -664,6 +591,7 @@ void intel_gt_pm_debugfs_register(struct intel_gt *gt, struct dentry *root) { "forcewake_user", &forcewake_user_fops, NULL}, { "llc", &llc_fops, llc_eval }, { "rps_boost", &rps_boost_fops, rps_eval }, + { "perf_limit_reasons", &perf_limit_reasons_fops, perf_limit_reasons_eval }, }; intel_gt_debugfs_register_files(root, files, ARRAY_SIZE(files), gt); diff --git a/drivers/gpu/drm/i915/gt/intel_gt_regs.h b/drivers/gpu/drm/i915/gt/intel_gt_regs.h index 2275ee47da95..c3cd92691795 100644 --- a/drivers/gpu/drm/i915/gt/intel_gt_regs.h +++ b/drivers/gpu/drm/i915/gt/intel_gt_regs.h @@ -7,6 +7,27 @@ #define __INTEL_GT_REGS__ #include "i915_reg_defs.h" +#include "display/intel_display_reg_defs.h" /* VLV_DISPLAY_BASE */ + +#define MCR_REG(offset) ((const i915_mcr_reg_t){ .reg = (offset) }) + +/* + * The perf control registers are technically multicast registers, but the + * driver never needs to read/write them directly; we only use them to build + * lists of registers (where they're mixed in with other non-MCR registers) + * and then operate on the offset directly. For now we'll just define them + * as non-multicast so we can place them on the same list, but we may want + * to try to come up with a better way to handle heterogeneous lists of + * registers in the future. + */ +#define PERF_REG(offset) _MMIO(offset) + +/* MTL workpoint reg to get core C state and actual freq of 3D, SAMedia */ +#define MTL_MIRROR_TARGET_WP1 _MMIO(0xc60) +#define MTL_CAGF_MASK REG_GENMASK(8, 0) +#define MTL_CC0 0x0 +#define MTL_CC6 0x3 +#define MTL_CC_MASK REG_GENMASK(12, 9) /* RPM unit config (Gen8+) */ #define RPM_CONFIG0 _MMIO(0xd00) @@ -39,9 +60,17 @@ #define FORCEWAKE_ACK_RENDER_GEN9 _MMIO(0xd84) #define FORCEWAKE_ACK_MEDIA_GEN9 _MMIO(0xd88) +#define FORCEWAKE_ACK_GSC _MMIO(0xdf8) +#define FORCEWAKE_ACK_GT_MTL _MMIO(0xdfc) + +#define GMD_ID_GRAPHICS _MMIO(0xd8c) +#define GMD_ID_MEDIA _MMIO(MTL_MEDIA_GSI_BASE + 0xd8c) + #define MCFG_MCR_SELECTOR _MMIO(0xfd0) +#define MTL_MCR_SELECTOR _MMIO(0xfd4) #define SF_MCR_SELECTOR _MMIO(0xfd8) #define GEN8_MCR_SELECTOR _MMIO(0xfdc) +#define GAM_MCR_SELECTOR _MMIO(0xfe0) #define GEN8_MCR_SLICE(slice) (((slice) & 3) << 26) #define GEN8_MCR_SLICE_MASK GEN8_MCR_SLICE(3) #define GEN8_MCR_SUBSLICE(subslice) (((subslice) & 3) << 24) @@ -51,6 +80,8 @@ #define GEN11_MCR_SLICE_MASK GEN11_MCR_SLICE(0xf) #define GEN11_MCR_SUBSLICE(subslice) (((subslice) & 0x7) << 24) #define GEN11_MCR_SUBSLICE_MASK GEN11_MCR_SUBSLICE(0x7) +#define MTL_MCR_GROUPID REG_GENMASK(11, 8) +#define MTL_MCR_INSTANCEID REG_GENMASK(3, 0) #define IPEIR_I965 _MMIO(0x2064) #define IPEHR_I965 _MMIO(0x2068) @@ -326,11 +357,12 @@ #define GEN7_TLB_RD_ADDR _MMIO(0x4700) #define GEN12_PAT_INDEX(index) _MMIO(0x4800 + (index) * 4) +#define XEHP_PAT_INDEX(index) MCR_REG(0x4800 + (index) * 4) -#define XEHP_TILE0_ADDR_RANGE _MMIO(0x4900) +#define XEHP_TILE0_ADDR_RANGE MCR_REG(0x4900) #define XEHP_TILE_LMEM_RANGE_SHIFT 8 -#define XEHP_FLAT_CCS_BASE_ADDR _MMIO(0x4910) +#define XEHP_FLAT_CCS_BASE_ADDR MCR_REG(0x4910) #define XEHP_CCS_BASE_SHIFT 8 #define GAMTARBMODE _MMIO(0x4a08) @@ -380,17 +412,18 @@ #define CHICKEN_RASTER_2 _MMIO(0x6208) #define TBIMR_FAST_CLIP REG_BIT(5) -#define VFLSKPD _MMIO(0x62a8) +#define VFLSKPD MCR_REG(0x62a8) #define DIS_OVER_FETCH_CACHE REG_BIT(1) #define DIS_MULT_MISS_RD_SQUASH REG_BIT(0) -#define FF_MODE2 _MMIO(0x6604) +#define GEN12_FF_MODE2 _MMIO(0x6604) +#define XEHP_FF_MODE2 MCR_REG(0x6604) #define FF_MODE2_GS_TIMER_MASK REG_GENMASK(31, 24) #define FF_MODE2_GS_TIMER_224 REG_FIELD_PREP(FF_MODE2_GS_TIMER_MASK, 224) #define FF_MODE2_TDS_TIMER_MASK REG_GENMASK(23, 16) #define FF_MODE2_TDS_TIMER_128 REG_FIELD_PREP(FF_MODE2_TDS_TIMER_MASK, 4) -#define XEHPG_INSTDONE_GEOM_SVG _MMIO(0x666c) +#define XEHPG_INSTDONE_GEOM_SVG MCR_REG(0x666c) #define CACHE_MODE_0_GEN7 _MMIO(0x7000) /* IVB+ */ #define RC_OP_FLUSH_ENABLE (1 << 0) @@ -418,6 +451,7 @@ #define HIZ_CHICKEN _MMIO(0x7018) #define CHV_HZ_8X8_MODE_IN_1X REG_BIT(15) #define DG1_HZ_READ_SUPPRESSION_OPTIMIZATION_DISABLE REG_BIT(14) +#define HZ_DEPTH_TEST_LE_GE_OPT_DISABLE REG_BIT(13) #define BDW_HIZ_POWER_COMPILER_CLOCK_GATING_DISABLE REG_BIT(3) #define GEN8_L3CNTLREG _MMIO(0x7034) @@ -439,23 +473,16 @@ #define GEN8_HDC_CHICKEN1 _MMIO(0x7304) #define GEN11_COMMON_SLICE_CHICKEN3 _MMIO(0x7304) +#define XEHP_COMMON_SLICE_CHICKEN3 MCR_REG(0x7304) #define DG1_FLOAT_POINT_BLEND_OPT_STRICT_MODE_EN REG_BIT(12) #define XEHP_DUAL_SIMD8_SEQ_MERGE_DISABLE REG_BIT(12) #define GEN11_BLEND_EMB_FIX_DISABLE_IN_RCC REG_BIT(11) #define GEN12_DISABLE_CPS_AWARE_COLOR_PIPE REG_BIT(9) -/* GEN9 chicken */ -#define SLICE_ECO_CHICKEN0 _MMIO(0x7308) -#define PIXEL_MASK_CAMMING_DISABLE (1 << 14) - -#define GEN9_SLICE_COMMON_ECO_CHICKEN0 _MMIO(0x7308) -#define DISABLE_PIXEL_MASK_CAMMING (1 << 14) - #define GEN9_SLICE_COMMON_ECO_CHICKEN1 _MMIO(0x731c) -#define GEN11_STATE_CACHE_REDIRECT_TO_CS (1 << 11) - -#define SLICE_COMMON_ECO_CHICKEN1 _MMIO(0x731c) +#define XEHP_SLICE_COMMON_ECO_CHICKEN1 MCR_REG(0x731c) #define MSC_MSAA_REODER_BUF_BYPASS_DISABLE REG_BIT(14) +#define GEN11_STATE_CACHE_REDIRECT_TO_CS (1 << 11) #define GEN9_SLICE_PGCTL_ACK(slice) _MMIO(0x804c + (slice) * 0x4) #define GEN10_SLICE_PGCTL_ACK(slice) _MMIO(0x804c + ((slice) / 3) * 0x34 + \ @@ -482,9 +509,12 @@ #define VF_PREEMPTION _MMIO(0x83a4) #define PREEMPTION_VERTEX_COUNT REG_GENMASK(15, 0) +#define VFG_PREEMPTION_CHICKEN _MMIO(0x83b4) +#define POLYGON_TRIFAN_LINELOOP_DISABLE REG_BIT(4) + #define GEN8_RC6_CTX_INFO _MMIO(0x8504) -#define GEN12_SQCM _MMIO(0x8724) +#define XEHP_SQCM MCR_REG(0x8724) #define EN_32B_ACCESS REG_BIT(30) #define HSW_IDICR _MMIO(0x9008) @@ -516,6 +546,8 @@ #define GEN6_MBCTL_BOOT_FETCH_MECH (1 << 0) /* Fuse readout registers for GT */ +#define XEHP_FUSE4 _MMIO(0x9114) +#define GT_L3_EXC_MASK REG_GENMASK(6, 4) #define GEN10_MIRROR_FUSE3 _MMIO(0x9118) #define GEN10_L3BANK_PAIR_COUNT 4 #define GEN10_L3BANK_MASK 0x0F @@ -619,6 +651,7 @@ #define XEHPC_GRDOM_BLT3 REG_BIT(26) #define XEHPC_GRDOM_BLT2 REG_BIT(25) #define XEHPC_GRDOM_BLT1 REG_BIT(24) +#define GEN12_GRDOM_GSC REG_BIT(21) #define GEN11_GRDOM_SFC3 REG_BIT(20) #define GEN11_GRDOM_SFC2 REG_BIT(19) #define GEN11_GRDOM_SFC1 REG_BIT(18) @@ -644,6 +677,9 @@ #define GEN7_MISCCPCTL _MMIO(0x9424) #define GEN7_DOP_CLOCK_GATE_ENABLE (1 << 0) + +#define GEN8_MISCCPCTL MCR_REG(0x9424) +#define GEN8_DOP_CLOCK_GATE_ENABLE REG_BIT(0) #define GEN12_DOP_CLOCK_GATE_RENDER_ENABLE REG_BIT(1) #define GEN8_DOP_CLOCK_GATE_CFCLK_ENABLE (1 << 2) #define GEN8_DOP_CLOCK_GATE_GUC_ENABLE (1 << 4) @@ -697,7 +733,8 @@ #define GAMTLBVEBOX0_CLKGATE_DIS REG_BIT(16) #define LTCDD_CLKGATE_DIS REG_BIT(10) -#define SLICE_UNIT_LEVEL_CLKGATE _MMIO(0x94d4) +#define GEN11_SLICE_UNIT_LEVEL_CLKGATE _MMIO(0x94d4) +#define XEHP_SLICE_UNIT_LEVEL_CLKGATE MCR_REG(0x94d4) #define SARBUNIT_CLKGATE_DIS (1 << 5) #define RCCUNIT_CLKGATE_DIS (1 << 7) #define MSCUNIT_CLKGATE_DIS (1 << 10) @@ -705,27 +742,27 @@ #define L3_CLKGATE_DIS REG_BIT(16) #define L3_CR2X_CLKGATE_DIS REG_BIT(17) -#define SCCGCTL94DC _MMIO(0x94dc) +#define SCCGCTL94DC MCR_REG(0x94dc) #define CG3DDISURB REG_BIT(14) #define UNSLICE_UNIT_LEVEL_CLKGATE2 _MMIO(0x94e4) #define VSUNIT_CLKGATE_DIS_TGL REG_BIT(19) #define PSDUNIT_CLKGATE_DIS REG_BIT(5) -#define SUBSLICE_UNIT_LEVEL_CLKGATE _MMIO(0x9524) +#define GEN11_SUBSLICE_UNIT_LEVEL_CLKGATE MCR_REG(0x9524) #define DSS_ROUTER_CLKGATE_DIS REG_BIT(28) #define GWUNIT_CLKGATE_DIS REG_BIT(16) -#define SUBSLICE_UNIT_LEVEL_CLKGATE2 _MMIO(0x9528) +#define SUBSLICE_UNIT_LEVEL_CLKGATE2 MCR_REG(0x9528) #define CPSSUNIT_CLKGATE_DIS REG_BIT(9) -#define SSMCGCTL9530 _MMIO(0x9530) +#define SSMCGCTL9530 MCR_REG(0x9530) #define RTFUNIT_CLKGATE_DIS REG_BIT(18) -#define GEN10_DFR_RATIO_EN_AND_CHICKEN _MMIO(0x9550) +#define GEN10_DFR_RATIO_EN_AND_CHICKEN MCR_REG(0x9550) #define DFR_DISABLE (1 << 9) -#define INF_UNIT_LEVEL_CLKGATE _MMIO(0x9560) +#define INF_UNIT_LEVEL_CLKGATE MCR_REG(0x9560) #define CGPSF_CLKGATE_DIS (1 << 3) #define MICRO_BP0_0 _MMIO(0x9800) @@ -770,12 +807,9 @@ #define GEN6_RP_DOWN_TIMEOUT _MMIO(0xa010) #define GEN6_RP_INTERRUPT_LIMITS _MMIO(0xa014) #define GEN6_RPSTAT1 _MMIO(0xa01c) -#define GEN6_CAGF_SHIFT 8 -#define HSW_CAGF_SHIFT 7 -#define GEN9_CAGF_SHIFT 23 -#define GEN6_CAGF_MASK (0x7f << GEN6_CAGF_SHIFT) -#define HSW_CAGF_MASK (0x7f << HSW_CAGF_SHIFT) -#define GEN9_CAGF_MASK (0x1ff << GEN9_CAGF_SHIFT) +#define GEN6_CAGF_MASK REG_GENMASK(14, 8) +#define HSW_CAGF_MASK REG_GENMASK(13, 7) +#define GEN9_CAGF_MASK REG_GENMASK(31, 23) #define GEN6_RP_CONTROL _MMIO(0xa024) #define GEN6_RP_MEDIA_TURBO (1 << 11) #define GEN6_RP_MEDIA_MODE_MASK (3 << 9) @@ -898,6 +932,8 @@ #define FORCEWAKE_MEDIA_VDBOX_GEN11(n) _MMIO(0xa540 + (n) * 4) #define FORCEWAKE_MEDIA_VEBOX_GEN11(n) _MMIO(0xa560 + (n) * 4) +#define FORCEWAKE_REQ_GSC _MMIO(0xa618) + #define CHV_POWER_SS0_SIG1 _MMIO(0xa720) #define CHV_POWER_SS0_SIG2 _MMIO(0xa724) #define CHV_POWER_SS1_SIG1 _MMIO(0xa728) @@ -935,7 +971,8 @@ /* MOCS (Memory Object Control State) registers */ #define GEN9_LNCFCMOCS(i) _MMIO(0xb020 + (i) * 4) /* L3 Cache Control */ -#define GEN9_LNCFCMOCS_REG_COUNT 32 +#define XEHP_LNCFCMOCS(i) MCR_REG(0xb020 + (i) * 4) +#define LNCFCMOCS_REG_COUNT 32 #define GEN7_L3CNTLREG3 _MMIO(0xb024) @@ -951,15 +988,10 @@ #define GEN7_L3LOG(slice, i) _MMIO(0xb070 + (slice) * 0x200 + (i) * 4) #define GEN7_L3LOG_SIZE 0x80 -#define GEN10_SCRATCH_LNCF2 _MMIO(0xb0a0) -#define PMFLUSHDONE_LNICRSDROP (1 << 20) -#define PMFLUSH_GAPL3UNBLOCK (1 << 21) -#define PMFLUSHDONE_LNEBLK (1 << 22) - -#define XEHP_L3NODEARBCFG _MMIO(0xb0b4) +#define XEHP_L3NODEARBCFG MCR_REG(0xb0b4) #define XEHP_LNESPARE REG_BIT(19) -#define GEN8_L3SQCREG1 _MMIO(0xb100) +#define GEN8_L3SQCREG1 MCR_REG(0xb100) /* * Note that on CHV the following has an off-by-one error wrt. to BSpec. * Using the formula in BSpec leads to a hang, while the formula here works @@ -970,31 +1002,28 @@ #define L3_HIGH_PRIO_CREDITS(x) (((x) >> 1) << 14) #define L3_PRIO_CREDITS_MASK ((0x1f << 19) | (0x1f << 14)) -#define GEN10_L3_CHICKEN_MODE_REGISTER _MMIO(0xb114) -#define GEN11_I2M_WRITE_DISABLE (1 << 28) - -#define GEN8_L3SQCREG4 _MMIO(0xb118) +#define GEN8_L3SQCREG4 MCR_REG(0xb118) #define GEN11_LQSC_CLEAN_EVICT_DISABLE (1 << 6) #define GEN8_LQSC_RO_PERF_DIS (1 << 27) #define GEN8_LQSC_FLUSH_COHERENT_LINES (1 << 21) #define GEN8_LQSQ_NONIA_COHERENT_ATOMICS_ENABLE REG_BIT(22) -#define GEN9_SCRATCH1 _MMIO(0xb11c) +#define GEN9_SCRATCH1 MCR_REG(0xb11c) #define EVICTION_PERF_FIX_ENABLE REG_BIT(8) -#define BDW_SCRATCH1 _MMIO(0xb11c) +#define BDW_SCRATCH1 MCR_REG(0xb11c) #define GEN9_LBS_SLA_RETRY_TIMER_DECREMENT_ENABLE (1 << 2) -#define GEN11_SCRATCH2 _MMIO(0xb140) +#define GEN11_SCRATCH2 MCR_REG(0xb140) #define GEN11_COHERENT_PARTIAL_WRITE_MERGE_ENABLE (1 << 19) -#define GEN11_L3SQCREG5 _MMIO(0xb158) +#define XEHP_L3SQCREG5 MCR_REG(0xb158) #define L3_PWM_TIMER_INIT_VAL_MASK REG_GENMASK(9, 0) -#define MLTICTXCTL _MMIO(0xb170) +#define MLTICTXCTL MCR_REG(0xb170) #define TDONRENDER REG_BIT(2) -#define XEHP_L3SCQREG7 _MMIO(0xb188) +#define XEHP_L3SCQREG7 MCR_REG(0xb188) #define BLEND_FILL_CACHING_OPT_DIS REG_BIT(3) #define XEHPC_L3SCRUB _MMIO(0xb18c) @@ -1002,7 +1031,7 @@ #define SCRUB_RATE_PER_BANK_MASK REG_GENMASK(2, 0) #define SCRUB_RATE_4B_PER_CLK REG_FIELD_PREP(SCRUB_RATE_PER_BANK_MASK, 0x6) -#define L3SQCREG1_CCS0 _MMIO(0xb200) +#define L3SQCREG1_CCS0 MCR_REG(0xb200) #define FLUSHALLNONCOH REG_BIT(5) #define GEN11_GLBLINVL _MMIO(0xb404) @@ -1027,11 +1056,14 @@ #define GEN9_BLT_MOCS(i) _MMIO(__GEN9_BCS0_MOCS0 + (i) * 4) #define GEN12_FAULT_TLB_DATA0 _MMIO(0xceb8) +#define XEHP_FAULT_TLB_DATA0 MCR_REG(0xceb8) #define GEN12_FAULT_TLB_DATA1 _MMIO(0xcebc) +#define XEHP_FAULT_TLB_DATA1 MCR_REG(0xcebc) #define FAULT_VA_HIGH_BITS (0xf << 0) #define FAULT_GTT_SEL (1 << 4) #define GEN12_RING_FAULT_REG _MMIO(0xcec4) +#define XEHP_RING_FAULT_REG MCR_REG(0xcec4) #define GEN8_RING_FAULT_ENGINE_ID(x) (((x) >> 12) & 0x7) #define RING_FAULT_GTTSEL_MASK (1 << 11) #define RING_FAULT_SRCID(x) (((x) >> 3) & 0xff) @@ -1039,16 +1071,21 @@ #define RING_FAULT_VALID (1 << 0) #define GEN12_GFX_TLB_INV_CR _MMIO(0xced8) +#define XEHP_GFX_TLB_INV_CR MCR_REG(0xced8) #define GEN12_VD_TLB_INV_CR _MMIO(0xcedc) +#define XEHP_VD_TLB_INV_CR MCR_REG(0xcedc) #define GEN12_VE_TLB_INV_CR _MMIO(0xcee0) +#define XEHP_VE_TLB_INV_CR MCR_REG(0xcee0) #define GEN12_BLT_TLB_INV_CR _MMIO(0xcee4) +#define XEHP_BLT_TLB_INV_CR MCR_REG(0xcee4) #define GEN12_COMPCTX_TLB_INV_CR _MMIO(0xcf04) +#define XEHP_COMPCTX_TLB_INV_CR MCR_REG(0xcf04) -#define GEN12_MERT_MOD_CTRL _MMIO(0xcf28) -#define RENDER_MOD_CTRL _MMIO(0xcf2c) -#define COMP_MOD_CTRL _MMIO(0xcf30) -#define VDBX_MOD_CTRL _MMIO(0xcf34) -#define VEBX_MOD_CTRL _MMIO(0xcf38) +#define XEHP_MERT_MOD_CTRL MCR_REG(0xcf28) +#define RENDER_MOD_CTRL MCR_REG(0xcf2c) +#define COMP_MOD_CTRL MCR_REG(0xcf30) +#define VDBX_MOD_CTRL MCR_REG(0xcf34) +#define VEBX_MOD_CTRL MCR_REG(0xcf38) #define FORCE_MISS_FTLB REG_BIT(3) #define GEN12_GAMSTLB_CTRL _MMIO(0xcf4c) @@ -1063,48 +1100,52 @@ #define GEN12_GAM_DONE _MMIO(0xcf68) #define GEN7_HALF_SLICE_CHICKEN1 _MMIO(0xe100) /* IVB GT1 + VLV */ +#define GEN8_HALF_SLICE_CHICKEN1 MCR_REG(0xe100) #define GEN7_MAX_PS_THREAD_DEP (8 << 12) #define GEN7_SINGLE_SUBSCAN_DISPATCH_ENABLE (1 << 10) #define GEN7_SBE_SS_CACHE_DISPATCH_PORT_SHARING_DISABLE (1 << 4) #define GEN7_PSD_SINGLE_PORT_DISPATCH_ENABLE (1 << 3) #define GEN7_SAMPLER_INSTDONE _MMIO(0xe160) +#define GEN8_SAMPLER_INSTDONE MCR_REG(0xe160) #define GEN7_ROW_INSTDONE _MMIO(0xe164) +#define GEN8_ROW_INSTDONE MCR_REG(0xe164) -#define HALF_SLICE_CHICKEN2 _MMIO(0xe180) +#define HALF_SLICE_CHICKEN2 MCR_REG(0xe180) #define GEN8_ST_PO_DISABLE (1 << 13) -#define HALF_SLICE_CHICKEN3 _MMIO(0xe184) +#define HSW_HALF_SLICE_CHICKEN3 _MMIO(0xe184) +#define GEN8_HALF_SLICE_CHICKEN3 MCR_REG(0xe184) #define HSW_SAMPLE_C_PERFORMANCE (1 << 9) #define GEN8_CENTROID_PIXEL_OPT_DIS (1 << 8) #define GEN9_DISABLE_OCL_OOB_SUPPRESS_LOGIC (1 << 5) #define GEN8_SAMPLER_POWER_BYPASS_DIS (1 << 1) -#define GEN9_HALF_SLICE_CHICKEN5 _MMIO(0xe188) +#define GEN9_HALF_SLICE_CHICKEN5 MCR_REG(0xe188) #define GEN9_DG_MIRROR_FIX_ENABLE (1 << 5) #define GEN9_CCS_TLB_PREFETCH_ENABLE (1 << 3) -#define GEN10_SAMPLER_MODE _MMIO(0xe18c) +#define GEN10_SAMPLER_MODE MCR_REG(0xe18c) #define ENABLE_SMALLPL REG_BIT(15) #define SC_DISABLE_POWER_OPTIMIZATION_EBB REG_BIT(9) #define GEN11_SAMPLER_ENABLE_HEADLESS_MSG REG_BIT(5) -#define GEN9_HALF_SLICE_CHICKEN7 _MMIO(0xe194) +#define GEN9_HALF_SLICE_CHICKEN7 MCR_REG(0xe194) #define DG2_DISABLE_ROUND_ENABLE_ALLOW_FOR_SSLA REG_BIT(15) #define GEN9_SAMPLER_HASH_COMPRESSED_READ_ADDR REG_BIT(8) #define GEN9_ENABLE_YV12_BUGFIX REG_BIT(4) #define GEN9_ENABLE_GPGPU_PREEMPTION REG_BIT(2) -#define GEN10_CACHE_MODE_SS _MMIO(0xe420) +#define GEN10_CACHE_MODE_SS MCR_REG(0xe420) #define ENABLE_EU_COUNT_FOR_TDL_FLUSH REG_BIT(10) #define DISABLE_ECC REG_BIT(5) #define FLOAT_BLEND_OPTIMIZATION_ENABLE REG_BIT(4) #define ENABLE_PREFETCH_INTO_IC REG_BIT(3) -#define EU_PERF_CNTL0 _MMIO(0xe458) -#define EU_PERF_CNTL4 _MMIO(0xe45c) +#define EU_PERF_CNTL0 PERF_REG(0xe458) +#define EU_PERF_CNTL4 PERF_REG(0xe45c) -#define GEN9_ROW_CHICKEN4 _MMIO(0xe48c) +#define GEN9_ROW_CHICKEN4 MCR_REG(0xe48c) #define GEN12_DISABLE_GRF_CLEAR REG_BIT(13) #define XEHP_DIS_BBL_SYSPIPE REG_BIT(11) #define GEN12_DISABLE_TDL_PUSH REG_BIT(9) @@ -1116,7 +1157,7 @@ #define HSW_ROW_CHICKEN3 _MMIO(0xe49c) #define HSW_ROW_CHICKEN3_L3_GLOBAL_ATOMICS_DISABLE (1 << 6) -#define GEN8_ROW_CHICKEN _MMIO(0xe4f0) +#define GEN8_ROW_CHICKEN MCR_REG(0xe4f0) #define FLOW_CONTROL_ENABLE REG_BIT(15) #define UGM_BACKUP_MODE REG_BIT(13) #define MDQ_ARBITRATION_MODE REG_BIT(12) @@ -1127,42 +1168,43 @@ #define DISABLE_EARLY_EOT REG_BIT(1) #define GEN7_ROW_CHICKEN2 _MMIO(0xe4f4) + +#define GEN8_ROW_CHICKEN2 MCR_REG(0xe4f4) #define GEN12_DISABLE_READ_SUPPRESSION REG_BIT(15) #define GEN12_DISABLE_EARLY_READ REG_BIT(14) #define GEN12_ENABLE_LARGE_GRF_MODE REG_BIT(12) #define GEN12_PUSH_CONST_DEREF_HOLD_DIS REG_BIT(8) +#define GEN12_DISABLE_DOP_GATING REG_BIT(0) -#define RT_CTRL _MMIO(0xe530) +#define RT_CTRL MCR_REG(0xe530) #define DIS_NULL_QUERY REG_BIT(10) #define STACKID_CTRL REG_GENMASK(6, 5) #define STACKID_CTRL_512 REG_FIELD_PREP(STACKID_CTRL, 0x2) -#define EU_PERF_CNTL1 _MMIO(0xe558) -#define EU_PERF_CNTL5 _MMIO(0xe55c) +#define EU_PERF_CNTL1 PERF_REG(0xe558) +#define EU_PERF_CNTL5 PERF_REG(0xe55c) -#define GEN12_HDC_CHICKEN0 _MMIO(0xe5f0) +#define XEHP_HDC_CHICKEN0 MCR_REG(0xe5f0) #define LSC_L1_FLUSH_CTL_3D_DATAPORT_FLUSH_EVENTS_MASK REG_GENMASK(13, 11) -#define ICL_HDC_MODE _MMIO(0xe5f4) +#define ICL_HDC_MODE MCR_REG(0xe5f4) -#define EU_PERF_CNTL2 _MMIO(0xe658) -#define EU_PERF_CNTL6 _MMIO(0xe65c) -#define EU_PERF_CNTL3 _MMIO(0xe758) +#define EU_PERF_CNTL2 PERF_REG(0xe658) +#define EU_PERF_CNTL6 PERF_REG(0xe65c) +#define EU_PERF_CNTL3 PERF_REG(0xe758) -#define LSC_CHICKEN_BIT_0 _MMIO(0xe7c8) +#define LSC_CHICKEN_BIT_0 MCR_REG(0xe7c8) #define DISABLE_D8_D16_COASLESCE REG_BIT(30) #define FORCE_1_SUB_MESSAGE_PER_FRAGMENT REG_BIT(15) -#define LSC_CHICKEN_BIT_0_UDW _MMIO(0xe7c8 + 4) +#define LSC_CHICKEN_BIT_0_UDW MCR_REG(0xe7c8 + 4) #define DIS_CHAIN_2XSIMD8 REG_BIT(55 - 32) #define FORCE_SLM_FENCE_SCOPE_TO_TILE REG_BIT(42 - 32) #define FORCE_UGM_FENCE_SCOPE_TO_TILE REG_BIT(41 - 32) #define MAXREQS_PER_BANK REG_GENMASK(39 - 32, 37 - 32) #define DISABLE_128B_EVICTION_COMMAND_UDW REG_BIT(36 - 32) -#define SARB_CHICKEN1 _MMIO(0xe90c) +#define SARB_CHICKEN1 MCR_REG(0xe90c) #define COMP_CKN_IN REG_GENMASK(30, 29) -#define GEN7_HALF_SLICE_CHICKEN1_GT2 _MMIO(0xf100) - #define GEN7_ROW_CHICKEN2_GT2 _MMIO(0xf4f4) #define DOP_CLOCK_GATING_DISABLE (1 << 0) #define PUSH_CONSTANT_DEREF_DISABLE (1 << 8) @@ -1339,8 +1381,7 @@ #define MEMSTAT_ILK _MMIO(0x111f8) #define MEMSTAT_VID_MASK 0x7f00 #define MEMSTAT_VID_SHIFT 8 -#define MEMSTAT_PSTATE_MASK 0x00f8 -#define MEMSTAT_PSTATE_SHIFT 3 +#define MEMSTAT_PSTATE_MASK REG_GENMASK(7, 3) #define MEMSTAT_MON_ACTV (1 << 2) #define MEMSTAT_SRC_CTL_MASK 0x0003 #define MEMSTAT_SRC_CTL_CORE 0 @@ -1481,6 +1522,8 @@ #define FORCEWAKE_MEDIA_VLV _MMIO(0x1300b8) #define FORCEWAKE_ACK_MEDIA_VLV _MMIO(0x1300bc) +#define MTL_MEDIA_MC6 _MMIO(0x138048) + #define GEN6_GT_THREAD_STATUS_REG _MMIO(0x13805c) #define GEN6_GT_THREAD_STATUS_CORE_MASK 0x7 @@ -1510,10 +1553,15 @@ #define VLV_RENDER_C0_COUNT _MMIO(0x138118) #define VLV_MEDIA_C0_COUNT _MMIO(0x13811c) +#define GEN12_RPSTAT1 _MMIO(0x1381b4) +#define GEN12_VOLTAGE_MASK REG_GENMASK(10, 0) +#define GEN12_CAGF_MASK REG_GENMASK(19, 11) + #define GEN11_GT_INTR_DW(x) _MMIO(0x190018 + ((x) * 4)) #define GEN11_CSME (31) #define GEN11_GUNIT (28) #define GEN11_GUC (25) +#define MTL_MGUC (24) #define GEN11_WDPERF (20) #define GEN11_KCR (19) #define GEN11_GTPM (16) @@ -1568,6 +1616,7 @@ #define GEN11_VECS0_VECS1_INTR_MASK _MMIO(0x1900d0) #define GEN12_VECS2_VECS3_INTR_MASK _MMIO(0x1900d4) #define GEN11_GUC_SG_INTR_MASK _MMIO(0x1900e8) +#define MTL_GUC_MGUC_INTR_MASK _MMIO(0x1900e8) /* MTL+ */ #define GEN11_GPM_WGBOXPERF_INTR_MASK _MMIO(0x1900ec) #define GEN11_CRYPTO_RSVD_INTR_MASK _MMIO(0x1900f0) #define GEN11_GUNIT_CSME_INTR_MASK _MMIO(0x1900f4) @@ -1580,6 +1629,11 @@ #define GEN12_SFC_DONE(n) _MMIO(0x1cc000 + (n) * 0x1000) +#define GT0_PACKAGE_ENERGY_STATUS _MMIO(0x250004) +#define GT0_PACKAGE_RAPL_LIMIT _MMIO(0x250008) +#define GT0_PACKAGE_POWER_SKU_UNIT _MMIO(0x250068) +#define GT0_PLATFORM_ENERGY_STATUS _MMIO(0x25006c) + /* * Standalone Media's non-engine GT registers are located at their regular GT * offsets plus 0x380000. This extra offset is stored inside the intel_uncore diff --git a/drivers/gpu/drm/i915/gt/intel_gt_sysfs.c b/drivers/gpu/drm/i915/gt/intel_gt_sysfs.c index d651ccd0ab20..9486dd3bed99 100644 --- a/drivers/gpu/drm/i915/gt/intel_gt_sysfs.c +++ b/drivers/gpu/drm/i915/gt/intel_gt_sysfs.c @@ -22,11 +22,9 @@ bool is_object_gt(struct kobject *kobj) return !strncmp(kobj->name, "gt", 2); } -struct intel_gt *intel_gt_sysfs_get_drvdata(struct device *dev, +struct intel_gt *intel_gt_sysfs_get_drvdata(struct kobject *kobj, const char *name) { - struct kobject *kobj = &dev->kobj; - /* * We are interested at knowing from where the interface * has been called, whether it's called from gt/ or from @@ -38,6 +36,7 @@ struct intel_gt *intel_gt_sysfs_get_drvdata(struct device *dev, * "struct drm_i915_private *" type. */ if (!is_object_gt(kobj)) { + struct device *dev = kobj_to_dev(kobj); struct drm_i915_private *i915 = kdev_minor_to_i915(dev); return to_gt(i915); @@ -51,18 +50,18 @@ static struct kobject *gt_get_parent_obj(struct intel_gt *gt) return >->i915->drm.primary->kdev->kobj; } -static ssize_t id_show(struct device *dev, - struct device_attribute *attr, +static ssize_t id_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) { - struct intel_gt *gt = intel_gt_sysfs_get_drvdata(dev, attr->attr.name); + struct intel_gt *gt = intel_gt_sysfs_get_drvdata(kobj, attr->attr.name); return sysfs_emit(buf, "%u\n", gt->info.id); } -static DEVICE_ATTR_RO(id); +static struct kobj_attribute attr_id = __ATTR_RO(id); static struct attribute *id_attrs[] = { - &dev_attr_id.attr, + &attr_id.attr, NULL, }; ATTRIBUTE_GROUPS(id); diff --git a/drivers/gpu/drm/i915/gt/intel_gt_sysfs.h b/drivers/gpu/drm/i915/gt/intel_gt_sysfs.h index 6232923a420d..18bab835be02 100644 --- a/drivers/gpu/drm/i915/gt/intel_gt_sysfs.h +++ b/drivers/gpu/drm/i915/gt/intel_gt_sysfs.h @@ -18,11 +18,6 @@ bool is_object_gt(struct kobject *kobj); struct drm_i915_private *kobj_to_i915(struct kobject *kobj); -struct kobject * -intel_gt_create_kobj(struct intel_gt *gt, - struct kobject *dir, - const char *name); - static inline struct intel_gt *kobj_to_gt(struct kobject *kobj) { return container_of(kobj, struct intel_gt, sysfs_gt); @@ -30,7 +25,7 @@ static inline struct intel_gt *kobj_to_gt(struct kobject *kobj) void intel_gt_sysfs_register(struct intel_gt *gt); void intel_gt_sysfs_unregister(struct intel_gt *gt); -struct intel_gt *intel_gt_sysfs_get_drvdata(struct device *dev, +struct intel_gt *intel_gt_sysfs_get_drvdata(struct kobject *kobj, const char *name); #endif /* SYSFS_GT_H */ diff --git a/drivers/gpu/drm/i915/gt/intel_gt_sysfs_pm.c b/drivers/gpu/drm/i915/gt/intel_gt_sysfs_pm.c index 180dd6f3ef57..cf71305ad586 100644 --- a/drivers/gpu/drm/i915/gt/intel_gt_sysfs_pm.c +++ b/drivers/gpu/drm/i915/gt/intel_gt_sysfs_pm.c @@ -24,14 +24,15 @@ enum intel_gt_sysfs_op { }; static int -sysfs_gt_attribute_w_func(struct device *dev, struct device_attribute *attr, +sysfs_gt_attribute_w_func(struct kobject *kobj, struct attribute *attr, int (func)(struct intel_gt *gt, u32 val), u32 val) { struct intel_gt *gt; int ret; - if (!is_object_gt(&dev->kobj)) { + if (!is_object_gt(kobj)) { int i; + struct device *dev = kobj_to_dev(kobj); struct drm_i915_private *i915 = kdev_minor_to_i915(dev); for_each_gt(gt, i915, i) { @@ -40,7 +41,7 @@ sysfs_gt_attribute_w_func(struct device *dev, struct device_attribute *attr, break; } } else { - gt = intel_gt_sysfs_get_drvdata(dev, attr->attr.name); + gt = intel_gt_sysfs_get_drvdata(kobj, attr->name); ret = func(gt, val); } @@ -48,7 +49,7 @@ sysfs_gt_attribute_w_func(struct device *dev, struct device_attribute *attr, } static u32 -sysfs_gt_attribute_r_func(struct device *dev, struct device_attribute *attr, +sysfs_gt_attribute_r_func(struct kobject *kobj, struct attribute *attr, u32 (func)(struct intel_gt *gt), enum intel_gt_sysfs_op op) { @@ -57,8 +58,9 @@ sysfs_gt_attribute_r_func(struct device *dev, struct device_attribute *attr, ret = (op == INTEL_GT_SYSFS_MAX) ? 0 : (u32) -1; - if (!is_object_gt(&dev->kobj)) { + if (!is_object_gt(kobj)) { int i; + struct device *dev = kobj_to_dev(kobj); struct drm_i915_private *i915 = kdev_minor_to_i915(dev); for_each_gt(gt, i915, i) { @@ -77,7 +79,7 @@ sysfs_gt_attribute_r_func(struct device *dev, struct device_attribute *attr, } } } else { - gt = intel_gt_sysfs_get_drvdata(dev, attr->attr.name); + gt = intel_gt_sysfs_get_drvdata(kobj, attr->name); ret = func(gt); } @@ -92,23 +94,90 @@ sysfs_gt_attribute_r_func(struct device *dev, struct device_attribute *attr, #define sysfs_gt_attribute_r_max_func(d, a, f) \ sysfs_gt_attribute_r_func(d, a, f, INTEL_GT_SYSFS_MAX) +#define INTEL_GT_SYSFS_SHOW(_name, _attr_type) \ + static ssize_t _name##_show_common(struct kobject *kobj, \ + struct attribute *attr, char *buff) \ + { \ + u32 val = sysfs_gt_attribute_r_##_attr_type##_func(kobj, attr, \ + __##_name##_show); \ + \ + return sysfs_emit(buff, "%u\n", val); \ + } \ + static ssize_t _name##_show(struct kobject *kobj, \ + struct kobj_attribute *attr, char *buff) \ + { \ + return _name ##_show_common(kobj, &attr->attr, buff); \ + } \ + static ssize_t _name##_dev_show(struct device *dev, \ + struct device_attribute *attr, char *buff) \ + { \ + return _name##_show_common(&dev->kobj, &attr->attr, buff); \ + } + +#define INTEL_GT_SYSFS_STORE(_name, _func) \ + static ssize_t _name##_store_common(struct kobject *kobj, \ + struct attribute *attr, \ + const char *buff, size_t count) \ + { \ + int ret; \ + u32 val; \ + \ + ret = kstrtou32(buff, 0, &val); \ + if (ret) \ + return ret; \ + \ + ret = sysfs_gt_attribute_w_func(kobj, attr, _func, val); \ + \ + return ret ?: count; \ + } \ + static ssize_t _name##_store(struct kobject *kobj, \ + struct kobj_attribute *attr, const char *buff, \ + size_t count) \ + { \ + return _name##_store_common(kobj, &attr->attr, buff, count); \ + } \ + static ssize_t _name##_dev_store(struct device *dev, \ + struct device_attribute *attr, \ + const char *buff, size_t count) \ + { \ + return _name##_store_common(&dev->kobj, &attr->attr, buff, count); \ + } + +#define INTEL_GT_SYSFS_SHOW_MAX(_name) INTEL_GT_SYSFS_SHOW(_name, max) +#define INTEL_GT_SYSFS_SHOW_MIN(_name) INTEL_GT_SYSFS_SHOW(_name, min) + +#define INTEL_GT_ATTR_RW(_name) \ + static struct kobj_attribute attr_##_name = __ATTR_RW(_name) + +#define INTEL_GT_ATTR_RO(_name) \ + static struct kobj_attribute attr_##_name = __ATTR_RO(_name) + +#define INTEL_GT_DUAL_ATTR_RW(_name) \ + static struct device_attribute dev_attr_##_name = __ATTR(_name, 0644, \ + _name##_dev_show, \ + _name##_dev_store); \ + INTEL_GT_ATTR_RW(_name) + +#define INTEL_GT_DUAL_ATTR_RO(_name) \ + static struct device_attribute dev_attr_##_name = __ATTR(_name, 0444, \ + _name##_dev_show, \ + NULL); \ + INTEL_GT_ATTR_RO(_name) + #ifdef CONFIG_PM -static u32 get_residency(struct intel_gt *gt, i915_reg_t reg) +static u32 get_residency(struct intel_gt *gt, enum intel_rc6_res_type id) { intel_wakeref_t wakeref; u64 res = 0; with_intel_runtime_pm(gt->uncore->rpm, wakeref) - res = intel_rc6_residency_us(>->rc6, reg); + res = intel_rc6_residency_us(>->rc6, id); return DIV_ROUND_CLOSEST_ULL(res, 1000); } -static ssize_t rc6_enable_show(struct device *dev, - struct device_attribute *attr, - char *buff) +static u8 get_rc6_mask(struct intel_gt *gt) { - struct intel_gt *gt = intel_gt_sysfs_get_drvdata(dev, attr->attr.name); u8 mask = 0; if (HAS_RC6(gt->i915)) @@ -118,105 +187,105 @@ static ssize_t rc6_enable_show(struct device *dev, if (HAS_RC6pp(gt->i915)) mask |= BIT(2); - return sysfs_emit(buff, "%x\n", mask); + return mask; } -static u32 __rc6_residency_ms_show(struct intel_gt *gt) +static ssize_t rc6_enable_show(struct kobject *kobj, + struct kobj_attribute *attr, + char *buff) { - return get_residency(gt, GEN6_GT_GFX_RC6); + struct intel_gt *gt = intel_gt_sysfs_get_drvdata(kobj, attr->attr.name); + + return sysfs_emit(buff, "%x\n", get_rc6_mask(gt)); } -static ssize_t rc6_residency_ms_show(struct device *dev, - struct device_attribute *attr, - char *buff) +static ssize_t rc6_enable_dev_show(struct device *dev, + struct device_attribute *attr, + char *buff) { - u32 rc6_residency = sysfs_gt_attribute_r_min_func(dev, attr, - __rc6_residency_ms_show); + struct intel_gt *gt = intel_gt_sysfs_get_drvdata(&dev->kobj, attr->attr.name); - return sysfs_emit(buff, "%u\n", rc6_residency); + return sysfs_emit(buff, "%x\n", get_rc6_mask(gt)); } -static u32 __rc6p_residency_ms_show(struct intel_gt *gt) +static u32 __rc6_residency_ms_show(struct intel_gt *gt) { - return get_residency(gt, GEN6_GT_GFX_RC6p); + return get_residency(gt, INTEL_RC6_RES_RC6); } -static ssize_t rc6p_residency_ms_show(struct device *dev, - struct device_attribute *attr, - char *buff) +static u32 __rc6p_residency_ms_show(struct intel_gt *gt) { - u32 rc6p_residency = sysfs_gt_attribute_r_min_func(dev, attr, - __rc6p_residency_ms_show); - - return sysfs_emit(buff, "%u\n", rc6p_residency); + return get_residency(gt, INTEL_RC6_RES_RC6p); } static u32 __rc6pp_residency_ms_show(struct intel_gt *gt) { - return get_residency(gt, GEN6_GT_GFX_RC6pp); -} - -static ssize_t rc6pp_residency_ms_show(struct device *dev, - struct device_attribute *attr, - char *buff) -{ - u32 rc6pp_residency = sysfs_gt_attribute_r_min_func(dev, attr, - __rc6pp_residency_ms_show); - - return sysfs_emit(buff, "%u\n", rc6pp_residency); + return get_residency(gt, INTEL_RC6_RES_RC6pp); } static u32 __media_rc6_residency_ms_show(struct intel_gt *gt) { - return get_residency(gt, VLV_GT_MEDIA_RC6); + return get_residency(gt, INTEL_RC6_RES_VLV_MEDIA); } -static ssize_t media_rc6_residency_ms_show(struct device *dev, - struct device_attribute *attr, - char *buff) -{ - u32 rc6_residency = sysfs_gt_attribute_r_min_func(dev, attr, - __media_rc6_residency_ms_show); +INTEL_GT_SYSFS_SHOW_MIN(rc6_residency_ms); +INTEL_GT_SYSFS_SHOW_MIN(rc6p_residency_ms); +INTEL_GT_SYSFS_SHOW_MIN(rc6pp_residency_ms); +INTEL_GT_SYSFS_SHOW_MIN(media_rc6_residency_ms); - return sysfs_emit(buff, "%u\n", rc6_residency); -} - -static DEVICE_ATTR_RO(rc6_enable); -static DEVICE_ATTR_RO(rc6_residency_ms); -static DEVICE_ATTR_RO(rc6p_residency_ms); -static DEVICE_ATTR_RO(rc6pp_residency_ms); -static DEVICE_ATTR_RO(media_rc6_residency_ms); +INTEL_GT_DUAL_ATTR_RO(rc6_enable); +INTEL_GT_DUAL_ATTR_RO(rc6_residency_ms); +INTEL_GT_DUAL_ATTR_RO(rc6p_residency_ms); +INTEL_GT_DUAL_ATTR_RO(rc6pp_residency_ms); +INTEL_GT_DUAL_ATTR_RO(media_rc6_residency_ms); static struct attribute *rc6_attrs[] = { + &attr_rc6_enable.attr, + &attr_rc6_residency_ms.attr, + NULL +}; + +static struct attribute *rc6p_attrs[] = { + &attr_rc6p_residency_ms.attr, + &attr_rc6pp_residency_ms.attr, + NULL +}; + +static struct attribute *media_rc6_attrs[] = { + &attr_media_rc6_residency_ms.attr, + NULL +}; + +static struct attribute *rc6_dev_attrs[] = { &dev_attr_rc6_enable.attr, &dev_attr_rc6_residency_ms.attr, NULL }; -static struct attribute *rc6p_attrs[] = { +static struct attribute *rc6p_dev_attrs[] = { &dev_attr_rc6p_residency_ms.attr, &dev_attr_rc6pp_residency_ms.attr, NULL }; -static struct attribute *media_rc6_attrs[] = { +static struct attribute *media_rc6_dev_attrs[] = { &dev_attr_media_rc6_residency_ms.attr, NULL }; static const struct attribute_group rc6_attr_group[] = { { .attrs = rc6_attrs, }, - { .name = power_group_name, .attrs = rc6_attrs, }, + { .name = power_group_name, .attrs = rc6_dev_attrs, }, }; static const struct attribute_group rc6p_attr_group[] = { { .attrs = rc6p_attrs, }, - { .name = power_group_name, .attrs = rc6p_attrs, }, + { .name = power_group_name, .attrs = rc6p_dev_attrs, }, }; static const struct attribute_group media_rc6_attr_group[] = { { .attrs = media_rc6_attrs, }, - { .name = power_group_name, .attrs = media_rc6_attrs, }, + { .name = power_group_name, .attrs = media_rc6_dev_attrs, }, }; static int __intel_gt_sysfs_create_group(struct kobject *kobj, @@ -271,104 +340,34 @@ static u32 __act_freq_mhz_show(struct intel_gt *gt) return intel_rps_read_actual_frequency(>->rps); } -static ssize_t act_freq_mhz_show(struct device *dev, - struct device_attribute *attr, char *buff) -{ - u32 actual_freq = sysfs_gt_attribute_r_max_func(dev, attr, - __act_freq_mhz_show); - - return sysfs_emit(buff, "%u\n", actual_freq); -} - static u32 __cur_freq_mhz_show(struct intel_gt *gt) { return intel_rps_get_requested_frequency(>->rps); } -static ssize_t cur_freq_mhz_show(struct device *dev, - struct device_attribute *attr, char *buff) -{ - u32 cur_freq = sysfs_gt_attribute_r_max_func(dev, attr, - __cur_freq_mhz_show); - - return sysfs_emit(buff, "%u\n", cur_freq); -} - static u32 __boost_freq_mhz_show(struct intel_gt *gt) { return intel_rps_get_boost_frequency(>->rps); } -static ssize_t boost_freq_mhz_show(struct device *dev, - struct device_attribute *attr, - char *buff) -{ - u32 boost_freq = sysfs_gt_attribute_r_max_func(dev, attr, - __boost_freq_mhz_show); - - return sysfs_emit(buff, "%u\n", boost_freq); -} - static int __boost_freq_mhz_store(struct intel_gt *gt, u32 val) { return intel_rps_set_boost_frequency(>->rps, val); } -static ssize_t boost_freq_mhz_store(struct device *dev, - struct device_attribute *attr, - const char *buff, size_t count) -{ - ssize_t ret; - u32 val; - - ret = kstrtou32(buff, 0, &val); - if (ret) - return ret; - - return sysfs_gt_attribute_w_func(dev, attr, - __boost_freq_mhz_store, val) ?: count; -} - -static u32 __rp0_freq_mhz_show(struct intel_gt *gt) +static u32 __RP0_freq_mhz_show(struct intel_gt *gt) { return intel_rps_get_rp0_frequency(>->rps); } -static ssize_t RP0_freq_mhz_show(struct device *dev, - struct device_attribute *attr, char *buff) -{ - u32 rp0_freq = sysfs_gt_attribute_r_max_func(dev, attr, - __rp0_freq_mhz_show); - - return sysfs_emit(buff, "%u\n", rp0_freq); -} - -static u32 __rp1_freq_mhz_show(struct intel_gt *gt) -{ - return intel_rps_get_rp1_frequency(>->rps); -} - -static ssize_t RP1_freq_mhz_show(struct device *dev, - struct device_attribute *attr, char *buff) -{ - u32 rp1_freq = sysfs_gt_attribute_r_max_func(dev, attr, - __rp1_freq_mhz_show); - - return sysfs_emit(buff, "%u\n", rp1_freq); -} - -static u32 __rpn_freq_mhz_show(struct intel_gt *gt) +static u32 __RPn_freq_mhz_show(struct intel_gt *gt) { return intel_rps_get_rpn_frequency(>->rps); } -static ssize_t RPn_freq_mhz_show(struct device *dev, - struct device_attribute *attr, char *buff) +static u32 __RP1_freq_mhz_show(struct intel_gt *gt) { - u32 rpn_freq = sysfs_gt_attribute_r_max_func(dev, attr, - __rpn_freq_mhz_show); - - return sysfs_emit(buff, "%u\n", rpn_freq); + return intel_rps_get_rp1_frequency(>->rps); } static u32 __max_freq_mhz_show(struct intel_gt *gt) @@ -376,71 +375,21 @@ static u32 __max_freq_mhz_show(struct intel_gt *gt) return intel_rps_get_max_frequency(>->rps); } -static ssize_t max_freq_mhz_show(struct device *dev, - struct device_attribute *attr, char *buff) -{ - u32 max_freq = sysfs_gt_attribute_r_max_func(dev, attr, - __max_freq_mhz_show); - - return sysfs_emit(buff, "%u\n", max_freq); -} - static int __set_max_freq(struct intel_gt *gt, u32 val) { return intel_rps_set_max_frequency(>->rps, val); } -static ssize_t max_freq_mhz_store(struct device *dev, - struct device_attribute *attr, - const char *buff, size_t count) -{ - int ret; - u32 val; - - ret = kstrtou32(buff, 0, &val); - if (ret) - return ret; - - ret = sysfs_gt_attribute_w_func(dev, attr, __set_max_freq, val); - - return ret ?: count; -} - static u32 __min_freq_mhz_show(struct intel_gt *gt) { return intel_rps_get_min_frequency(>->rps); } -static ssize_t min_freq_mhz_show(struct device *dev, - struct device_attribute *attr, char *buff) -{ - u32 min_freq = sysfs_gt_attribute_r_min_func(dev, attr, - __min_freq_mhz_show); - - return sysfs_emit(buff, "%u\n", min_freq); -} - static int __set_min_freq(struct intel_gt *gt, u32 val) { return intel_rps_set_min_frequency(>->rps, val); } -static ssize_t min_freq_mhz_store(struct device *dev, - struct device_attribute *attr, - const char *buff, size_t count) -{ - int ret; - u32 val; - - ret = kstrtou32(buff, 0, &val); - if (ret) - return ret; - - ret = sysfs_gt_attribute_w_func(dev, attr, __set_min_freq, val); - - return ret ?: count; -} - static u32 __vlv_rpe_freq_mhz_show(struct intel_gt *gt) { struct intel_rps *rps = >->rps; @@ -448,23 +397,31 @@ static u32 __vlv_rpe_freq_mhz_show(struct intel_gt *gt) return intel_gpu_freq(rps, rps->efficient_freq); } -static ssize_t vlv_rpe_freq_mhz_show(struct device *dev, - struct device_attribute *attr, char *buff) -{ - u32 rpe_freq = sysfs_gt_attribute_r_max_func(dev, attr, - __vlv_rpe_freq_mhz_show); - - return sysfs_emit(buff, "%u\n", rpe_freq); -} - -#define INTEL_GT_RPS_SYSFS_ATTR(_name, _mode, _show, _store) \ - static struct device_attribute dev_attr_gt_##_name = __ATTR(gt_##_name, _mode, _show, _store); \ - static struct device_attribute dev_attr_rps_##_name = __ATTR(rps_##_name, _mode, _show, _store) - -#define INTEL_GT_RPS_SYSFS_ATTR_RO(_name) \ - INTEL_GT_RPS_SYSFS_ATTR(_name, 0444, _name##_show, NULL) -#define INTEL_GT_RPS_SYSFS_ATTR_RW(_name) \ - INTEL_GT_RPS_SYSFS_ATTR(_name, 0644, _name##_show, _name##_store) +INTEL_GT_SYSFS_SHOW_MAX(act_freq_mhz); +INTEL_GT_SYSFS_SHOW_MAX(boost_freq_mhz); +INTEL_GT_SYSFS_SHOW_MAX(cur_freq_mhz); +INTEL_GT_SYSFS_SHOW_MAX(RP0_freq_mhz); +INTEL_GT_SYSFS_SHOW_MAX(RP1_freq_mhz); +INTEL_GT_SYSFS_SHOW_MAX(RPn_freq_mhz); +INTEL_GT_SYSFS_SHOW_MAX(max_freq_mhz); +INTEL_GT_SYSFS_SHOW_MIN(min_freq_mhz); +INTEL_GT_SYSFS_SHOW_MAX(vlv_rpe_freq_mhz); +INTEL_GT_SYSFS_STORE(boost_freq_mhz, __boost_freq_mhz_store); +INTEL_GT_SYSFS_STORE(max_freq_mhz, __set_max_freq); +INTEL_GT_SYSFS_STORE(min_freq_mhz, __set_min_freq); + +#define INTEL_GT_RPS_SYSFS_ATTR(_name, _mode, _show, _store, _show_dev, _store_dev) \ + static struct device_attribute dev_attr_gt_##_name = __ATTR(gt_##_name, _mode, \ + _show_dev, _store_dev); \ + static struct kobj_attribute attr_rps_##_name = __ATTR(rps_##_name, _mode, \ + _show, _store) + +#define INTEL_GT_RPS_SYSFS_ATTR_RO(_name) \ + INTEL_GT_RPS_SYSFS_ATTR(_name, 0444, _name##_show, NULL, \ + _name##_dev_show, NULL) +#define INTEL_GT_RPS_SYSFS_ATTR_RW(_name) \ + INTEL_GT_RPS_SYSFS_ATTR(_name, 0644, _name##_show, _name##_store, \ + _name##_dev_show, _name##_dev_store) /* The below macros generate static structures */ INTEL_GT_RPS_SYSFS_ATTR_RO(act_freq_mhz); @@ -475,32 +432,31 @@ INTEL_GT_RPS_SYSFS_ATTR_RO(RP1_freq_mhz); INTEL_GT_RPS_SYSFS_ATTR_RO(RPn_freq_mhz); INTEL_GT_RPS_SYSFS_ATTR_RW(max_freq_mhz); INTEL_GT_RPS_SYSFS_ATTR_RW(min_freq_mhz); - -static DEVICE_ATTR_RO(vlv_rpe_freq_mhz); - -#define GEN6_ATTR(s) { \ - &dev_attr_##s##_act_freq_mhz.attr, \ - &dev_attr_##s##_cur_freq_mhz.attr, \ - &dev_attr_##s##_boost_freq_mhz.attr, \ - &dev_attr_##s##_max_freq_mhz.attr, \ - &dev_attr_##s##_min_freq_mhz.attr, \ - &dev_attr_##s##_RP0_freq_mhz.attr, \ - &dev_attr_##s##_RP1_freq_mhz.attr, \ - &dev_attr_##s##_RPn_freq_mhz.attr, \ +INTEL_GT_RPS_SYSFS_ATTR_RO(vlv_rpe_freq_mhz); + +#define GEN6_ATTR(p, s) { \ + &p##attr_##s##_act_freq_mhz.attr, \ + &p##attr_##s##_cur_freq_mhz.attr, \ + &p##attr_##s##_boost_freq_mhz.attr, \ + &p##attr_##s##_max_freq_mhz.attr, \ + &p##attr_##s##_min_freq_mhz.attr, \ + &p##attr_##s##_RP0_freq_mhz.attr, \ + &p##attr_##s##_RP1_freq_mhz.attr, \ + &p##attr_##s##_RPn_freq_mhz.attr, \ NULL, \ } -#define GEN6_RPS_ATTR GEN6_ATTR(rps) -#define GEN6_GT_ATTR GEN6_ATTR(gt) +#define GEN6_RPS_ATTR GEN6_ATTR(, rps) +#define GEN6_GT_ATTR GEN6_ATTR(dev_, gt) static const struct attribute * const gen6_rps_attrs[] = GEN6_RPS_ATTR; static const struct attribute * const gen6_gt_attrs[] = GEN6_GT_ATTR; -static ssize_t punit_req_freq_mhz_show(struct device *dev, - struct device_attribute *attr, +static ssize_t punit_req_freq_mhz_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buff) { - struct intel_gt *gt = intel_gt_sysfs_get_drvdata(dev, attr->attr.name); + struct intel_gt *gt = intel_gt_sysfs_get_drvdata(kobj, attr->attr.name); u32 preq = intel_rps_read_punit_req_frequency(>->rps); return sysfs_emit(buff, "%u\n", preq); @@ -508,20 +464,20 @@ static ssize_t punit_req_freq_mhz_show(struct device *dev, struct intel_gt_bool_throttle_attr { struct attribute attr; - ssize_t (*show)(struct device *dev, struct device_attribute *attr, + ssize_t (*show)(struct kobject *kobj, struct kobj_attribute *attr, char *buf); - i915_reg_t reg32; + i915_reg_t (*reg32)(struct intel_gt *gt); u32 mask; }; -static ssize_t throttle_reason_bool_show(struct device *dev, - struct device_attribute *attr, +static ssize_t throttle_reason_bool_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buff) { - struct intel_gt *gt = intel_gt_sysfs_get_drvdata(dev, attr->attr.name); + struct intel_gt *gt = intel_gt_sysfs_get_drvdata(kobj, attr->attr.name); struct intel_gt_bool_throttle_attr *t_attr = (struct intel_gt_bool_throttle_attr *) attr; - bool val = rps_read_mask_mmio(>->rps, t_attr->reg32, t_attr->mask); + bool val = rps_read_mask_mmio(>->rps, t_attr->reg32(gt), t_attr->mask); return sysfs_emit(buff, "%u\n", val); } @@ -530,11 +486,11 @@ static ssize_t throttle_reason_bool_show(struct device *dev, struct intel_gt_bool_throttle_attr attr_##sysfs_func__ = { \ .attr = { .name = __stringify(sysfs_func__), .mode = 0444 }, \ .show = throttle_reason_bool_show, \ - .reg32 = GT0_PERF_LIMIT_REASONS, \ + .reg32 = intel_gt_perf_limit_reasons_reg, \ .mask = mask__, \ } -static DEVICE_ATTR_RO(punit_req_freq_mhz); +INTEL_GT_ATTR_RO(punit_req_freq_mhz); static INTEL_GT_RPS_BOOL_ATTR_RO(throttle_reason_status, GT0_PERF_LIMIT_REASONS_MASK); static INTEL_GT_RPS_BOOL_ATTR_RO(throttle_reason_pl1, POWER_LIMIT_1_MASK); static INTEL_GT_RPS_BOOL_ATTR_RO(throttle_reason_pl2, POWER_LIMIT_2_MASK); @@ -597,8 +553,8 @@ static const struct attribute *throttle_reason_attrs[] = { #define U8_8_VAL_MASK 0xffff #define U8_8_SCALE_TO_VALUE "0.00390625" -static ssize_t freq_factor_scale_show(struct device *dev, - struct device_attribute *attr, +static ssize_t freq_factor_scale_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buff) { return sysfs_emit(buff, "%s\n", U8_8_SCALE_TO_VALUE); @@ -610,11 +566,11 @@ static u32 media_ratio_mode_to_factor(u32 mode) return !mode ? mode : 256 / mode; } -static ssize_t media_freq_factor_show(struct device *dev, - struct device_attribute *attr, +static ssize_t media_freq_factor_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buff) { - struct intel_gt *gt = intel_gt_sysfs_get_drvdata(dev, attr->attr.name); + struct intel_gt *gt = intel_gt_sysfs_get_drvdata(kobj, attr->attr.name); struct intel_guc_slpc *slpc = >->uc.guc.slpc; intel_wakeref_t wakeref; u32 mode; @@ -641,11 +597,11 @@ static ssize_t media_freq_factor_show(struct device *dev, return sysfs_emit(buff, "%u\n", media_ratio_mode_to_factor(mode)); } -static ssize_t media_freq_factor_store(struct device *dev, - struct device_attribute *attr, +static ssize_t media_freq_factor_store(struct kobject *kobj, + struct kobj_attribute *attr, const char *buff, size_t count) { - struct intel_gt *gt = intel_gt_sysfs_get_drvdata(dev, attr->attr.name); + struct intel_gt *gt = intel_gt_sysfs_get_drvdata(kobj, attr->attr.name); struct intel_guc_slpc *slpc = >->uc.guc.slpc; u32 factor, mode; int err; @@ -670,11 +626,11 @@ static ssize_t media_freq_factor_store(struct device *dev, return err ?: count; } -static ssize_t media_RP0_freq_mhz_show(struct device *dev, - struct device_attribute *attr, +static ssize_t media_RP0_freq_mhz_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buff) { - struct intel_gt *gt = intel_gt_sysfs_get_drvdata(dev, attr->attr.name); + struct intel_gt *gt = intel_gt_sysfs_get_drvdata(kobj, attr->attr.name); u32 val; int err; @@ -691,11 +647,11 @@ static ssize_t media_RP0_freq_mhz_show(struct device *dev, return sysfs_emit(buff, "%u\n", val); } -static ssize_t media_RPn_freq_mhz_show(struct device *dev, - struct device_attribute *attr, +static ssize_t media_RPn_freq_mhz_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buff) { - struct intel_gt *gt = intel_gt_sysfs_get_drvdata(dev, attr->attr.name); + struct intel_gt *gt = intel_gt_sysfs_get_drvdata(kobj, attr->attr.name); u32 val; int err; @@ -712,17 +668,17 @@ static ssize_t media_RPn_freq_mhz_show(struct device *dev, return sysfs_emit(buff, "%u\n", val); } -static DEVICE_ATTR_RW(media_freq_factor); -static struct device_attribute dev_attr_media_freq_factor_scale = +INTEL_GT_ATTR_RW(media_freq_factor); +static struct kobj_attribute attr_media_freq_factor_scale = __ATTR(media_freq_factor.scale, 0444, freq_factor_scale_show, NULL); -static DEVICE_ATTR_RO(media_RP0_freq_mhz); -static DEVICE_ATTR_RO(media_RPn_freq_mhz); +INTEL_GT_ATTR_RO(media_RP0_freq_mhz); +INTEL_GT_ATTR_RO(media_RPn_freq_mhz); static const struct attribute *media_perf_power_attrs[] = { - &dev_attr_media_freq_factor.attr, - &dev_attr_media_freq_factor_scale.attr, - &dev_attr_media_RP0_freq_mhz.attr, - &dev_attr_media_RPn_freq_mhz.attr, + &attr_media_freq_factor.attr, + &attr_media_freq_factor_scale.attr, + &attr_media_RP0_freq_mhz.attr, + &attr_media_RPn_freq_mhz.attr, NULL }; @@ -754,20 +710,29 @@ static const struct attribute * const rps_defaults_attrs[] = { NULL }; -static int intel_sysfs_rps_init(struct intel_gt *gt, struct kobject *kobj, - const struct attribute * const *attrs) +static int intel_sysfs_rps_init(struct intel_gt *gt, struct kobject *kobj) { + const struct attribute * const *attrs; + struct attribute *vlv_attr; int ret; if (GRAPHICS_VER(gt->i915) < 6) return 0; + if (is_object_gt(kobj)) { + attrs = gen6_rps_attrs; + vlv_attr = &attr_rps_vlv_rpe_freq_mhz.attr; + } else { + attrs = gen6_gt_attrs; + vlv_attr = &dev_attr_gt_vlv_rpe_freq_mhz.attr; + } + ret = sysfs_create_files(kobj, attrs); if (ret) return ret; if (IS_VALLEYVIEW(gt->i915) || IS_CHERRYVIEW(gt->i915)) - ret = sysfs_create_file(kobj, &dev_attr_vlv_rpe_freq_mhz.attr); + ret = sysfs_create_file(kobj, vlv_attr); return ret; } @@ -778,9 +743,7 @@ void intel_gt_sysfs_pm_init(struct intel_gt *gt, struct kobject *kobj) intel_sysfs_rc6_init(gt, kobj); - ret = is_object_gt(kobj) ? - intel_sysfs_rps_init(gt, kobj, gen6_rps_attrs) : - intel_sysfs_rps_init(gt, kobj, gen6_gt_attrs); + ret = intel_sysfs_rps_init(gt, kobj); if (ret) drm_warn(>->i915->drm, "failed to create gt%u RPS sysfs files (%pe)", @@ -790,13 +753,13 @@ void intel_gt_sysfs_pm_init(struct intel_gt *gt, struct kobject *kobj) if (!is_object_gt(kobj)) return; - ret = sysfs_create_file(kobj, &dev_attr_punit_req_freq_mhz.attr); + ret = sysfs_create_file(kobj, &attr_punit_req_freq_mhz.attr); if (ret) drm_warn(>->i915->drm, "failed to create gt%u punit_req_freq_mhz sysfs (%pe)", gt->info.id, ERR_PTR(ret)); - if (GRAPHICS_VER(gt->i915) >= 11) { + if (i915_mmio_reg_valid(intel_gt_perf_limit_reasons_reg(gt))) { ret = sysfs_create_files(kobj, throttle_reason_attrs); if (ret) drm_warn(>->i915->drm, diff --git a/drivers/gpu/drm/i915/gt/intel_gt_types.h b/drivers/gpu/drm/i915/gt/intel_gt_types.h index f19c2de77ff6..c1d9cd255e06 100644 --- a/drivers/gpu/drm/i915/gt/intel_gt_types.h +++ b/drivers/gpu/drm/i915/gt/intel_gt_types.h @@ -20,6 +20,7 @@ #include "intel_gsc.h" #include "i915_vma.h" +#include "i915_perf_types.h" #include "intel_engine_types.h" #include "intel_gt_buffer_pool_types.h" #include "intel_hwconfig.h" @@ -30,6 +31,7 @@ #include "intel_migrate_types.h" #include "intel_wakeref.h" #include "pxp/intel_pxp_types.h" +#include "intel_wopcm.h" struct drm_i915_private; struct i915_ggtt; @@ -59,6 +61,9 @@ enum intel_steering_type { L3BANK, MSLICE, LNCF, + GAM, + DSS, + OADDRM, /* * On some platforms there are multiple types of MCR registers that @@ -97,6 +102,7 @@ struct intel_gt { struct intel_uc uc; struct intel_gsc gsc; + struct intel_wopcm wopcm; struct { /* Serialize global tlb invalidations */ @@ -141,20 +147,6 @@ struct intel_gt { struct intel_wakeref wakeref; atomic_t user_wakeref; - /** - * Protects access to lmem usefault list. - * It is required, if we are outside of the runtime suspend path, - * access to @lmem_userfault_list requires always first grabbing the - * runtime pm, to ensure we can't race against runtime suspend. - * Once we have that we also need to grab @lmem_userfault_lock, - * at which point we have exclusive access. - * The runtime suspend path is special since it doesn't really hold any locks, - * but instead has exclusive access by virtue of all other accesses requiring - * holding the runtime pm wakeref. - */ - struct mutex lmem_userfault_lock; - struct list_head lmem_userfault_list; - struct list_head closed_vma; spinlock_t closed_lock; /* guards the list of closed_vma */ @@ -170,9 +162,6 @@ struct intel_gt { */ intel_wakeref_t awake; - /* Manual runtime pm autosuspend delay for user GGTT/lmem mmaps */ - struct intel_wakeref_auto userfault_wakeref; - u32 clock_frequency; u32 clock_period_ns; @@ -286,6 +275,8 @@ struct intel_gt { /* sysfs defaults per gt */ struct gt_defaults defaults; struct kobject *sysfs_defaults; + + struct i915_perf_gt perf; }; struct intel_gt_definition { diff --git a/drivers/gpu/drm/i915/gt/intel_gtt.c b/drivers/gpu/drm/i915/gt/intel_gtt.c index 2eaeba14319e..2ba3983984b9 100644 --- a/drivers/gpu/drm/i915/gt/intel_gtt.c +++ b/drivers/gpu/drm/i915/gt/intel_gtt.c @@ -12,9 +12,11 @@ #include "gem/i915_gem_internal.h" #include "gem/i915_gem_lmem.h" +#include "i915_reg.h" #include "i915_trace.h" #include "i915_utils.h" #include "intel_gt.h" +#include "intel_gt_mcr.h" #include "intel_gt_regs.h" #include "intel_gtt.h" @@ -269,11 +271,7 @@ void i915_address_space_init(struct i915_address_space *vm, int subclass) memset64(vm->min_alignment, I915_GTT_MIN_ALIGNMENT, ARRAY_SIZE(vm->min_alignment)); - if (HAS_64K_PAGES(vm->i915) && NEEDS_COMPACT_PT(vm->i915) && - subclass == VM_CLASS_PPGTT) { - vm->min_alignment[INTEL_MEMORY_LOCAL] = I915_GTT_PAGE_SIZE_2M; - vm->min_alignment[INTEL_MEMORY_STOLEN_LOCAL] = I915_GTT_PAGE_SIZE_2M; - } else if (HAS_64K_PAGES(vm->i915)) { + if (HAS_64K_PAGES(vm->i915)) { vm->min_alignment[INTEL_MEMORY_LOCAL] = I915_GTT_PAGE_SIZE_64K; vm->min_alignment[INTEL_MEMORY_STOLEN_LOCAL] = I915_GTT_PAGE_SIZE_64K; } @@ -343,7 +341,8 @@ int setup_scratch_page(struct i915_address_space *vm) */ size = I915_GTT_PAGE_SIZE_4K; if (i915_vm_is_4lvl(vm) && - HAS_PAGE_SIZES(vm->i915, I915_GTT_PAGE_SIZE_64K)) + HAS_PAGE_SIZES(vm->i915, I915_GTT_PAGE_SIZE_64K) && + !HAS_64K_PAGES(vm->i915)) size = I915_GTT_PAGE_SIZE_64K; do { @@ -385,18 +384,6 @@ skip: if (size == I915_GTT_PAGE_SIZE_4K) return -ENOMEM; - /* - * If we need 64K minimum GTT pages for device local-memory, - * like on XEHPSDV, then we need to fail the allocation here, - * otherwise we can't safely support the insertion of - * local-memory pages for this vm, since the HW expects the - * correct physical alignment and size when the page-table is - * operating in 64K GTT mode, which includes any scratch PTEs, - * since userspace can still touch them. - */ - if (HAS_64K_PAGES(vm->i915)) - return -ENOMEM; - size = I915_GTT_PAGE_SIZE_4K; } while (1); } @@ -493,6 +480,18 @@ static void tgl_setup_private_ppat(struct intel_uncore *uncore) intel_uncore_write(uncore, GEN12_PAT_INDEX(7), GEN8_PPAT_WB); } +static void xehp_setup_private_ppat(struct intel_gt *gt) +{ + intel_gt_mcr_multicast_write(gt, XEHP_PAT_INDEX(0), GEN8_PPAT_WB); + intel_gt_mcr_multicast_write(gt, XEHP_PAT_INDEX(1), GEN8_PPAT_WC); + intel_gt_mcr_multicast_write(gt, XEHP_PAT_INDEX(2), GEN8_PPAT_WT); + intel_gt_mcr_multicast_write(gt, XEHP_PAT_INDEX(3), GEN8_PPAT_UC); + intel_gt_mcr_multicast_write(gt, XEHP_PAT_INDEX(4), GEN8_PPAT_WB); + intel_gt_mcr_multicast_write(gt, XEHP_PAT_INDEX(5), GEN8_PPAT_WB); + intel_gt_mcr_multicast_write(gt, XEHP_PAT_INDEX(6), GEN8_PPAT_WB); + intel_gt_mcr_multicast_write(gt, XEHP_PAT_INDEX(7), GEN8_PPAT_WB); +} + static void icl_setup_private_ppat(struct intel_uncore *uncore) { intel_uncore_write(uncore, @@ -585,13 +584,16 @@ static void chv_setup_private_ppat(struct intel_uncore *uncore) intel_uncore_write(uncore, GEN8_PRIVATE_PAT_HI, upper_32_bits(pat)); } -void setup_private_pat(struct intel_uncore *uncore) +void setup_private_pat(struct intel_gt *gt) { - struct drm_i915_private *i915 = uncore->i915; + struct intel_uncore *uncore = gt->uncore; + struct drm_i915_private *i915 = gt->i915; GEM_BUG_ON(GRAPHICS_VER(i915) < 8); - if (GRAPHICS_VER(i915) >= 12) + if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) + xehp_setup_private_ppat(gt); + else if (GRAPHICS_VER(i915) >= 12) tgl_setup_private_ppat(uncore); else if (GRAPHICS_VER(i915) >= 11) icl_setup_private_ppat(uncore); diff --git a/drivers/gpu/drm/i915/gt/intel_gtt.h b/drivers/gpu/drm/i915/gt/intel_gtt.h index c0ca53cba9f0..4d75ba4bb41d 100644 --- a/drivers/gpu/drm/i915/gt/intel_gtt.h +++ b/drivers/gpu/drm/i915/gt/intel_gtt.h @@ -93,6 +93,7 @@ typedef u64 gen8_pte_t; #define GEN12_GGTT_PTE_LM BIT_ULL(1) #define GEN12_PDE_64K BIT(6) +#define GEN12_PTE_PS64 BIT(8) /* * Cacheability Control is a 4-bit value. The low three bits are stored in bits @@ -667,7 +668,7 @@ void ppgtt_unbind_vma(struct i915_address_space *vm, void gtt_write_workarounds(struct intel_gt *gt); -void setup_private_pat(struct intel_uncore *uncore); +void setup_private_pat(struct intel_gt *gt); int i915_vm_alloc_pt_stash(struct i915_address_space *vm, struct i915_vm_pt_stash *stash, diff --git a/drivers/gpu/drm/i915/gt/intel_lrc.c b/drivers/gpu/drm/i915/gt/intel_lrc.c index 3955292483a6..7771a19008c6 100644 --- a/drivers/gpu/drm/i915/gt/intel_lrc.c +++ b/drivers/gpu/drm/i915/gt/intel_lrc.c @@ -20,6 +20,30 @@ #include "intel_ring.h" #include "shmem_utils.h" +/* + * The per-platform tables are u8-encoded in @data. Decode @data and set the + * addresses' offset and commands in @regs. The following encoding is used + * for each byte. There are 2 steps: decoding commands and decoding addresses. + * + * Commands: + * [7]: create NOPs - number of NOPs are set in lower bits + * [6]: When creating MI_LOAD_REGISTER_IMM command, allow to set + * MI_LRI_FORCE_POSTED + * [5:0]: Number of NOPs or registers to set values to in case of + * MI_LOAD_REGISTER_IMM + * + * Addresses: these are decoded after a MI_LOAD_REGISTER_IMM command by "count" + * number of registers. They are set by using the REG/REG16 macros: the former + * is used for offsets smaller than 0x200 while the latter is for values bigger + * than that. Those macros already set all the bits documented below correctly: + * + * [7]: When a register offset needs more than 6 bits, use additional bytes, to + * follow, for the lower bits + * [6:0]: Register offset, without considering the engine base. + * + * This function only tweaks the commands and register offsets. Values are not + * filled out. + */ static void set_offsets(u32 *regs, const u8 *data, const struct intel_engine_cs *engine, @@ -264,6 +288,39 @@ static const u8 dg2_xcs_offsets[] = { END }; +static const u8 mtl_xcs_offsets[] = { + NOP(1), + LRI(13, POSTED), + REG16(0x244), + REG(0x034), + REG(0x030), + REG(0x038), + REG(0x03c), + REG(0x168), + REG(0x140), + REG(0x110), + REG(0x1c0), + REG(0x1c4), + REG(0x1c8), + REG(0x180), + REG16(0x2b4), + NOP(4), + + NOP(1), + LRI(9, POSTED), + REG16(0x3a8), + REG16(0x28c), + REG16(0x288), + REG16(0x284), + REG16(0x280), + REG16(0x27c), + REG16(0x278), + REG16(0x274), + REG16(0x270), + + END +}; + static const u8 gen8_rcs_offsets[] = { NOP(1), LRI(14, POSTED), @@ -606,6 +663,49 @@ static const u8 dg2_rcs_offsets[] = { END }; +static const u8 mtl_rcs_offsets[] = { + NOP(1), + LRI(15, POSTED), + REG16(0x244), + REG(0x034), + REG(0x030), + REG(0x038), + REG(0x03c), + REG(0x168), + REG(0x140), + REG(0x110), + REG(0x1c0), + REG(0x1c4), + REG(0x1c8), + REG(0x180), + REG16(0x2b4), + REG(0x120), + REG(0x124), + + NOP(1), + LRI(9, POSTED), + REG16(0x3a8), + REG16(0x28c), + REG16(0x288), + REG16(0x284), + REG16(0x280), + REG16(0x27c), + REG16(0x278), + REG16(0x274), + REG16(0x270), + + NOP(2), + LRI(2, POSTED), + REG16(0x5a8), + REG16(0x5ac), + + NOP(6), + LRI(1, 0), + REG(0x0c8), + + END +}; + #undef END #undef REG16 #undef REG @@ -624,7 +724,9 @@ static const u8 *reg_offsets(const struct intel_engine_cs *engine) !intel_engine_has_relative_mmio(engine)); if (engine->flags & I915_ENGINE_HAS_RCS_REG_STATE) { - if (GRAPHICS_VER_FULL(engine->i915) >= IP_VER(12, 55)) + if (GRAPHICS_VER_FULL(engine->i915) >= IP_VER(12, 70)) + return mtl_rcs_offsets; + else if (GRAPHICS_VER_FULL(engine->i915) >= IP_VER(12, 55)) return dg2_rcs_offsets; else if (GRAPHICS_VER_FULL(engine->i915) >= IP_VER(12, 50)) return xehp_rcs_offsets; @@ -637,7 +739,9 @@ static const u8 *reg_offsets(const struct intel_engine_cs *engine) else return gen8_rcs_offsets; } else { - if (GRAPHICS_VER_FULL(engine->i915) >= IP_VER(12, 55)) + if (GRAPHICS_VER_FULL(engine->i915) >= IP_VER(12, 70)) + return mtl_xcs_offsets; + else if (GRAPHICS_VER_FULL(engine->i915) >= IP_VER(12, 55)) return dg2_xcs_offsets; else if (GRAPHICS_VER(engine->i915) >= 12) return gen12_xcs_offsets; @@ -745,19 +849,18 @@ static int lrc_ring_cmd_buf_cctl(const struct intel_engine_cs *engine) static u32 lrc_ring_indirect_offset_default(const struct intel_engine_cs *engine) { - switch (GRAPHICS_VER(engine->i915)) { - default: - MISSING_CASE(GRAPHICS_VER(engine->i915)); - fallthrough; - case 12: + if (GRAPHICS_VER(engine->i915) >= 12) return GEN12_CTX_RCS_INDIRECT_CTX_OFFSET_DEFAULT; - case 11: + else if (GRAPHICS_VER(engine->i915) >= 11) return GEN11_CTX_RCS_INDIRECT_CTX_OFFSET_DEFAULT; - case 9: + else if (GRAPHICS_VER(engine->i915) >= 9) return GEN9_CTX_RCS_INDIRECT_CTX_OFFSET_DEFAULT; - case 8: + else if (GRAPHICS_VER(engine->i915) >= 8) return GEN8_CTX_RCS_INDIRECT_CTX_OFFSET_DEFAULT; - } + + GEM_BUG_ON(GRAPHICS_VER(engine->i915) < 8); + + return 0; } static void @@ -1012,7 +1115,7 @@ __lrc_alloc_state(struct intel_context *ce, struct intel_engine_cs *engine) if (IS_ENABLED(CONFIG_DRM_I915_DEBUG_GEM)) context_size += I915_GTT_PAGE_SIZE; /* for redzone */ - if (GRAPHICS_VER(engine->i915) == 12) { + if (GRAPHICS_VER(engine->i915) >= 12) { ce->wa_bb_page = context_size / PAGE_SIZE; context_size += PAGE_SIZE; } @@ -1718,24 +1821,16 @@ void lrc_init_wa_ctx(struct intel_engine_cs *engine) unsigned int i; int err; - if (!(engine->flags & I915_ENGINE_HAS_RCS_REG_STATE)) + if (GRAPHICS_VER(engine->i915) >= 11 || + !(engine->flags & I915_ENGINE_HAS_RCS_REG_STATE)) return; - switch (GRAPHICS_VER(engine->i915)) { - case 12: - case 11: - return; - case 9: + if (GRAPHICS_VER(engine->i915) == 9) { wa_bb_fn[0] = gen9_init_indirectctx_bb; wa_bb_fn[1] = NULL; - break; - case 8: + } else if (GRAPHICS_VER(engine->i915) == 8) { wa_bb_fn[0] = gen8_init_indirectctx_bb; wa_bb_fn[1] = NULL; - break; - default: - MISSING_CASE(GRAPHICS_VER(engine->i915)); - return; } err = lrc_create_wa_ctx(engine); diff --git a/drivers/gpu/drm/i915/gt/intel_lrc.h b/drivers/gpu/drm/i915/gt/intel_lrc.h index a390f0813c8b..7111bae759f3 100644 --- a/drivers/gpu/drm/i915/gt/intel_lrc.h +++ b/drivers/gpu/drm/i915/gt/intel_lrc.h @@ -110,6 +110,8 @@ enum { #define XEHP_SW_CTX_ID_WIDTH 16 #define XEHP_SW_COUNTER_SHIFT 58 #define XEHP_SW_COUNTER_WIDTH 6 +#define GEN12_GUC_SW_CTX_ID_SHIFT 39 +#define GEN12_GUC_SW_CTX_ID_WIDTH 16 static inline void lrc_runtime_start(struct intel_context *ce) { diff --git a/drivers/gpu/drm/i915/gt/intel_migrate.c b/drivers/gpu/drm/i915/gt/intel_migrate.c index aaaf1906026c..b405a04135ca 100644 --- a/drivers/gpu/drm/i915/gt/intel_migrate.c +++ b/drivers/gpu/drm/i915/gt/intel_migrate.c @@ -10,6 +10,7 @@ #include "intel_gtt.h" #include "intel_migrate.h" #include "intel_ring.h" +#include "gem/i915_gem_lmem.h" struct insert_pte_data { u64 offset; diff --git a/drivers/gpu/drm/i915/gt/intel_mocs.c b/drivers/gpu/drm/i915/gt/intel_mocs.c index 152244d7f62a..49fdd509527a 100644 --- a/drivers/gpu/drm/i915/gt/intel_mocs.c +++ b/drivers/gpu/drm/i915/gt/intel_mocs.c @@ -7,6 +7,7 @@ #include "intel_engine.h" #include "intel_gt.h" +#include "intel_gt_mcr.h" #include "intel_gt_regs.h" #include "intel_mocs.h" #include "intel_ring.h" @@ -609,14 +610,17 @@ static u32 l3cc_combine(u16 low, u16 high) 0; \ i++) -static void init_l3cc_table(struct intel_uncore *uncore, +static void init_l3cc_table(struct intel_gt *gt, const struct drm_i915_mocs_table *table) { unsigned int i; u32 l3cc; for_each_l3cc(l3cc, table, i) - intel_uncore_write_fw(uncore, GEN9_LNCFCMOCS(i), l3cc); + if (GRAPHICS_VER_FULL(gt->i915) >= IP_VER(12, 50)) + intel_gt_mcr_multicast_write_fw(gt, XEHP_LNCFCMOCS(i), l3cc); + else + intel_uncore_write_fw(gt->uncore, GEN9_LNCFCMOCS(i), l3cc); } void intel_mocs_init_engine(struct intel_engine_cs *engine) @@ -636,7 +640,7 @@ void intel_mocs_init_engine(struct intel_engine_cs *engine) init_mocs_table(engine, &table); if (flags & HAS_RENDER_L3CC && engine->class == RENDER_CLASS) - init_l3cc_table(engine->uncore, &table); + init_l3cc_table(engine->gt, &table); } static u32 global_mocs_offset(void) @@ -672,7 +676,7 @@ void intel_mocs_init(struct intel_gt *gt) * memory transactions including guc transactions */ if (flags & HAS_RENDER_L3CC) - init_l3cc_table(gt->uncore, &table); + init_l3cc_table(gt, &table); } #if IS_ENABLED(CONFIG_DRM_I915_SELFTEST) diff --git a/drivers/gpu/drm/i915/gt/intel_rc6.c b/drivers/gpu/drm/i915/gt/intel_rc6.c index f8d0523f4c18..2ee4051e4d96 100644 --- a/drivers/gpu/drm/i915/gt/intel_rc6.c +++ b/drivers/gpu/drm/i915/gt/intel_rc6.c @@ -551,6 +551,23 @@ static void __intel_rc6_disable(struct intel_rc6 *rc6) intel_uncore_forcewake_put(uncore, FORCEWAKE_ALL); } +static void rc6_res_reg_init(struct intel_rc6 *rc6) +{ + memset(rc6->res_reg, INVALID_MMIO_REG.reg, sizeof(rc6->res_reg)); + + switch (rc6_to_gt(rc6)->type) { + case GT_MEDIA: + rc6->res_reg[INTEL_RC6_RES_RC6] = MTL_MEDIA_MC6; + break; + default: + rc6->res_reg[INTEL_RC6_RES_RC6_LOCKED] = GEN6_GT_GFX_RC6_LOCKED; + rc6->res_reg[INTEL_RC6_RES_RC6] = GEN6_GT_GFX_RC6; + rc6->res_reg[INTEL_RC6_RES_RC6p] = GEN6_GT_GFX_RC6p; + rc6->res_reg[INTEL_RC6_RES_RC6pp] = GEN6_GT_GFX_RC6pp; + break; + } +} + void intel_rc6_init(struct intel_rc6 *rc6) { struct drm_i915_private *i915 = rc6_to_i915(rc6); @@ -562,6 +579,8 @@ void intel_rc6_init(struct intel_rc6 *rc6) if (!rc6_supported(rc6)) return; + rc6_res_reg_init(rc6); + if (IS_CHERRYVIEW(i915)) err = chv_rc6_init(rc6); else if (IS_VALLEYVIEW(i915)) @@ -736,31 +755,19 @@ static u64 vlv_residency_raw(struct intel_uncore *uncore, const i915_reg_t reg) return lower | (u64)upper << 8; } -u64 intel_rc6_residency_ns(struct intel_rc6 *rc6, const i915_reg_t reg) +u64 intel_rc6_residency_ns(struct intel_rc6 *rc6, enum intel_rc6_res_type id) { struct drm_i915_private *i915 = rc6_to_i915(rc6); struct intel_uncore *uncore = rc6_to_uncore(rc6); u64 time_hw, prev_hw, overflow_hw; + i915_reg_t reg = rc6->res_reg[id]; unsigned int fw_domains; unsigned long flags; - unsigned int i; u32 mul, div; if (!rc6->supported) return 0; - /* - * Store previous hw counter values for counter wrap-around handling. - * - * There are only four interesting registers and they live next to each - * other so we can use the relative address, compared to the smallest - * one as the index into driver storage. - */ - i = (i915_mmio_reg_offset(reg) - - i915_mmio_reg_offset(GEN6_GT_GFX_RC6_LOCKED)) / sizeof(u32); - if (drm_WARN_ON_ONCE(&i915->drm, i >= ARRAY_SIZE(rc6->cur_residency))) - return 0; - fw_domains = intel_uncore_forcewake_for_reg(uncore, reg, FW_REG_READ); spin_lock_irqsave(&uncore->lock, flags); @@ -789,11 +796,11 @@ u64 intel_rc6_residency_ns(struct intel_rc6 *rc6, const i915_reg_t reg) /* * Counter wrap handling. * - * But relying on a sufficient frequency of queries otherwise counters - * can still wrap. + * Store previous hw counter values for counter wrap-around handling. But + * relying on a sufficient frequency of queries otherwise counters can still wrap. */ - prev_hw = rc6->prev_hw_residency[i]; - rc6->prev_hw_residency[i] = time_hw; + prev_hw = rc6->prev_hw_residency[id]; + rc6->prev_hw_residency[id] = time_hw; /* RC6 delta from last sample. */ if (time_hw >= prev_hw) @@ -802,8 +809,8 @@ u64 intel_rc6_residency_ns(struct intel_rc6 *rc6, const i915_reg_t reg) time_hw += overflow_hw - prev_hw; /* Add delta to RC6 extended raw driver copy. */ - time_hw += rc6->cur_residency[i]; - rc6->cur_residency[i] = time_hw; + time_hw += rc6->cur_residency[id]; + rc6->cur_residency[id] = time_hw; intel_uncore_forcewake_put__locked(uncore, fw_domains); spin_unlock_irqrestore(&uncore->lock, flags); @@ -811,9 +818,22 @@ u64 intel_rc6_residency_ns(struct intel_rc6 *rc6, const i915_reg_t reg) return mul_u64_u32_div(time_hw, mul, div); } -u64 intel_rc6_residency_us(struct intel_rc6 *rc6, i915_reg_t reg) +u64 intel_rc6_residency_us(struct intel_rc6 *rc6, enum intel_rc6_res_type id) +{ + return DIV_ROUND_UP_ULL(intel_rc6_residency_ns(rc6, id), 1000); +} + +void intel_rc6_print_residency(struct seq_file *m, const char *title, + enum intel_rc6_res_type id) { - return DIV_ROUND_UP_ULL(intel_rc6_residency_ns(rc6, reg), 1000); + struct intel_gt *gt = m->private; + i915_reg_t reg = gt->rc6.res_reg[id]; + intel_wakeref_t wakeref; + + with_intel_runtime_pm(gt->uncore->rpm, wakeref) + seq_printf(m, "%s %u (%llu us)\n", title, + intel_uncore_read(gt->uncore, reg), + intel_rc6_residency_us(>->rc6, id)); } #if IS_ENABLED(CONFIG_DRM_I915_SELFTEST) diff --git a/drivers/gpu/drm/i915/gt/intel_rc6.h b/drivers/gpu/drm/i915/gt/intel_rc6.h index b6fea71afc22..456fa668a276 100644 --- a/drivers/gpu/drm/i915/gt/intel_rc6.h +++ b/drivers/gpu/drm/i915/gt/intel_rc6.h @@ -6,10 +6,11 @@ #ifndef INTEL_RC6_H #define INTEL_RC6_H -#include "i915_reg_defs.h" +#include <linux/types.h> -struct intel_engine_cs; +enum intel_rc6_res_type; struct intel_rc6; +struct seq_file; void intel_rc6_init(struct intel_rc6 *rc6); void intel_rc6_fini(struct intel_rc6 *rc6); @@ -21,7 +22,9 @@ void intel_rc6_sanitize(struct intel_rc6 *rc6); void intel_rc6_enable(struct intel_rc6 *rc6); void intel_rc6_disable(struct intel_rc6 *rc6); -u64 intel_rc6_residency_ns(struct intel_rc6 *rc6, i915_reg_t reg); -u64 intel_rc6_residency_us(struct intel_rc6 *rc6, i915_reg_t reg); +u64 intel_rc6_residency_ns(struct intel_rc6 *rc6, enum intel_rc6_res_type id); +u64 intel_rc6_residency_us(struct intel_rc6 *rc6, enum intel_rc6_res_type id); +void intel_rc6_print_residency(struct seq_file *m, const char *title, + enum intel_rc6_res_type id); #endif /* INTEL_RC6_H */ diff --git a/drivers/gpu/drm/i915/gt/intel_rc6_types.h b/drivers/gpu/drm/i915/gt/intel_rc6_types.h index e747492b2f46..fa23c4dce00b 100644 --- a/drivers/gpu/drm/i915/gt/intel_rc6_types.h +++ b/drivers/gpu/drm/i915/gt/intel_rc6_types.h @@ -13,9 +13,20 @@ struct drm_i915_gem_object; +/* RC6 residency types */ +enum intel_rc6_res_type { + INTEL_RC6_RES_RC6_LOCKED, + INTEL_RC6_RES_RC6, + INTEL_RC6_RES_RC6p, + INTEL_RC6_RES_RC6pp, + INTEL_RC6_RES_MAX, + INTEL_RC6_RES_VLV_MEDIA = INTEL_RC6_RES_RC6p, +}; + struct intel_rc6 { - u64 prev_hw_residency[4]; - u64 cur_residency[4]; + i915_reg_t res_reg[INTEL_RC6_RES_MAX]; + u64 prev_hw_residency[INTEL_RC6_RES_MAX]; + u64 cur_residency[INTEL_RC6_RES_MAX]; u32 ctl_enable; diff --git a/drivers/gpu/drm/i915/gt/intel_renderstate.c b/drivers/gpu/drm/i915/gt/intel_renderstate.c index 5121e6dc2fa5..9c1ae070ee7b 100644 --- a/drivers/gpu/drm/i915/gt/intel_renderstate.c +++ b/drivers/gpu/drm/i915/gt/intel_renderstate.c @@ -215,9 +215,7 @@ int intel_renderstate_emit(struct intel_renderstate *so, if (!so->vma) return 0; - err = i915_request_await_object(rq, so->vma->obj, false); - if (err == 0) - err = i915_vma_move_to_active(so->vma, rq, 0); + err = i915_vma_move_to_active(so->vma, rq, 0); if (err) return err; diff --git a/drivers/gpu/drm/i915/gt/intel_reset.c b/drivers/gpu/drm/i915/gt/intel_reset.c index b36674356986..24736ebee17c 100644 --- a/drivers/gpu/drm/i915/gt/intel_reset.c +++ b/drivers/gpu/drm/i915/gt/intel_reset.c @@ -1278,7 +1278,7 @@ static void intel_gt_reset_global(struct intel_gt *gt, kobject_uevent_env(kobj, KOBJ_CHANGE, reset_event); /* Use a watchdog to ensure that our reset completes */ - intel_wedge_on_timeout(&w, gt, 5 * HZ) { + intel_wedge_on_timeout(&w, gt, 60 * HZ) { intel_display_prepare_reset(gt->i915); intel_gt_reset(gt, engine_mask, reason); @@ -1407,15 +1407,19 @@ out: intel_runtime_pm_put(gt->uncore->rpm, wakeref); } -int intel_gt_reset_trylock(struct intel_gt *gt, int *srcu) +static int _intel_gt_reset_lock(struct intel_gt *gt, int *srcu, bool retry) { might_lock(>->reset.backoff_srcu); - might_sleep(); + if (retry) + might_sleep(); rcu_read_lock(); while (test_bit(I915_RESET_BACKOFF, >->reset.flags)) { rcu_read_unlock(); + if (!retry) + return -EBUSY; + if (wait_event_interruptible(gt->reset.queue, !test_bit(I915_RESET_BACKOFF, >->reset.flags))) @@ -1429,6 +1433,16 @@ int intel_gt_reset_trylock(struct intel_gt *gt, int *srcu) return 0; } +int intel_gt_reset_trylock(struct intel_gt *gt, int *srcu) +{ + return _intel_gt_reset_lock(gt, srcu, false); +} + +int intel_gt_reset_lock_interruptible(struct intel_gt *gt, int *srcu) +{ + return _intel_gt_reset_lock(gt, srcu, true); +} + void intel_gt_reset_unlock(struct intel_gt *gt, int tag) __releases(>->reset.backoff_srcu) { diff --git a/drivers/gpu/drm/i915/gt/intel_reset.h b/drivers/gpu/drm/i915/gt/intel_reset.h index adc734e67387..25c975b6e8fc 100644 --- a/drivers/gpu/drm/i915/gt/intel_reset.h +++ b/drivers/gpu/drm/i915/gt/intel_reset.h @@ -39,6 +39,7 @@ int __intel_engine_reset_bh(struct intel_engine_cs *engine, void __i915_request_reset(struct i915_request *rq, bool guilty); int __must_check intel_gt_reset_trylock(struct intel_gt *gt, int *srcu); +int __must_check intel_gt_reset_lock_interruptible(struct intel_gt *gt, int *srcu); void intel_gt_reset_unlock(struct intel_gt *gt, int tag); void intel_gt_set_wedged(struct intel_gt *gt); diff --git a/drivers/gpu/drm/i915/gt/intel_ring_submission.c b/drivers/gpu/drm/i915/gt/intel_ring_submission.c index d5d6f1fadcae..356c787e11d3 100644 --- a/drivers/gpu/drm/i915/gt/intel_ring_submission.c +++ b/drivers/gpu/drm/i915/gt/intel_ring_submission.c @@ -12,7 +12,9 @@ #include "gen6_ppgtt.h" #include "gen7_renderclear.h" #include "i915_drv.h" +#include "i915_irq.h" #include "i915_mitigations.h" +#include "i915_reg.h" #include "intel_breadcrumbs.h" #include "intel_context.h" #include "intel_engine_regs.h" diff --git a/drivers/gpu/drm/i915/gt/intel_rps.c b/drivers/gpu/drm/i915/gt/intel_rps.c index 6b86250c31ab..9ad3bc7201cb 100644 --- a/drivers/gpu/drm/i915/gt/intel_rps.c +++ b/drivers/gpu/drm/i915/gt/intel_rps.c @@ -430,7 +430,8 @@ static int __gen5_rps_set(struct intel_rps *rps, u8 val) rgvswctl = intel_uncore_read16(uncore, MEMSWCTL); if (rgvswctl & MEMCTL_CMD_STS) { - DRM_DEBUG("gpu busy, RCS change rejected\n"); + drm_dbg(&rps_to_i915(rps)->drm, + "gpu busy, RCS change rejected\n"); return -EBUSY; /* still busy with another command */ } @@ -625,9 +626,7 @@ static void gen5_rps_disable(struct intel_rps *rps) rgvswctl = intel_uncore_read16(uncore, MEMSWCTL); /* Ack interrupts, disable EFC interrupt */ - intel_uncore_write(uncore, MEMINTREN, - intel_uncore_read(uncore, MEMINTREN) & - ~MEMINT_EVAL_CHG_EN); + intel_uncore_rmw(uncore, MEMINTREN, MEMINT_EVAL_CHG_EN, 0); intel_uncore_write(uncore, MEMINTRSTS, MEMINT_EVAL_CHG); /* Go back to the starting frequency */ @@ -1016,9 +1015,15 @@ void intel_rps_boost(struct i915_request *rq) if (rps_uses_slpc(rps)) { slpc = rps_to_slpc(rps); + if (slpc->min_freq_softlimit >= slpc->boost_freq) + return; + /* Return if old value is non zero */ - if (!atomic_fetch_inc(&slpc->num_waiters)) + if (!atomic_fetch_inc(&slpc->num_waiters)) { + GT_TRACE(rps_to_gt(rps), "boost fence:%llx:%llx\n", + rq->fence.context, rq->fence.seqno); schedule_work(&slpc->boost_work); + } return; } @@ -1085,15 +1090,25 @@ static u32 intel_rps_read_state_cap(struct intel_rps *rps) return intel_uncore_read(uncore, GEN6_RP_STATE_CAP); } -/** - * gen6_rps_get_freq_caps - Get freq caps exposed by HW - * @rps: the intel_rps structure - * @caps: returned freq caps - * - * Returned "caps" frequencies should be converted to MHz using - * intel_gpu_freq() - */ -void gen6_rps_get_freq_caps(struct intel_rps *rps, struct intel_rps_freq_caps *caps) +static void +mtl_get_freq_caps(struct intel_rps *rps, struct intel_rps_freq_caps *caps) +{ + struct intel_uncore *uncore = rps_to_uncore(rps); + u32 rp_state_cap = rps_to_gt(rps)->type == GT_MEDIA ? + intel_uncore_read(uncore, MTL_MEDIAP_STATE_CAP) : + intel_uncore_read(uncore, MTL_RP_STATE_CAP); + u32 rpe = rps_to_gt(rps)->type == GT_MEDIA ? + intel_uncore_read(uncore, MTL_MPE_FREQUENCY) : + intel_uncore_read(uncore, MTL_GT_RPE_FREQUENCY); + + /* MTL values are in units of 16.67 MHz */ + caps->rp0_freq = REG_FIELD_GET(MTL_RP0_CAP_MASK, rp_state_cap); + caps->min_freq = REG_FIELD_GET(MTL_RPN_CAP_MASK, rp_state_cap); + caps->rp1_freq = REG_FIELD_GET(MTL_RPE_MASK, rpe); +} + +static void +__gen6_rps_get_freq_caps(struct intel_rps *rps, struct intel_rps_freq_caps *caps) { struct drm_i915_private *i915 = rps_to_i915(rps); u32 rp_state_cap; @@ -1128,6 +1143,24 @@ void gen6_rps_get_freq_caps(struct intel_rps *rps, struct intel_rps_freq_caps *c } } +/** + * gen6_rps_get_freq_caps - Get freq caps exposed by HW + * @rps: the intel_rps structure + * @caps: returned freq caps + * + * Returned "caps" frequencies should be converted to MHz using + * intel_gpu_freq() + */ +void gen6_rps_get_freq_caps(struct intel_rps *rps, struct intel_rps_freq_caps *caps) +{ + struct drm_i915_private *i915 = rps_to_i915(rps); + + if (IS_METEORLAKE(i915)) + return mtl_get_freq_caps(rps, caps); + else + return __gen6_rps_get_freq_caps(rps, caps); +} + static void gen6_rps_init(struct intel_rps *rps) { struct drm_i915_private *i915 = rps_to_i915(rps); @@ -1921,7 +1954,8 @@ void gen6_rps_irq_handler(struct intel_rps *rps, u32 pm_iir) intel_engine_cs_irq(gt->engine[VECS0], pm_iir >> 10); if (pm_iir & PM_VEBOX_CS_ERROR_INTERRUPT) - DRM_DEBUG("Command parser error, pm_iir 0x%08x\n", pm_iir); + drm_dbg(&rps_to_i915(rps)->drm, + "Command parser error, pm_iir 0x%08x\n", pm_iir); } void gen5_rps_irq_handler(struct intel_rps *rps) @@ -2040,22 +2074,45 @@ void intel_rps_sanitize(struct intel_rps *rps) rps_disable_interrupts(rps); } +u32 intel_rps_read_rpstat_fw(struct intel_rps *rps) +{ + struct drm_i915_private *i915 = rps_to_i915(rps); + i915_reg_t rpstat; + + rpstat = (GRAPHICS_VER(i915) >= 12) ? GEN12_RPSTAT1 : GEN6_RPSTAT1; + + return intel_uncore_read_fw(rps_to_gt(rps)->uncore, rpstat); +} + +u32 intel_rps_read_rpstat(struct intel_rps *rps) +{ + struct drm_i915_private *i915 = rps_to_i915(rps); + i915_reg_t rpstat; + + rpstat = (GRAPHICS_VER(i915) >= 12) ? GEN12_RPSTAT1 : GEN6_RPSTAT1; + + return intel_uncore_read(rps_to_gt(rps)->uncore, rpstat); +} + u32 intel_rps_get_cagf(struct intel_rps *rps, u32 rpstat) { struct drm_i915_private *i915 = rps_to_i915(rps); u32 cagf; - if (IS_VALLEYVIEW(i915) || IS_CHERRYVIEW(i915)) - cagf = (rpstat >> 8) & 0xff; + if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 70)) + cagf = REG_FIELD_GET(MTL_CAGF_MASK, rpstat); + else if (GRAPHICS_VER(i915) >= 12) + cagf = REG_FIELD_GET(GEN12_CAGF_MASK, rpstat); + else if (IS_VALLEYVIEW(i915) || IS_CHERRYVIEW(i915)) + cagf = REG_FIELD_GET(RPE_MASK, rpstat); else if (GRAPHICS_VER(i915) >= 9) - cagf = (rpstat & GEN9_CAGF_MASK) >> GEN9_CAGF_SHIFT; + cagf = REG_FIELD_GET(GEN9_CAGF_MASK, rpstat); else if (IS_HASWELL(i915) || IS_BROADWELL(i915)) - cagf = (rpstat & HSW_CAGF_MASK) >> HSW_CAGF_SHIFT; + cagf = REG_FIELD_GET(HSW_CAGF_MASK, rpstat); else if (GRAPHICS_VER(i915) >= 6) - cagf = (rpstat & GEN6_CAGF_MASK) >> GEN6_CAGF_SHIFT; + cagf = REG_FIELD_GET(GEN6_CAGF_MASK, rpstat); else - cagf = gen5_invert_freq(rps, (rpstat & MEMSTAT_PSTATE_MASK) >> - MEMSTAT_PSTATE_SHIFT); + cagf = gen5_invert_freq(rps, REG_FIELD_GET(MEMSTAT_PSTATE_MASK, rpstat)); return cagf; } @@ -2066,7 +2123,15 @@ static u32 read_cagf(struct intel_rps *rps) struct intel_uncore *uncore = rps_to_uncore(rps); u32 freq; - if (IS_VALLEYVIEW(i915) || IS_CHERRYVIEW(i915)) { + /* + * For Gen12+ reading freq from HW does not need a forcewake and + * registers will return 0 freq when GT is in RC6 + */ + if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 70)) { + freq = intel_uncore_read(uncore, MTL_MIRROR_TARGET_WP1); + } else if (GRAPHICS_VER(i915) >= 12) { + freq = intel_uncore_read(uncore, GEN12_RPSTAT1); + } else if (IS_VALLEYVIEW(i915) || IS_CHERRYVIEW(i915)) { vlv_punit_get(i915); freq = vlv_punit_read(i915, PUNIT_REG_GPU_FREQ_STS); vlv_punit_put(i915); @@ -2191,6 +2256,213 @@ u32 intel_rps_get_rpn_frequency(struct intel_rps *rps) return intel_gpu_freq(rps, rps->min_freq); } +static void rps_frequency_dump(struct intel_rps *rps, struct drm_printer *p) +{ + struct intel_gt *gt = rps_to_gt(rps); + struct drm_i915_private *i915 = gt->i915; + struct intel_uncore *uncore = gt->uncore; + struct intel_rps_freq_caps caps; + u32 rp_state_limits; + u32 gt_perf_status; + u32 rpmodectl, rpinclimit, rpdeclimit; + u32 rpstat, cagf, reqf; + u32 rpcurupei, rpcurup, rpprevup; + u32 rpcurdownei, rpcurdown, rpprevdown; + u32 rpupei, rpupt, rpdownei, rpdownt; + u32 pm_ier, pm_imr, pm_isr, pm_iir, pm_mask; + + rp_state_limits = intel_uncore_read(uncore, GEN6_RP_STATE_LIMITS); + gen6_rps_get_freq_caps(rps, &caps); + if (IS_GEN9_LP(i915)) + gt_perf_status = intel_uncore_read(uncore, BXT_GT_PERF_STATUS); + else + gt_perf_status = intel_uncore_read(uncore, GEN6_GT_PERF_STATUS); + + /* RPSTAT1 is in the GT power well */ + intel_uncore_forcewake_get(uncore, FORCEWAKE_ALL); + + reqf = intel_uncore_read(uncore, GEN6_RPNSWREQ); + if (GRAPHICS_VER(i915) >= 9) { + reqf >>= 23; + } else { + reqf &= ~GEN6_TURBO_DISABLE; + if (IS_HASWELL(i915) || IS_BROADWELL(i915)) + reqf >>= 24; + else + reqf >>= 25; + } + reqf = intel_gpu_freq(rps, reqf); + + rpmodectl = intel_uncore_read(uncore, GEN6_RP_CONTROL); + rpinclimit = intel_uncore_read(uncore, GEN6_RP_UP_THRESHOLD); + rpdeclimit = intel_uncore_read(uncore, GEN6_RP_DOWN_THRESHOLD); + + rpstat = intel_rps_read_rpstat(rps); + rpcurupei = intel_uncore_read(uncore, GEN6_RP_CUR_UP_EI) & GEN6_CURICONT_MASK; + rpcurup = intel_uncore_read(uncore, GEN6_RP_CUR_UP) & GEN6_CURBSYTAVG_MASK; + rpprevup = intel_uncore_read(uncore, GEN6_RP_PREV_UP) & GEN6_CURBSYTAVG_MASK; + rpcurdownei = intel_uncore_read(uncore, GEN6_RP_CUR_DOWN_EI) & GEN6_CURIAVG_MASK; + rpcurdown = intel_uncore_read(uncore, GEN6_RP_CUR_DOWN) & GEN6_CURBSYTAVG_MASK; + rpprevdown = intel_uncore_read(uncore, GEN6_RP_PREV_DOWN) & GEN6_CURBSYTAVG_MASK; + + rpupei = intel_uncore_read(uncore, GEN6_RP_UP_EI); + rpupt = intel_uncore_read(uncore, GEN6_RP_UP_THRESHOLD); + + rpdownei = intel_uncore_read(uncore, GEN6_RP_DOWN_EI); + rpdownt = intel_uncore_read(uncore, GEN6_RP_DOWN_THRESHOLD); + + cagf = intel_rps_read_actual_frequency(rps); + + intel_uncore_forcewake_put(uncore, FORCEWAKE_ALL); + + if (GRAPHICS_VER(i915) >= 11) { + pm_ier = intel_uncore_read(uncore, GEN11_GPM_WGBOXPERF_INTR_ENABLE); + pm_imr = intel_uncore_read(uncore, GEN11_GPM_WGBOXPERF_INTR_MASK); + /* + * The equivalent to the PM ISR & IIR cannot be read + * without affecting the current state of the system + */ + pm_isr = 0; + pm_iir = 0; + } else if (GRAPHICS_VER(i915) >= 8) { + pm_ier = intel_uncore_read(uncore, GEN8_GT_IER(2)); + pm_imr = intel_uncore_read(uncore, GEN8_GT_IMR(2)); + pm_isr = intel_uncore_read(uncore, GEN8_GT_ISR(2)); + pm_iir = intel_uncore_read(uncore, GEN8_GT_IIR(2)); + } else { + pm_ier = intel_uncore_read(uncore, GEN6_PMIER); + pm_imr = intel_uncore_read(uncore, GEN6_PMIMR); + pm_isr = intel_uncore_read(uncore, GEN6_PMISR); + pm_iir = intel_uncore_read(uncore, GEN6_PMIIR); + } + pm_mask = intel_uncore_read(uncore, GEN6_PMINTRMSK); + + drm_printf(p, "Video Turbo Mode: %s\n", + str_yes_no(rpmodectl & GEN6_RP_MEDIA_TURBO)); + drm_printf(p, "HW control enabled: %s\n", + str_yes_no(rpmodectl & GEN6_RP_ENABLE)); + drm_printf(p, "SW control enabled: %s\n", + str_yes_no((rpmodectl & GEN6_RP_MEDIA_MODE_MASK) == GEN6_RP_MEDIA_SW_MODE)); + + drm_printf(p, "PM IER=0x%08x IMR=0x%08x, MASK=0x%08x\n", + pm_ier, pm_imr, pm_mask); + if (GRAPHICS_VER(i915) <= 10) + drm_printf(p, "PM ISR=0x%08x IIR=0x%08x\n", + pm_isr, pm_iir); + drm_printf(p, "pm_intrmsk_mbz: 0x%08x\n", + rps->pm_intrmsk_mbz); + drm_printf(p, "GT_PERF_STATUS: 0x%08x\n", gt_perf_status); + drm_printf(p, "Render p-state ratio: %d\n", + (gt_perf_status & (GRAPHICS_VER(i915) >= 9 ? 0x1ff00 : 0xff00)) >> 8); + drm_printf(p, "Render p-state VID: %d\n", + gt_perf_status & 0xff); + drm_printf(p, "Render p-state limit: %d\n", + rp_state_limits & 0xff); + drm_printf(p, "RPSTAT1: 0x%08x\n", rpstat); + drm_printf(p, "RPMODECTL: 0x%08x\n", rpmodectl); + drm_printf(p, "RPINCLIMIT: 0x%08x\n", rpinclimit); + drm_printf(p, "RPDECLIMIT: 0x%08x\n", rpdeclimit); + drm_printf(p, "RPNSWREQ: %dMHz\n", reqf); + drm_printf(p, "CAGF: %dMHz\n", cagf); + drm_printf(p, "RP CUR UP EI: %d (%lldns)\n", + rpcurupei, + intel_gt_pm_interval_to_ns(gt, rpcurupei)); + drm_printf(p, "RP CUR UP: %d (%lldns)\n", + rpcurup, intel_gt_pm_interval_to_ns(gt, rpcurup)); + drm_printf(p, "RP PREV UP: %d (%lldns)\n", + rpprevup, intel_gt_pm_interval_to_ns(gt, rpprevup)); + drm_printf(p, "Up threshold: %d%%\n", + rps->power.up_threshold); + drm_printf(p, "RP UP EI: %d (%lldns)\n", + rpupei, intel_gt_pm_interval_to_ns(gt, rpupei)); + drm_printf(p, "RP UP THRESHOLD: %d (%lldns)\n", + rpupt, intel_gt_pm_interval_to_ns(gt, rpupt)); + + drm_printf(p, "RP CUR DOWN EI: %d (%lldns)\n", + rpcurdownei, + intel_gt_pm_interval_to_ns(gt, rpcurdownei)); + drm_printf(p, "RP CUR DOWN: %d (%lldns)\n", + rpcurdown, + intel_gt_pm_interval_to_ns(gt, rpcurdown)); + drm_printf(p, "RP PREV DOWN: %d (%lldns)\n", + rpprevdown, + intel_gt_pm_interval_to_ns(gt, rpprevdown)); + drm_printf(p, "Down threshold: %d%%\n", + rps->power.down_threshold); + drm_printf(p, "RP DOWN EI: %d (%lldns)\n", + rpdownei, intel_gt_pm_interval_to_ns(gt, rpdownei)); + drm_printf(p, "RP DOWN THRESHOLD: %d (%lldns)\n", + rpdownt, intel_gt_pm_interval_to_ns(gt, rpdownt)); + + drm_printf(p, "Lowest (RPN) frequency: %dMHz\n", + intel_gpu_freq(rps, caps.min_freq)); + drm_printf(p, "Nominal (RP1) frequency: %dMHz\n", + intel_gpu_freq(rps, caps.rp1_freq)); + drm_printf(p, "Max non-overclocked (RP0) frequency: %dMHz\n", + intel_gpu_freq(rps, caps.rp0_freq)); + drm_printf(p, "Max overclocked frequency: %dMHz\n", + intel_gpu_freq(rps, rps->max_freq)); + + drm_printf(p, "Current freq: %d MHz\n", + intel_gpu_freq(rps, rps->cur_freq)); + drm_printf(p, "Actual freq: %d MHz\n", cagf); + drm_printf(p, "Idle freq: %d MHz\n", + intel_gpu_freq(rps, rps->idle_freq)); + drm_printf(p, "Min freq: %d MHz\n", + intel_gpu_freq(rps, rps->min_freq)); + drm_printf(p, "Boost freq: %d MHz\n", + intel_gpu_freq(rps, rps->boost_freq)); + drm_printf(p, "Max freq: %d MHz\n", + intel_gpu_freq(rps, rps->max_freq)); + drm_printf(p, + "efficient (RPe) frequency: %d MHz\n", + intel_gpu_freq(rps, rps->efficient_freq)); +} + +static void slpc_frequency_dump(struct intel_rps *rps, struct drm_printer *p) +{ + struct intel_gt *gt = rps_to_gt(rps); + struct intel_uncore *uncore = gt->uncore; + struct intel_rps_freq_caps caps; + u32 pm_mask; + + gen6_rps_get_freq_caps(rps, &caps); + pm_mask = intel_uncore_read(uncore, GEN6_PMINTRMSK); + + drm_printf(p, "PM MASK=0x%08x\n", pm_mask); + drm_printf(p, "pm_intrmsk_mbz: 0x%08x\n", + rps->pm_intrmsk_mbz); + drm_printf(p, "RPSTAT1: 0x%08x\n", intel_rps_read_rpstat(rps)); + drm_printf(p, "RPNSWREQ: %dMHz\n", intel_rps_get_requested_frequency(rps)); + drm_printf(p, "Lowest (RPN) frequency: %dMHz\n", + intel_gpu_freq(rps, caps.min_freq)); + drm_printf(p, "Nominal (RP1) frequency: %dMHz\n", + intel_gpu_freq(rps, caps.rp1_freq)); + drm_printf(p, "Max non-overclocked (RP0) frequency: %dMHz\n", + intel_gpu_freq(rps, caps.rp0_freq)); + drm_printf(p, "Current freq: %d MHz\n", + intel_rps_get_requested_frequency(rps)); + drm_printf(p, "Actual freq: %d MHz\n", + intel_rps_read_actual_frequency(rps)); + drm_printf(p, "Min freq: %d MHz\n", + intel_rps_get_min_frequency(rps)); + drm_printf(p, "Boost freq: %d MHz\n", + intel_rps_get_boost_frequency(rps)); + drm_printf(p, "Max freq: %d MHz\n", + intel_rps_get_max_frequency(rps)); + drm_printf(p, + "efficient (RPe) frequency: %d MHz\n", + intel_gpu_freq(rps, caps.rp1_freq)); +} + +void gen6_rps_frequency_dump(struct intel_rps *rps, struct drm_printer *p) +{ + if (rps_uses_slpc(rps)) + return slpc_frequency_dump(rps, p); + else + return rps_frequency_dump(rps, p); +} + static int set_max_freq(struct intel_rps *rps, u32 val) { struct drm_i915_private *i915 = rps_to_i915(rps); diff --git a/drivers/gpu/drm/i915/gt/intel_rps.h b/drivers/gpu/drm/i915/gt/intel_rps.h index 4509dfdc52e0..9e1cad9ba0e9 100644 --- a/drivers/gpu/drm/i915/gt/intel_rps.h +++ b/drivers/gpu/drm/i915/gt/intel_rps.h @@ -10,6 +10,7 @@ #include "i915_reg_defs.h" struct i915_request; +struct drm_printer; void intel_rps_init_early(struct intel_rps *rps); void intel_rps_init(struct intel_rps *rps); @@ -47,6 +48,8 @@ u32 intel_rps_get_rp1_frequency(struct intel_rps *rps); u32 intel_rps_get_rpn_frequency(struct intel_rps *rps); u32 intel_rps_read_punit_req(struct intel_rps *rps); u32 intel_rps_read_punit_req_frequency(struct intel_rps *rps); +u32 intel_rps_read_rpstat(struct intel_rps *rps); +u32 intel_rps_read_rpstat_fw(struct intel_rps *rps); void gen6_rps_get_freq_caps(struct intel_rps *rps, struct intel_rps_freq_caps *caps); void intel_rps_raise_unslice(struct intel_rps *rps); void intel_rps_lower_unslice(struct intel_rps *rps); @@ -54,6 +57,8 @@ void intel_rps_lower_unslice(struct intel_rps *rps); u32 intel_rps_read_throttle_reason(struct intel_rps *rps); bool rps_read_mask_mmio(struct intel_rps *rps, i915_reg_t reg32, u32 mask); +void gen6_rps_frequency_dump(struct intel_rps *rps, struct drm_printer *p); + void gen5_rps_irq_handler(struct intel_rps *rps); void gen6_rps_irq_handler(struct intel_rps *rps, u32 pm_iir); void gen11_rps_irq_handler(struct intel_rps *rps, u32 pm_iir); diff --git a/drivers/gpu/drm/i915/gt/intel_sseu.c b/drivers/gpu/drm/i915/gt/intel_sseu.c index 66f21c735d54..6c6198a257ac 100644 --- a/drivers/gpu/drm/i915/gt/intel_sseu.c +++ b/drivers/gpu/drm/i915/gt/intel_sseu.c @@ -677,8 +677,8 @@ u32 intel_sseu_make_rpcs(struct intel_gt *gt, * If i915/perf is active, we want a stable powergating configuration * on the system. Use the configuration pinned by i915/perf. */ - if (i915->perf.exclusive_stream) - req_sseu = &i915->perf.sseu; + if (gt->perf.exclusive_stream) + req_sseu = >->perf.sseu; slices = hweight8(req_sseu->slice_mask); subslices = hweight8(req_sseu->subslice_mask); diff --git a/drivers/gpu/drm/i915/intel_wopcm.c b/drivers/gpu/drm/i915/gt/intel_wopcm.c index 322fb9eeb880..7ebbcc191c2d 100644 --- a/drivers/gpu/drm/i915/intel_wopcm.c +++ b/drivers/gpu/drm/i915/gt/intel_wopcm.c @@ -64,9 +64,9 @@ #define GEN9_GUC_FW_RESERVED SZ_128K #define GEN9_GUC_WOPCM_OFFSET (GUC_WOPCM_RESERVED + GEN9_GUC_FW_RESERVED) -static inline struct drm_i915_private *wopcm_to_i915(struct intel_wopcm *wopcm) +static inline struct intel_gt *wopcm_to_gt(struct intel_wopcm *wopcm) { - return container_of(wopcm, struct drm_i915_private, wopcm); + return container_of(wopcm, struct intel_gt, wopcm); } /** @@ -77,7 +77,8 @@ static inline struct drm_i915_private *wopcm_to_i915(struct intel_wopcm *wopcm) */ void intel_wopcm_init_early(struct intel_wopcm *wopcm) { - struct drm_i915_private *i915 = wopcm_to_i915(wopcm); + struct intel_gt *gt = wopcm_to_gt(wopcm); + struct drm_i915_private *i915 = gt->i915; if (!HAS_GT_UC(i915)) return; @@ -157,10 +158,11 @@ static bool check_hw_restrictions(struct drm_i915_private *i915, return true; } -static bool __check_layout(struct drm_i915_private *i915, u32 wopcm_size, +static bool __check_layout(struct intel_gt *gt, u32 wopcm_size, u32 guc_wopcm_base, u32 guc_wopcm_size, u32 guc_fw_size, u32 huc_fw_size) { + struct drm_i915_private *i915 = gt->i915; const u32 ctx_rsvd = context_reserved_size(i915); u32 size; @@ -181,12 +183,14 @@ static bool __check_layout(struct drm_i915_private *i915, u32 wopcm_size, return false; } - size = huc_fw_size + WOPCM_RESERVED_SIZE; - if (unlikely(guc_wopcm_base < size)) { - drm_err(&i915->drm, "WOPCM: no space for %s: %uK < %uK\n", - intel_uc_fw_type_repr(INTEL_UC_FW_TYPE_HUC), - guc_wopcm_base / SZ_1K, size / SZ_1K); - return false; + if (intel_uc_supports_huc(>->uc)) { + size = huc_fw_size + WOPCM_RESERVED_SIZE; + if (unlikely(guc_wopcm_base < size)) { + drm_err(&i915->drm, "WOPCM: no space for %s: %uK < %uK\n", + intel_uc_fw_type_repr(INTEL_UC_FW_TYPE_HUC), + guc_wopcm_base / SZ_1K, size / SZ_1K); + return false; + } } return check_hw_restrictions(i915, guc_wopcm_base, guc_wopcm_size, @@ -228,8 +232,8 @@ static bool __wopcm_regs_writable(struct intel_uncore *uncore) */ void intel_wopcm_init(struct intel_wopcm *wopcm) { - struct drm_i915_private *i915 = wopcm_to_i915(wopcm); - struct intel_gt *gt = to_gt(i915); + struct intel_gt *gt = wopcm_to_gt(wopcm); + struct drm_i915_private *i915 = gt->i915; u32 guc_fw_size = intel_uc_fw_get_upload_size(>->uc.guc.fw); u32 huc_fw_size = intel_uc_fw_get_upload_size(>->uc.huc.fw); u32 ctx_rsvd = context_reserved_size(i915); @@ -275,6 +279,19 @@ void intel_wopcm_init(struct intel_wopcm *wopcm) } /* + * On platforms with a media GT, the WOPCM is partitioned between the + * two GTs, so we would have to take that into account when doing the + * math below. There is also a new section reserved for the GSC context + * that would have to be factored in. However, all platforms with a + * media GT also have GuC depriv enabled, so the WOPCM regs are + * pre-locked and therefore we don't have to do the math ourselves. + */ + if (unlikely(i915->media_gt)) { + drm_err(&i915->drm, "Unlocked WOPCM regs with media GT\n"); + return; + } + + /* * Aligned value of guc_wopcm_base will determine available WOPCM space * for HuC firmware and mandatory reserved area. */ @@ -295,7 +312,7 @@ void intel_wopcm_init(struct intel_wopcm *wopcm) guc_wopcm_base / SZ_1K, guc_wopcm_size / SZ_1K); check: - if (__check_layout(i915, wopcm_size, guc_wopcm_base, guc_wopcm_size, + if (__check_layout(gt, wopcm_size, guc_wopcm_base, guc_wopcm_size, guc_fw_size, huc_fw_size)) { wopcm->guc.base = guc_wopcm_base; wopcm->guc.size = guc_wopcm_size; diff --git a/drivers/gpu/drm/i915/intel_wopcm.h b/drivers/gpu/drm/i915/gt/intel_wopcm.h index 17d6aa86008a..17d6aa86008a 100644 --- a/drivers/gpu/drm/i915/intel_wopcm.h +++ b/drivers/gpu/drm/i915/gt/intel_wopcm.h diff --git a/drivers/gpu/drm/i915/gt/intel_workarounds.c b/drivers/gpu/drm/i915/gt/intel_workarounds.c index a821e3d405db..2afb4f80a954 100644 --- a/drivers/gpu/drm/i915/gt/intel_workarounds.c +++ b/drivers/gpu/drm/i915/gt/intel_workarounds.c @@ -4,6 +4,7 @@ */ #include "i915_drv.h" +#include "i915_reg.h" #include "intel_context.h" #include "intel_engine_pm.h" #include "intel_engine_regs.h" @@ -17,46 +18,68 @@ /** * DOC: Hardware workarounds * - * This file is intended as a central place to implement most [1]_ of the - * required workarounds for hardware to work as originally intended. They fall - * in five basic categories depending on how/when they are applied: + * Hardware workarounds are register programming documented to be executed in + * the driver that fall outside of the normal programming sequences for a + * platform. There are some basic categories of workarounds, depending on + * how/when they are applied: * - * - Workarounds that touch registers that are saved/restored to/from the HW - * context image. The list is emitted (via Load Register Immediate commands) - * everytime a new context is created. - * - GT workarounds. The list of these WAs is applied whenever these registers - * revert to default values (on GPU reset, suspend/resume [2]_, etc..). - * - Display workarounds. The list is applied during display clock-gating - * initialization. - * - Workarounds that whitelist a privileged register, so that UMDs can manage - * them directly. This is just a special case of a MMMIO workaround (as we - * write the list of these to/be-whitelisted registers to some special HW - * registers). - * - Workaround batchbuffers, that get executed automatically by the hardware - * on every HW context restore. + * - Context workarounds: workarounds that touch registers that are + * saved/restored to/from the HW context image. The list is emitted (via Load + * Register Immediate commands) once when initializing the device and saved in + * the default context. That default context is then used on every context + * creation to have a "primed golden context", i.e. a context image that + * already contains the changes needed to all the registers. * - * .. [1] Please notice that there are other WAs that, due to their nature, - * cannot be applied from a central place. Those are peppered around the rest - * of the code, as needed. + * - Engine workarounds: the list of these WAs is applied whenever the specific + * engine is reset. It's also possible that a set of engine classes share a + * common power domain and they are reset together. This happens on some + * platforms with render and compute engines. In this case (at least) one of + * them need to keeep the workaround programming: the approach taken in the + * driver is to tie those workarounds to the first compute/render engine that + * is registered. When executing with GuC submission, engine resets are + * outside of kernel driver control, hence the list of registers involved in + * written once, on engine initialization, and then passed to GuC, that + * saves/restores their values before/after the reset takes place. See + * ``drivers/gpu/drm/i915/gt/uc/intel_guc_ads.c`` for reference. * - * .. [2] Technically, some registers are powercontext saved & restored, so they - * survive a suspend/resume. In practice, writing them again is not too - * costly and simplifies things. We can revisit this in the future. + * - GT workarounds: the list of these WAs is applied whenever these registers + * revert to their default values: on GPU reset, suspend/resume [1]_, etc. + * + * - Register whitelist: some workarounds need to be implemented in userspace, + * but need to touch privileged registers. The whitelist in the kernel + * instructs the hardware to allow the access to happen. From the kernel side, + * this is just a special case of a MMIO workaround (as we write the list of + * these to/be-whitelisted registers to some special HW registers). + * + * - Workaround batchbuffers: buffers that get executed automatically by the + * hardware on every HW context restore. These buffers are created and + * programmed in the default context so the hardware always go through those + * programming sequences when switching contexts. The support for workaround + * batchbuffers is enabled these hardware mechanisms: * - * Layout - * ~~~~~~ + * #. INDIRECT_CTX: A batchbuffer and an offset are provided in the default + * context, pointing the hardware to jump to that location when that offset + * is reached in the context restore. Workaround batchbuffer in the driver + * currently uses this mechanism for all platforms. * - * Keep things in this file ordered by WA type, as per the above (context, GT, - * display, register whitelist, batchbuffer). Then, inside each type, keep the - * following order: + * #. BB_PER_CTX_PTR: A batchbuffer is provided in the default context, + * pointing the hardware to a buffer to continue executing after the + * engine registers are restored in a context restore sequence. This is + * currently not used in the driver. * - * - Infrastructure functions and macros - * - WAs per platform in standard gen/chrono order - * - Public functions to init or apply the given workaround type. + * - Other: There are WAs that, due to their nature, cannot be applied from a + * central place. Those are peppered around the rest of the code, as needed. + * Workarounds related to the display IP are the main example. + * + * .. [1] Technically, some registers are powercontext saved & restored, so they + * survive a suspend/resume. In practice, writing them again is not too + * costly and simplifies things, so it's the approach taken in the driver. */ -static void wa_init_start(struct i915_wa_list *wal, const char *name, const char *engine_name) +static void wa_init_start(struct i915_wa_list *wal, struct intel_gt *gt, + const char *name, const char *engine_name) { + wal->gt = gt; wal->name = name; wal->engine_name = engine_name; } @@ -80,13 +103,14 @@ static void wa_init_finish(struct i915_wa_list *wal) if (!wal->count) return; - DRM_DEBUG_DRIVER("Initialized %u %s workarounds on %s\n", - wal->wa_count, wal->name, wal->engine_name); + drm_dbg(&wal->gt->i915->drm, "Initialized %u %s workarounds on %s\n", + wal->wa_count, wal->name, wal->engine_name); } static void _wa_add(struct i915_wa_list *wal, const struct i915_wa *wa) { unsigned int addr = i915_mmio_reg_offset(wa->reg); + struct drm_i915_private *i915 = wal->gt->i915; unsigned int start = 0, end = wal->count; const unsigned int grow = WA_LIST_CHUNK; struct i915_wa *wa_; @@ -99,7 +123,7 @@ static void _wa_add(struct i915_wa_list *wal, const struct i915_wa *wa) list = kmalloc_array(ALIGN(wal->count + 1, grow), sizeof(*wa), GFP_KERNEL); if (!list) { - DRM_ERROR("No space for workaround init!\n"); + drm_err(&i915->drm, "No space for workaround init!\n"); return; } @@ -122,9 +146,10 @@ static void _wa_add(struct i915_wa_list *wal, const struct i915_wa *wa) wa_ = &wal->list[mid]; if ((wa->clr | wa_->clr) && !(wa->clr & ~wa_->clr)) { - DRM_ERROR("Discarding overwritten w/a for reg %04x (clear: %08x, set: %08x)\n", - i915_mmio_reg_offset(wa_->reg), - wa_->clr, wa_->set); + drm_err(&i915->drm, + "Discarding overwritten w/a for reg %04x (clear: %08x, set: %08x)\n", + i915_mmio_reg_offset(wa_->reg), + wa_->clr, wa_->set); wa_->set &= ~wa->clr; } @@ -166,6 +191,21 @@ static void wa_add(struct i915_wa_list *wal, i915_reg_t reg, _wa_add(wal, &wa); } +static void wa_mcr_add(struct i915_wa_list *wal, i915_mcr_reg_t reg, + u32 clear, u32 set, u32 read_mask, bool masked_reg) +{ + struct i915_wa wa = { + .mcr_reg = reg, + .clr = clear, + .set = set, + .read = read_mask, + .masked_reg = masked_reg, + .is_mcr = 1, + }; + + _wa_add(wal, &wa); +} + static void wa_write_clr_set(struct i915_wa_list *wal, i915_reg_t reg, u32 clear, u32 set) { @@ -173,6 +213,12 @@ wa_write_clr_set(struct i915_wa_list *wal, i915_reg_t reg, u32 clear, u32 set) } static void +wa_mcr_write_clr_set(struct i915_wa_list *wal, i915_mcr_reg_t reg, u32 clear, u32 set) +{ + wa_mcr_add(wal, reg, clear, set, clear, false); +} + +static void wa_write(struct i915_wa_list *wal, i915_reg_t reg, u32 set) { wa_write_clr_set(wal, reg, ~0, set); @@ -185,11 +231,23 @@ wa_write_or(struct i915_wa_list *wal, i915_reg_t reg, u32 set) } static void +wa_mcr_write_or(struct i915_wa_list *wal, i915_mcr_reg_t reg, u32 set) +{ + wa_mcr_write_clr_set(wal, reg, set, set); +} + +static void wa_write_clr(struct i915_wa_list *wal, i915_reg_t reg, u32 clr) { wa_write_clr_set(wal, reg, clr, 0); } +static void +wa_mcr_write_clr(struct i915_wa_list *wal, i915_mcr_reg_t reg, u32 clr) +{ + wa_mcr_write_clr_set(wal, reg, clr, 0); +} + /* * WA operations on "masked register". A masked register has the upper 16 bits * documented as "masked" in b-spec. Its purpose is to allow writing to just a @@ -208,18 +266,37 @@ wa_masked_en(struct i915_wa_list *wal, i915_reg_t reg, u32 val) } static void +wa_mcr_masked_en(struct i915_wa_list *wal, i915_mcr_reg_t reg, u32 val) +{ + wa_mcr_add(wal, reg, 0, _MASKED_BIT_ENABLE(val), val, true); +} + +static void wa_masked_dis(struct i915_wa_list *wal, i915_reg_t reg, u32 val) { wa_add(wal, reg, 0, _MASKED_BIT_DISABLE(val), val, true); } static void +wa_mcr_masked_dis(struct i915_wa_list *wal, i915_mcr_reg_t reg, u32 val) +{ + wa_mcr_add(wal, reg, 0, _MASKED_BIT_DISABLE(val), val, true); +} + +static void wa_masked_field_set(struct i915_wa_list *wal, i915_reg_t reg, u32 mask, u32 val) { wa_add(wal, reg, 0, _MASKED_FIELD(mask, val), mask, true); } +static void +wa_mcr_masked_field_set(struct i915_wa_list *wal, i915_mcr_reg_t reg, + u32 mask, u32 val) +{ + wa_mcr_add(wal, reg, 0, _MASKED_FIELD(mask, val), mask, true); +} + static void gen6_ctx_workarounds_init(struct intel_engine_cs *engine, struct i915_wa_list *wal) { @@ -241,8 +318,8 @@ static void gen8_ctx_workarounds_init(struct intel_engine_cs *engine, wa_masked_en(wal, RING_MI_MODE(RENDER_RING_BASE), ASYNC_FLIP_PERF_DISABLE); /* WaDisablePartialInstShootdown:bdw,chv */ - wa_masked_en(wal, GEN8_ROW_CHICKEN, - PARTIAL_INSTRUCTION_SHOOTDOWN_DISABLE); + wa_mcr_masked_en(wal, GEN8_ROW_CHICKEN, + PARTIAL_INSTRUCTION_SHOOTDOWN_DISABLE); /* Use Force Non-Coherent whenever executing a 3D context. This is a * workaround for a possible hang in the unlikely event a TLB @@ -288,18 +365,18 @@ static void bdw_ctx_workarounds_init(struct intel_engine_cs *engine, gen8_ctx_workarounds_init(engine, wal); /* WaDisableThreadStallDopClockGating:bdw (pre-production) */ - wa_masked_en(wal, GEN8_ROW_CHICKEN, STALL_DOP_GATING_DISABLE); + wa_mcr_masked_en(wal, GEN8_ROW_CHICKEN, STALL_DOP_GATING_DISABLE); /* WaDisableDopClockGating:bdw * * Also see the related UCGTCL1 write in bdw_init_clock_gating() * to disable EUTC clock gating. */ - wa_masked_en(wal, GEN7_ROW_CHICKEN2, - DOP_CLOCK_GATING_DISABLE); + wa_mcr_masked_en(wal, GEN8_ROW_CHICKEN2, + DOP_CLOCK_GATING_DISABLE); - wa_masked_en(wal, HALF_SLICE_CHICKEN3, - GEN8_SAMPLER_POWER_BYPASS_DIS); + wa_mcr_masked_en(wal, GEN8_HALF_SLICE_CHICKEN3, + GEN8_SAMPLER_POWER_BYPASS_DIS); wa_masked_en(wal, HDC_CHICKEN0, /* WaForceContextSaveRestoreNonCoherent:bdw */ @@ -314,7 +391,7 @@ static void chv_ctx_workarounds_init(struct intel_engine_cs *engine, gen8_ctx_workarounds_init(engine, wal); /* WaDisableThreadStallDopClockGating:chv */ - wa_masked_en(wal, GEN8_ROW_CHICKEN, STALL_DOP_GATING_DISABLE); + wa_mcr_masked_en(wal, GEN8_ROW_CHICKEN, STALL_DOP_GATING_DISABLE); /* Improve HiZ throughput on CHV. */ wa_masked_en(wal, HIZ_CHICKEN, CHV_HZ_8X8_MODE_IN_1X); @@ -333,21 +410,21 @@ static void gen9_ctx_workarounds_init(struct intel_engine_cs *engine, */ wa_masked_en(wal, COMMON_SLICE_CHICKEN2, GEN9_PBE_COMPRESSED_HASH_SELECTION); - wa_masked_en(wal, GEN9_HALF_SLICE_CHICKEN7, - GEN9_SAMPLER_HASH_COMPRESSED_READ_ADDR); + wa_mcr_masked_en(wal, GEN9_HALF_SLICE_CHICKEN7, + GEN9_SAMPLER_HASH_COMPRESSED_READ_ADDR); } /* WaClearFlowControlGpgpuContextSave:skl,bxt,kbl,glk,cfl */ /* WaDisablePartialInstShootdown:skl,bxt,kbl,glk,cfl */ - wa_masked_en(wal, GEN8_ROW_CHICKEN, - FLOW_CONTROL_ENABLE | - PARTIAL_INSTRUCTION_SHOOTDOWN_DISABLE); + wa_mcr_masked_en(wal, GEN8_ROW_CHICKEN, + FLOW_CONTROL_ENABLE | + PARTIAL_INSTRUCTION_SHOOTDOWN_DISABLE); /* WaEnableYV12BugFixInHalfSliceChicken7:skl,bxt,kbl,glk,cfl */ /* WaEnableSamplerGPGPUPreemptionSupport:skl,bxt,kbl,cfl */ - wa_masked_en(wal, GEN9_HALF_SLICE_CHICKEN7, - GEN9_ENABLE_YV12_BUGFIX | - GEN9_ENABLE_GPGPU_PREEMPTION); + wa_mcr_masked_en(wal, GEN9_HALF_SLICE_CHICKEN7, + GEN9_ENABLE_YV12_BUGFIX | + GEN9_ENABLE_GPGPU_PREEMPTION); /* Wa4x4STCOptimizationDisable:skl,bxt,kbl,glk,cfl */ /* WaDisablePartialResolveInVc:skl,bxt,kbl,cfl */ @@ -356,8 +433,8 @@ static void gen9_ctx_workarounds_init(struct intel_engine_cs *engine, GEN9_PARTIAL_RESOLVE_IN_VC_DISABLE); /* WaCcsTlbPrefetchDisable:skl,bxt,kbl,glk,cfl */ - wa_masked_dis(wal, GEN9_HALF_SLICE_CHICKEN5, - GEN9_CCS_TLB_PREFETCH_ENABLE); + wa_mcr_masked_dis(wal, GEN9_HALF_SLICE_CHICKEN5, + GEN9_CCS_TLB_PREFETCH_ENABLE); /* WaForceContextSaveRestoreNonCoherent:skl,bxt,kbl,cfl */ wa_masked_en(wal, HDC_CHICKEN0, @@ -386,11 +463,11 @@ static void gen9_ctx_workarounds_init(struct intel_engine_cs *engine, IS_KABYLAKE(i915) || IS_COFFEELAKE(i915) || IS_COMETLAKE(i915)) - wa_masked_en(wal, HALF_SLICE_CHICKEN3, - GEN8_SAMPLER_POWER_BYPASS_DIS); + wa_mcr_masked_en(wal, GEN8_HALF_SLICE_CHICKEN3, + GEN8_SAMPLER_POWER_BYPASS_DIS); /* WaDisableSTUnitPowerOptimization:skl,bxt,kbl,glk,cfl */ - wa_masked_en(wal, HALF_SLICE_CHICKEN2, GEN8_ST_PO_DISABLE); + wa_mcr_masked_en(wal, HALF_SLICE_CHICKEN2, GEN8_ST_PO_DISABLE); /* * Supporting preemption with fine-granularity requires changes in the @@ -469,8 +546,8 @@ static void bxt_ctx_workarounds_init(struct intel_engine_cs *engine, gen9_ctx_workarounds_init(engine, wal); /* WaDisableThreadStallDopClockGating:bxt */ - wa_masked_en(wal, GEN8_ROW_CHICKEN, - STALL_DOP_GATING_DISABLE); + wa_mcr_masked_en(wal, GEN8_ROW_CHICKEN, + STALL_DOP_GATING_DISABLE); /* WaToEnableHwFixForPushConstHWBug:bxt */ wa_masked_en(wal, COMMON_SLICE_CHICKEN2, @@ -490,8 +567,8 @@ static void kbl_ctx_workarounds_init(struct intel_engine_cs *engine, GEN8_SBE_DISABLE_REPLAY_BUF_OPTIMIZATION); /* WaDisableSbeCacheDispatchPortSharing:kbl */ - wa_masked_en(wal, GEN7_HALF_SLICE_CHICKEN1, - GEN7_SBE_SS_CACHE_DISPATCH_PORT_SHARING_DISABLE); + wa_mcr_masked_en(wal, GEN8_HALF_SLICE_CHICKEN1, + GEN7_SBE_SS_CACHE_DISPATCH_PORT_SHARING_DISABLE); } static void glk_ctx_workarounds_init(struct intel_engine_cs *engine, @@ -514,8 +591,8 @@ static void cfl_ctx_workarounds_init(struct intel_engine_cs *engine, GEN8_SBE_DISABLE_REPLAY_BUF_OPTIMIZATION); /* WaDisableSbeCacheDispatchPortSharing:cfl */ - wa_masked_en(wal, GEN7_HALF_SLICE_CHICKEN1, - GEN7_SBE_SS_CACHE_DISPATCH_PORT_SHARING_DISABLE); + wa_mcr_masked_en(wal, GEN8_HALF_SLICE_CHICKEN1, + GEN7_SBE_SS_CACHE_DISPATCH_PORT_SHARING_DISABLE); } static void icl_ctx_workarounds_init(struct intel_engine_cs *engine, @@ -534,13 +611,13 @@ static void icl_ctx_workarounds_init(struct intel_engine_cs *engine, * (the register is whitelisted in hardware now, so UMDs can opt in * for coherency if they have a good reason). */ - wa_masked_en(wal, ICL_HDC_MODE, HDC_FORCE_NON_COHERENT); + wa_mcr_masked_en(wal, ICL_HDC_MODE, HDC_FORCE_NON_COHERENT); /* WaEnableFloatBlendOptimization:icl */ - wa_add(wal, GEN10_CACHE_MODE_SS, 0, - _MASKED_BIT_ENABLE(FLOAT_BLEND_OPTIMIZATION_ENABLE), - 0 /* write-only, so skip validation */, - true); + wa_mcr_add(wal, GEN10_CACHE_MODE_SS, 0, + _MASKED_BIT_ENABLE(FLOAT_BLEND_OPTIMIZATION_ENABLE), + 0 /* write-only, so skip validation */, + true); /* WaDisableGPGPUMidThreadPreemption:icl */ wa_masked_field_set(wal, GEN8_CS_CHICKEN1, @@ -548,8 +625,8 @@ static void icl_ctx_workarounds_init(struct intel_engine_cs *engine, GEN9_PREEMPT_GPGPU_THREAD_GROUP_LEVEL); /* allow headerless messages for preemptible GPGPU context */ - wa_masked_en(wal, GEN10_SAMPLER_MODE, - GEN11_SAMPLER_ENABLE_HEADLESS_MSG); + wa_mcr_masked_en(wal, GEN10_SAMPLER_MODE, + GEN11_SAMPLER_ENABLE_HEADLESS_MSG); /* Wa_1604278689:icl,ehl */ wa_write(wal, IVB_FBC_RT_BASE, 0xFFFFFFFF & ~ILK_FBC_RT_VALID); @@ -558,7 +635,7 @@ static void icl_ctx_workarounds_init(struct intel_engine_cs *engine, 0xFFFFFFFF); /* Wa_1406306137:icl,ehl */ - wa_masked_en(wal, GEN9_ROW_CHICKEN4, GEN11_DIS_PICK_2ND_EU); + wa_mcr_masked_en(wal, GEN9_ROW_CHICKEN4, GEN11_DIS_PICK_2ND_EU); } /* @@ -569,13 +646,13 @@ static void dg2_ctx_gt_tuning_init(struct intel_engine_cs *engine, struct i915_wa_list *wal) { wa_masked_en(wal, CHICKEN_RASTER_2, TBIMR_FAST_CLIP); - wa_write_clr_set(wal, GEN11_L3SQCREG5, L3_PWM_TIMER_INIT_VAL_MASK, - REG_FIELD_PREP(L3_PWM_TIMER_INIT_VAL_MASK, 0x7f)); - wa_add(wal, - FF_MODE2, - FF_MODE2_TDS_TIMER_MASK, - FF_MODE2_TDS_TIMER_128, - 0, false); + wa_mcr_write_clr_set(wal, XEHP_L3SQCREG5, L3_PWM_TIMER_INIT_VAL_MASK, + REG_FIELD_PREP(L3_PWM_TIMER_INIT_VAL_MASK, 0x7f)); + wa_mcr_add(wal, + XEHP_FF_MODE2, + FF_MODE2_TDS_TIMER_MASK, + FF_MODE2_TDS_TIMER_128, + 0, false); } /* @@ -599,7 +676,7 @@ static void gen12_ctx_gt_tuning_init(struct intel_engine_cs *engine, * verification is ignored. */ wa_add(wal, - FF_MODE2, + GEN12_FF_MODE2, FF_MODE2_TDS_TIMER_MASK, FF_MODE2_TDS_TIMER_128, 0, false); @@ -608,6 +685,8 @@ static void gen12_ctx_gt_tuning_init(struct intel_engine_cs *engine, static void gen12_ctx_workarounds_init(struct intel_engine_cs *engine, struct i915_wa_list *wal) { + struct drm_i915_private *i915 = engine->i915; + gen12_ctx_gt_tuning_init(engine, wal); /* @@ -637,10 +716,14 @@ static void gen12_ctx_workarounds_init(struct intel_engine_cs *engine, * to Wa_1608008084. */ wa_add(wal, - FF_MODE2, + GEN12_FF_MODE2, FF_MODE2_GS_TIMER_MASK, FF_MODE2_GS_TIMER_224, 0, false); + + if (!IS_DG1(i915)) + /* Wa_1806527549 */ + wa_masked_en(wal, HIZ_CHICKEN, HZ_DEPTH_TEST_LE_GE_OPT_DISABLE); } static void dg1_ctx_workarounds_init(struct intel_engine_cs *engine, @@ -664,27 +747,27 @@ static void dg2_ctx_workarounds_init(struct intel_engine_cs *engine, /* Wa_16011186671:dg2_g11 */ if (IS_DG2_GRAPHICS_STEP(engine->i915, G11, STEP_A0, STEP_B0)) { - wa_masked_dis(wal, VFLSKPD, DIS_MULT_MISS_RD_SQUASH); - wa_masked_en(wal, VFLSKPD, DIS_OVER_FETCH_CACHE); + wa_mcr_masked_dis(wal, VFLSKPD, DIS_MULT_MISS_RD_SQUASH); + wa_mcr_masked_en(wal, VFLSKPD, DIS_OVER_FETCH_CACHE); } if (IS_DG2_GRAPHICS_STEP(engine->i915, G10, STEP_A0, STEP_B0)) { /* Wa_14010469329:dg2_g10 */ - wa_masked_en(wal, GEN11_COMMON_SLICE_CHICKEN3, - XEHP_DUAL_SIMD8_SEQ_MERGE_DISABLE); + wa_mcr_masked_en(wal, XEHP_COMMON_SLICE_CHICKEN3, + XEHP_DUAL_SIMD8_SEQ_MERGE_DISABLE); /* * Wa_22010465075:dg2_g10 * Wa_22010613112:dg2_g10 * Wa_14010698770:dg2_g10 */ - wa_masked_en(wal, GEN11_COMMON_SLICE_CHICKEN3, - GEN12_DISABLE_CPS_AWARE_COLOR_PIPE); + wa_mcr_masked_en(wal, XEHP_COMMON_SLICE_CHICKEN3, + GEN12_DISABLE_CPS_AWARE_COLOR_PIPE); } /* Wa_16013271637:dg2 */ - wa_masked_en(wal, SLICE_COMMON_ECO_CHICKEN1, - MSC_MSAA_REODER_BUF_BYPASS_DISABLE); + wa_mcr_masked_en(wal, XEHP_SLICE_COMMON_ECO_CHICKEN1, + MSC_MSAA_REODER_BUF_BYPASS_DISABLE); /* Wa_14014947963:dg2 */ if (IS_DG2_GRAPHICS_STEP(engine->i915, G10, STEP_B0, STEP_FOREVER) || @@ -768,7 +851,7 @@ __intel_engine_init_ctx_wa(struct intel_engine_cs *engine, { struct drm_i915_private *i915 = engine->i915; - wa_init_start(wal, name, engine->name); + wa_init_start(wal, engine->gt, name, engine->name); /* Applies to all engines */ /* @@ -1076,18 +1159,23 @@ static void __set_mcr_steering(struct i915_wa_list *wal, wa_write_clr_set(wal, steering_reg, mcr_mask, mcr); } -static void __add_mcr_wa(struct intel_gt *gt, struct i915_wa_list *wal, - unsigned int slice, unsigned int subslice) +static void debug_dump_steering(struct intel_gt *gt) { struct drm_printer p = drm_debug_printer("MCR Steering:"); + if (drm_debug_enabled(DRM_UT_DRIVER)) + intel_gt_mcr_report_steering(&p, gt, false); +} + +static void __add_mcr_wa(struct intel_gt *gt, struct i915_wa_list *wal, + unsigned int slice, unsigned int subslice) +{ __set_mcr_steering(wal, GEN8_MCR_SELECTOR, slice, subslice); gt->default_steering.groupid = slice; gt->default_steering.instanceid = subslice; - if (drm_debug_enabled(DRM_UT_DRIVER)) - intel_gt_mcr_report_steering(&p, gt, false); + debug_dump_steering(gt); } static void @@ -1181,6 +1269,9 @@ xehp_init_mcr(struct intel_gt *gt, struct i915_wa_list *wal) gt->steering_table[MSLICE] = NULL; } + if (IS_XEHPSDV(gt->i915) && slice_mask & BIT(0)) + gt->steering_table[GAM] = NULL; + slice = __ffs(slice_mask); subslice = intel_sseu_find_first_xehp_dss(sseu, GEN_DSS_PER_GSLICE, slice) % GEN_DSS_PER_GSLICE; @@ -1198,6 +1289,13 @@ xehp_init_mcr(struct intel_gt *gt, struct i915_wa_list *wal) */ __set_mcr_steering(wal, MCFG_MCR_SELECTOR, 0, 2); __set_mcr_steering(wal, SF_MCR_SELECTOR, 0, 2); + + /* + * On DG2, GAM registers have a dedicated steering control register + * and must always be programmed to a hardcoded groupid of "1." + */ + if (IS_DG2(gt->i915)) + __set_mcr_steering(wal, GAM_MCR_SELECTOR, 1, 0); } static void @@ -1254,22 +1352,22 @@ icl_gt_workarounds_init(struct intel_gt *gt, struct i915_wa_list *wal) PSDUNIT_CLKGATE_DIS); /* Wa_1406680159:icl,ehl */ - wa_write_or(wal, - SUBSLICE_UNIT_LEVEL_CLKGATE, - GWUNIT_CLKGATE_DIS); + wa_mcr_write_or(wal, + GEN11_SUBSLICE_UNIT_LEVEL_CLKGATE, + GWUNIT_CLKGATE_DIS); /* Wa_1607087056:icl,ehl,jsl */ if (IS_ICELAKE(i915) || IS_JSL_EHL_GRAPHICS_STEP(i915, STEP_A0, STEP_B0)) wa_write_or(wal, - SLICE_UNIT_LEVEL_CLKGATE, + GEN11_SLICE_UNIT_LEVEL_CLKGATE, L3_CLKGATE_DIS | L3_CR2X_CLKGATE_DIS); /* * This is not a documented workaround, but rather an optimization * to reduce sampler power. */ - wa_write_clr(wal, GEN10_DFR_RATIO_EN_AND_CHICKEN, DFR_DISABLE); + wa_mcr_write_clr(wal, GEN10_DFR_RATIO_EN_AND_CHICKEN, DFR_DISABLE); } /* @@ -1303,7 +1401,7 @@ gen12_gt_workarounds_init(struct intel_gt *gt, struct i915_wa_list *wal) wa_14011060649(gt, wal); /* Wa_14011059788:tgl,rkl,adl-s,dg1,adl-p */ - wa_write_or(wal, GEN10_DFR_RATIO_EN_AND_CHICKEN, DFR_DISABLE); + wa_mcr_write_or(wal, GEN10_DFR_RATIO_EN_AND_CHICKEN, DFR_DISABLE); } static void @@ -1315,14 +1413,14 @@ tgl_gt_workarounds_init(struct intel_gt *gt, struct i915_wa_list *wal) /* Wa_1409420604:tgl */ if (IS_TGL_UY_GRAPHICS_STEP(i915, STEP_A0, STEP_B0)) - wa_write_or(wal, - SUBSLICE_UNIT_LEVEL_CLKGATE2, - CPSSUNIT_CLKGATE_DIS); + wa_mcr_write_or(wal, + SUBSLICE_UNIT_LEVEL_CLKGATE2, + CPSSUNIT_CLKGATE_DIS); /* Wa_1607087056:tgl also know as BUG:1409180338 */ if (IS_TGL_UY_GRAPHICS_STEP(i915, STEP_A0, STEP_B0)) wa_write_or(wal, - SLICE_UNIT_LEVEL_CLKGATE, + GEN11_SLICE_UNIT_LEVEL_CLKGATE, L3_CLKGATE_DIS | L3_CR2X_CLKGATE_DIS); /* Wa_1408615072:tgl[a0] */ @@ -1341,14 +1439,14 @@ dg1_gt_workarounds_init(struct intel_gt *gt, struct i915_wa_list *wal) /* Wa_1607087056:dg1 */ if (IS_DG1_GRAPHICS_STEP(i915, STEP_A0, STEP_B0)) wa_write_or(wal, - SLICE_UNIT_LEVEL_CLKGATE, + GEN11_SLICE_UNIT_LEVEL_CLKGATE, L3_CLKGATE_DIS | L3_CR2X_CLKGATE_DIS); /* Wa_1409420604:dg1 */ if (IS_DG1(i915)) - wa_write_or(wal, - SUBSLICE_UNIT_LEVEL_CLKGATE2, - CPSSUNIT_CLKGATE_DIS); + wa_mcr_write_or(wal, + SUBSLICE_UNIT_LEVEL_CLKGATE2, + CPSSUNIT_CLKGATE_DIS); /* Wa_1408615072:dg1 */ /* Empirical testing shows this register is unaffected by engine reset. */ @@ -1365,7 +1463,7 @@ xehpsdv_gt_workarounds_init(struct intel_gt *gt, struct i915_wa_list *wal) xehp_init_mcr(gt, wal); /* Wa_1409757795:xehpsdv */ - wa_write_or(wal, SCCGCTL94DC, CG3DDISURB); + wa_mcr_write_or(wal, SCCGCTL94DC, CG3DDISURB); /* Wa_16011155590:xehpsdv */ if (IS_XEHPSDV_GRAPHICS_STEP(i915, STEP_A0, STEP_B0)) @@ -1445,8 +1543,8 @@ dg2_gt_workarounds_init(struct intel_gt *gt, struct i915_wa_list *wal) CG3DDISCFEG_CLKGATE_DIS); /* Wa_14011006942:dg2 */ - wa_write_or(wal, SUBSLICE_UNIT_LEVEL_CLKGATE, - DSS_ROUTER_CLKGATE_DIS); + wa_mcr_write_or(wal, GEN11_SUBSLICE_UNIT_LEVEL_CLKGATE, + DSS_ROUTER_CLKGATE_DIS); } if (IS_DG2_GRAPHICS_STEP(gt->i915, G10, STEP_A0, STEP_B0)) { @@ -1457,7 +1555,7 @@ dg2_gt_workarounds_init(struct intel_gt *gt, struct i915_wa_list *wal) wa_write_or(wal, UNSLCGCTL9444, LTCDD_CLKGATE_DIS); /* Wa_14011371254:dg2_g10 */ - wa_write_or(wal, SLICE_UNIT_LEVEL_CLKGATE, NODEDSS_CLKGATE_DIS); + wa_mcr_write_or(wal, XEHP_SLICE_UNIT_LEVEL_CLKGATE, NODEDSS_CLKGATE_DIS); /* Wa_14011431319:dg2_g10 */ wa_write_or(wal, UNSLCGCTL9440, GAMTLBOACS_CLKGATE_DIS | @@ -1493,21 +1591,21 @@ dg2_gt_workarounds_init(struct intel_gt *gt, struct i915_wa_list *wal) GAMEDIA_CLKGATE_DIS); /* Wa_14011028019:dg2_g10 */ - wa_write_or(wal, SSMCGCTL9530, RTFUNIT_CLKGATE_DIS); + wa_mcr_write_or(wal, SSMCGCTL9530, RTFUNIT_CLKGATE_DIS); } /* Wa_14014830051:dg2 */ - wa_write_clr(wal, SARB_CHICKEN1, COMP_CKN_IN); + wa_mcr_write_clr(wal, SARB_CHICKEN1, COMP_CKN_IN); /* * The following are not actually "workarounds" but rather * recommended tuning settings documented in the bspec's * performance guide section. */ - wa_write_or(wal, GEN12_SQCM, EN_32B_ACCESS); + wa_mcr_write_or(wal, XEHP_SQCM, EN_32B_ACCESS); /* Wa_14015795083 */ - wa_write_clr(wal, GEN7_MISCCPCTL, GEN12_DOP_CLOCK_GATE_RENDER_ENABLE); + wa_mcr_write_clr(wal, GEN8_MISCCPCTL, GEN12_DOP_CLOCK_GATE_RENDER_ENABLE); } static void @@ -1516,7 +1614,27 @@ pvc_gt_workarounds_init(struct intel_gt *gt, struct i915_wa_list *wal) pvc_init_mcr(gt, wal); /* Wa_14015795083 */ - wa_write_clr(wal, GEN7_MISCCPCTL, GEN12_DOP_CLOCK_GATE_RENDER_ENABLE); + wa_mcr_write_clr(wal, GEN8_MISCCPCTL, GEN12_DOP_CLOCK_GATE_RENDER_ENABLE); +} + +static void +xelpg_gt_workarounds_init(struct intel_gt *gt, struct i915_wa_list *wal) +{ + /* FIXME: Actual workarounds will be added in future patch(es) */ + + /* + * Unlike older platforms, we no longer setup implicit steering here; + * all MCR accesses are explicitly steered. + */ + debug_dump_steering(gt); +} + +static void +xelpmp_gt_workarounds_init(struct intel_gt *gt, struct i915_wa_list *wal) +{ + /* FIXME: Actual workarounds will be added in future patch(es) */ + + debug_dump_steering(gt); } static void @@ -1524,7 +1642,18 @@ gt_init_workarounds(struct intel_gt *gt, struct i915_wa_list *wal) { struct drm_i915_private *i915 = gt->i915; - if (IS_PONTEVECCHIO(i915)) + if (gt->type == GT_MEDIA) { + if (MEDIA_VER(i915) >= 13) + xelpmp_gt_workarounds_init(gt, wal); + else + MISSING_CASE(MEDIA_VER(i915)); + + return; + } + + if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 70)) + xelpg_gt_workarounds_init(gt, wal); + else if (IS_PONTEVECCHIO(i915)) pvc_gt_workarounds_init(gt, wal); else if (IS_DG2(i915)) dg2_gt_workarounds_init(gt, wal); @@ -1572,7 +1701,7 @@ void intel_gt_init_workarounds(struct intel_gt *gt) { struct i915_wa_list *wal = >->wa_list; - wa_init_start(wal, "GT", "global"); + wa_init_start(wal, gt, "GT", "global"); gt_init_workarounds(gt, wal); wa_init_finish(wal); } @@ -1594,12 +1723,14 @@ wal_get_fw_for_rmw(struct intel_uncore *uncore, const struct i915_wa_list *wal) } static bool -wa_verify(const struct i915_wa *wa, u32 cur, const char *name, const char *from) +wa_verify(struct intel_gt *gt, const struct i915_wa *wa, u32 cur, + const char *name, const char *from) { if ((cur ^ wa->set) & wa->read) { - DRM_ERROR("%s workaround lost on %s! (reg[%x]=0x%x, relevant bits were 0x%x vs expected 0x%x)\n", - name, from, i915_mmio_reg_offset(wa->reg), - cur, cur & wa->read, wa->set & wa->read); + drm_err(>->i915->drm, + "%s workaround lost on %s! (reg[%x]=0x%x, relevant bits were 0x%x vs expected 0x%x)\n", + name, from, i915_mmio_reg_offset(wa->reg), + cur, cur & wa->read, wa->set & wa->read); return false; } @@ -1607,9 +1738,9 @@ wa_verify(const struct i915_wa *wa, u32 cur, const char *name, const char *from) return true; } -static void -wa_list_apply(struct intel_gt *gt, const struct i915_wa_list *wal) +static void wa_list_apply(const struct i915_wa_list *wal) { + struct intel_gt *gt = wal->gt; struct intel_uncore *uncore = gt->uncore; enum forcewake_domains fw; unsigned long flags; @@ -1628,14 +1759,25 @@ wa_list_apply(struct intel_gt *gt, const struct i915_wa_list *wal) u32 val, old = 0; /* open-coded rmw due to steering */ - old = wa->clr ? intel_gt_mcr_read_any_fw(gt, wa->reg) : 0; + if (wa->clr) + old = wa->is_mcr ? + intel_gt_mcr_read_any_fw(gt, wa->mcr_reg) : + intel_uncore_read_fw(uncore, wa->reg); val = (old & ~wa->clr) | wa->set; - if (val != old || !wa->clr) - intel_uncore_write_fw(uncore, wa->reg, val); + if (val != old || !wa->clr) { + if (wa->is_mcr) + intel_gt_mcr_multicast_write_fw(gt, wa->mcr_reg, val); + else + intel_uncore_write_fw(uncore, wa->reg, val); + } + + if (IS_ENABLED(CONFIG_DRM_I915_DEBUG_GEM)) { + u32 val = wa->is_mcr ? + intel_gt_mcr_read_any_fw(gt, wa->mcr_reg) : + intel_uncore_read_fw(uncore, wa->reg); - if (IS_ENABLED(CONFIG_DRM_I915_DEBUG_GEM)) - wa_verify(wa, intel_gt_mcr_read_any_fw(gt, wa->reg), - wal->name, "application"); + wa_verify(gt, wa, val, wal->name, "application"); + } } intel_uncore_forcewake_put__locked(uncore, fw); @@ -1644,7 +1786,7 @@ wa_list_apply(struct intel_gt *gt, const struct i915_wa_list *wal) void intel_gt_apply_workarounds(struct intel_gt *gt) { - wa_list_apply(gt, >->wa_list); + wa_list_apply(>->wa_list); } static bool wa_list_verify(struct intel_gt *gt, @@ -1664,8 +1806,9 @@ static bool wa_list_verify(struct intel_gt *gt, intel_uncore_forcewake_get__locked(uncore, fw); for (i = 0, wa = wal->list; i < wal->count; i++, wa++) - ok &= wa_verify(wa, - intel_gt_mcr_read_any_fw(gt, wa->reg), + ok &= wa_verify(wal->gt, wa, wa->is_mcr ? + intel_gt_mcr_read_any_fw(gt, wa->mcr_reg) : + intel_uncore_read_fw(uncore, wa->reg), wal->name, from); intel_uncore_forcewake_put__locked(uncore, fw); @@ -1712,11 +1855,35 @@ whitelist_reg_ext(struct i915_wa_list *wal, i915_reg_t reg, u32 flags) } static void +whitelist_mcr_reg_ext(struct i915_wa_list *wal, i915_mcr_reg_t reg, u32 flags) +{ + struct i915_wa wa = { + .mcr_reg = reg, + .is_mcr = 1, + }; + + if (GEM_DEBUG_WARN_ON(wal->count >= RING_MAX_NONPRIV_SLOTS)) + return; + + if (GEM_DEBUG_WARN_ON(!is_nonpriv_flags_valid(flags))) + return; + + wa.mcr_reg.reg |= flags; + _wa_add(wal, &wa); +} + +static void whitelist_reg(struct i915_wa_list *wal, i915_reg_t reg) { whitelist_reg_ext(wal, reg, RING_FORCE_TO_NONPRIV_ACCESS_RW); } +static void +whitelist_mcr_reg(struct i915_wa_list *wal, i915_mcr_reg_t reg) +{ + whitelist_mcr_reg_ext(wal, reg, RING_FORCE_TO_NONPRIV_ACCESS_RW); +} + static void gen9_whitelist_build(struct i915_wa_list *w) { /* WaVFEStateAfterPipeControlwithMediaStateClear:skl,bxt,glk,cfl */ @@ -1742,7 +1909,7 @@ static void skl_whitelist_build(struct intel_engine_cs *engine) gen9_whitelist_build(w); /* WaDisableLSQCROPERFforOCL:skl */ - whitelist_reg(w, GEN8_L3SQCREG4); + whitelist_mcr_reg(w, GEN8_L3SQCREG4); } static void bxt_whitelist_build(struct intel_engine_cs *engine) @@ -1763,7 +1930,7 @@ static void kbl_whitelist_build(struct intel_engine_cs *engine) gen9_whitelist_build(w); /* WaDisableLSQCROPERFforOCL:kbl */ - whitelist_reg(w, GEN8_L3SQCREG4); + whitelist_mcr_reg(w, GEN8_L3SQCREG4); } static void glk_whitelist_build(struct intel_engine_cs *engine) @@ -1828,10 +1995,10 @@ static void icl_whitelist_build(struct intel_engine_cs *engine) switch (engine->class) { case RENDER_CLASS: /* WaAllowUMDToModifyHalfSliceChicken7:icl */ - whitelist_reg(w, GEN9_HALF_SLICE_CHICKEN7); + whitelist_mcr_reg(w, GEN9_HALF_SLICE_CHICKEN7); /* WaAllowUMDToModifySamplerMode:icl */ - whitelist_reg(w, GEN10_SAMPLER_MODE); + whitelist_mcr_reg(w, GEN10_SAMPLER_MODE); /* WaEnableStateCacheRedirectToCS:icl */ whitelist_reg(w, GEN9_SLICE_COMMON_ECO_CHICKEN1); @@ -1987,7 +2154,7 @@ void intel_engine_init_whitelist(struct intel_engine_cs *engine) struct drm_i915_private *i915 = engine->i915; struct i915_wa_list *w = &engine->whitelist; - wa_init_start(w, "whitelist", engine->name); + wa_init_start(w, engine->gt, "whitelist", engine->name); if (IS_PONTEVECCHIO(i915)) pvc_whitelist_build(engine); @@ -2107,24 +2274,21 @@ rcs_engine_wa_init(struct intel_engine_cs *engine, struct i915_wa_list *wal) if (IS_DG2_GRAPHICS_STEP(i915, G11, STEP_A0, STEP_B0)) { /* Wa_14013392000:dg2_g11 */ - wa_masked_en(wal, GEN7_ROW_CHICKEN2, GEN12_ENABLE_LARGE_GRF_MODE); - - /* Wa_16011620976:dg2_g11 */ - wa_write_or(wal, LSC_CHICKEN_BIT_0_UDW, DIS_CHAIN_2XSIMD8); + wa_mcr_masked_en(wal, GEN8_ROW_CHICKEN2, GEN12_ENABLE_LARGE_GRF_MODE); } if (IS_DG2_GRAPHICS_STEP(i915, G10, STEP_B0, STEP_FOREVER) || IS_DG2_G11(i915) || IS_DG2_G12(i915)) { /* Wa_1509727124:dg2 */ - wa_masked_en(wal, GEN10_SAMPLER_MODE, - SC_DISABLE_POWER_OPTIMIZATION_EBB); + wa_mcr_masked_en(wal, GEN10_SAMPLER_MODE, + SC_DISABLE_POWER_OPTIMIZATION_EBB); } if (IS_DG2_GRAPHICS_STEP(i915, G10, STEP_A0, STEP_B0) || IS_DG2_GRAPHICS_STEP(i915, G11, STEP_A0, STEP_B0)) { /* Wa_14012419201:dg2 */ - wa_masked_en(wal, GEN9_ROW_CHICKEN4, - GEN12_DISABLE_HDR_PAST_PAYLOAD_HOLD_FIX); + wa_mcr_masked_en(wal, GEN9_ROW_CHICKEN4, + GEN12_DISABLE_HDR_PAST_PAYLOAD_HOLD_FIX); } if (IS_DG2_GRAPHICS_STEP(i915, G10, STEP_B0, STEP_C0) || @@ -2133,13 +2297,13 @@ rcs_engine_wa_init(struct intel_engine_cs *engine, struct i915_wa_list *wal) * Wa_22012826095:dg2 * Wa_22013059131:dg2 */ - wa_write_clr_set(wal, LSC_CHICKEN_BIT_0_UDW, - MAXREQS_PER_BANK, - REG_FIELD_PREP(MAXREQS_PER_BANK, 2)); + wa_mcr_write_clr_set(wal, LSC_CHICKEN_BIT_0_UDW, + MAXREQS_PER_BANK, + REG_FIELD_PREP(MAXREQS_PER_BANK, 2)); /* Wa_22013059131:dg2 */ - wa_write_or(wal, LSC_CHICKEN_BIT_0, - FORCE_1_SUB_MESSAGE_PER_FRAGMENT); + wa_mcr_write_or(wal, LSC_CHICKEN_BIT_0, + FORCE_1_SUB_MESSAGE_PER_FRAGMENT); } /* Wa_1308578152:dg2_g10 when first gslice is fused off */ @@ -2152,19 +2316,19 @@ rcs_engine_wa_init(struct intel_engine_cs *engine, struct i915_wa_list *wal) if (IS_DG2_GRAPHICS_STEP(i915, G10, STEP_B0, STEP_FOREVER) || IS_DG2_G11(i915) || IS_DG2_G12(i915)) { /* Wa_22013037850:dg2 */ - wa_write_or(wal, LSC_CHICKEN_BIT_0_UDW, - DISABLE_128B_EVICTION_COMMAND_UDW); + wa_mcr_write_or(wal, LSC_CHICKEN_BIT_0_UDW, + DISABLE_128B_EVICTION_COMMAND_UDW); /* Wa_22012856258:dg2 */ - wa_masked_en(wal, GEN7_ROW_CHICKEN2, - GEN12_DISABLE_READ_SUPPRESSION); + wa_mcr_masked_en(wal, GEN8_ROW_CHICKEN2, + GEN12_DISABLE_READ_SUPPRESSION); /* * Wa_22010960976:dg2 * Wa_14013347512:dg2 */ - wa_masked_dis(wal, GEN12_HDC_CHICKEN0, - LSC_L1_FLUSH_CTL_3D_DATAPORT_FLUSH_EVENTS_MASK); + wa_mcr_masked_dis(wal, XEHP_HDC_CHICKEN0, + LSC_L1_FLUSH_CTL_3D_DATAPORT_FLUSH_EVENTS_MASK); } if (IS_DG2_GRAPHICS_STEP(i915, G10, STEP_A0, STEP_B0)) { @@ -2172,8 +2336,8 @@ rcs_engine_wa_init(struct intel_engine_cs *engine, struct i915_wa_list *wal) * Wa_1608949956:dg2_g10 * Wa_14010198302:dg2_g10 */ - wa_masked_en(wal, GEN8_ROW_CHICKEN, - MDQ_ARBITRATION_MODE | UGM_BACKUP_MODE); + wa_mcr_masked_en(wal, GEN8_ROW_CHICKEN, + MDQ_ARBITRATION_MODE | UGM_BACKUP_MODE); /* * Wa_14010918519:dg2_g10 @@ -2181,31 +2345,31 @@ rcs_engine_wa_init(struct intel_engine_cs *engine, struct i915_wa_list *wal) * LSC_CHICKEN_BIT_0 always reads back as 0 is this stepping, * so ignoring verification. */ - wa_add(wal, LSC_CHICKEN_BIT_0_UDW, 0, - FORCE_SLM_FENCE_SCOPE_TO_TILE | FORCE_UGM_FENCE_SCOPE_TO_TILE, - 0, false); + wa_mcr_add(wal, LSC_CHICKEN_BIT_0_UDW, 0, + FORCE_SLM_FENCE_SCOPE_TO_TILE | FORCE_UGM_FENCE_SCOPE_TO_TILE, + 0, false); } if (IS_DG2_GRAPHICS_STEP(i915, G10, STEP_A0, STEP_B0)) { /* Wa_22010430635:dg2 */ - wa_masked_en(wal, - GEN9_ROW_CHICKEN4, - GEN12_DISABLE_GRF_CLEAR); + wa_mcr_masked_en(wal, + GEN9_ROW_CHICKEN4, + GEN12_DISABLE_GRF_CLEAR); /* Wa_14010648519:dg2 */ - wa_write_or(wal, XEHP_L3NODEARBCFG, XEHP_LNESPARE); + wa_mcr_write_or(wal, XEHP_L3NODEARBCFG, XEHP_LNESPARE); } /* Wa_14013202645:dg2 */ if (IS_DG2_GRAPHICS_STEP(i915, G10, STEP_B0, STEP_C0) || IS_DG2_GRAPHICS_STEP(i915, G11, STEP_A0, STEP_B0)) - wa_write_or(wal, RT_CTRL, DIS_NULL_QUERY); + wa_mcr_write_or(wal, RT_CTRL, DIS_NULL_QUERY); /* Wa_22012532006:dg2 */ if (IS_DG2_GRAPHICS_STEP(engine->i915, G10, STEP_A0, STEP_C0) || IS_DG2_GRAPHICS_STEP(engine->i915, G11, STEP_A0, STEP_B0)) - wa_masked_en(wal, GEN9_HALF_SLICE_CHICKEN7, - DG2_DISABLE_ROUND_ENABLE_ALLOW_FOR_SSLA); + wa_mcr_masked_en(wal, GEN9_HALF_SLICE_CHICKEN7, + DG2_DISABLE_ROUND_ENABLE_ALLOW_FOR_SSLA); if (IS_DG2_GRAPHICS_STEP(engine->i915, G10, STEP_A0, STEP_B0)) { /* Wa_14010680813:dg2_g10 */ @@ -2216,17 +2380,16 @@ rcs_engine_wa_init(struct intel_engine_cs *engine, struct i915_wa_list *wal) if (IS_DG2_GRAPHICS_STEP(engine->i915, G10, STEP_A0, STEP_B0) || IS_DG2_GRAPHICS_STEP(engine->i915, G11, STEP_A0, STEP_B0)) { /* Wa_14012362059:dg2 */ - wa_write_or(wal, GEN12_MERT_MOD_CTRL, FORCE_MISS_FTLB); + wa_mcr_write_or(wal, XEHP_MERT_MOD_CTRL, FORCE_MISS_FTLB); } if (IS_DG2_GRAPHICS_STEP(i915, G11, STEP_B0, STEP_FOREVER) || IS_DG2_G10(i915)) { /* Wa_22014600077:dg2 */ - wa_add(wal, GEN10_CACHE_MODE_SS, 0, - _MASKED_BIT_ENABLE(ENABLE_EU_COUNT_FOR_TDL_FLUSH), - 0 /* Wa_14012342262 :write-only reg, so skip - verification */, - true); + wa_mcr_add(wal, GEN10_CACHE_MODE_SS, 0, + _MASKED_BIT_ENABLE(ENABLE_EU_COUNT_FOR_TDL_FLUSH), + 0 /* Wa_14012342262 write-only reg, so skip verification */, + true); } if (IS_DG1_GRAPHICS_STEP(i915, STEP_A0, STEP_B0) || @@ -2253,7 +2416,7 @@ rcs_engine_wa_init(struct intel_engine_cs *engine, struct i915_wa_list *wal) if (IS_ALDERLAKE_P(i915) || IS_ALDERLAKE_S(i915) || IS_DG1(i915) || IS_ROCKETLAKE(i915) || IS_TIGERLAKE(i915)) { /* Wa_1606931601:tgl,rkl,dg1,adl-s,adl-p */ - wa_masked_en(wal, GEN7_ROW_CHICKEN2, GEN12_DISABLE_EARLY_READ); + wa_mcr_masked_en(wal, GEN8_ROW_CHICKEN2, GEN12_DISABLE_EARLY_READ); /* * Wa_1407928979:tgl A* @@ -2282,14 +2445,14 @@ rcs_engine_wa_init(struct intel_engine_cs *engine, struct i915_wa_list *wal) IS_DG1_GRAPHICS_STEP(i915, STEP_A0, STEP_B0) || IS_ROCKETLAKE(i915) || IS_TIGERLAKE(i915)) { /* Wa_1409804808:tgl,rkl,dg1[a0],adl-s,adl-p */ - wa_masked_en(wal, GEN7_ROW_CHICKEN2, - GEN12_PUSH_CONST_DEREF_HOLD_DIS); + wa_mcr_masked_en(wal, GEN8_ROW_CHICKEN2, + GEN12_PUSH_CONST_DEREF_HOLD_DIS); /* * Wa_1409085225:tgl * Wa_14010229206:tgl,rkl,dg1[a0],adl-s,adl-p */ - wa_masked_en(wal, GEN9_ROW_CHICKEN4, GEN12_DISABLE_TDL_PUSH); + wa_mcr_masked_en(wal, GEN9_ROW_CHICKEN4, GEN12_DISABLE_TDL_PUSH); } if (IS_DG1_GRAPHICS_STEP(i915, STEP_A0, STEP_B0) || @@ -2313,9 +2476,9 @@ rcs_engine_wa_init(struct intel_engine_cs *engine, struct i915_wa_list *wal) if (IS_DG1(i915) || IS_ROCKETLAKE(i915) || IS_TIGERLAKE(i915) || IS_ALDERLAKE_S(i915) || IS_ALDERLAKE_P(i915)) { /* Wa_1406941453:tgl,rkl,dg1,adl-s,adl-p */ - wa_masked_en(wal, - GEN10_SAMPLER_MODE, - ENABLE_SMALLPL); + wa_mcr_masked_en(wal, + GEN10_SAMPLER_MODE, + ENABLE_SMALLPL); } if (GRAPHICS_VER(i915) == 11) { @@ -2349,9 +2512,9 @@ rcs_engine_wa_init(struct intel_engine_cs *engine, struct i915_wa_list *wal) * Wa_1405733216:icl * Formerly known as WaDisableCleanEvicts */ - wa_write_or(wal, - GEN8_L3SQCREG4, - GEN11_LQSC_CLEAN_EVICT_DISABLE); + wa_mcr_write_or(wal, + GEN8_L3SQCREG4, + GEN11_LQSC_CLEAN_EVICT_DISABLE); /* Wa_1606682166:icl */ wa_write_or(wal, @@ -2359,10 +2522,10 @@ rcs_engine_wa_init(struct intel_engine_cs *engine, struct i915_wa_list *wal) GEN7_DISABLE_SAMPLER_PREFETCH); /* Wa_1409178092:icl */ - wa_write_clr_set(wal, - GEN11_SCRATCH2, - GEN11_COHERENT_PARTIAL_WRITE_MERGE_ENABLE, - 0); + wa_mcr_write_clr_set(wal, + GEN11_SCRATCH2, + GEN11_COHERENT_PARTIAL_WRITE_MERGE_ENABLE, + 0); /* WaEnable32PlaneMode:icl */ wa_masked_en(wal, GEN9_CSFE_CHICKEN1_RCS, @@ -2389,12 +2552,64 @@ rcs_engine_wa_init(struct intel_engine_cs *engine, struct i915_wa_list *wal) FF_DOP_CLOCK_GATE_DISABLE); } - if (IS_GRAPHICS_VER(i915, 9, 12)) { - /* FtrPerCtxtPreemptionGranularityControl:skl,bxt,kbl,cfl,cnl,icl,tgl */ + /* + * Intel platforms that support fine-grained preemption (i.e., gen9 and + * beyond) allow the kernel-mode driver to choose between two different + * options for controlling preemption granularity and behavior. + * + * Option 1 (hardware default): + * Preemption settings are controlled in a global manner via + * kernel-only register CS_DEBUG_MODE1 (0x20EC). Any granularity + * and settings chosen by the kernel-mode driver will apply to all + * userspace clients. + * + * Option 2: + * Preemption settings are controlled on a per-context basis via + * register CS_CHICKEN1 (0x2580). CS_CHICKEN1 is saved/restored on + * context switch and is writable by userspace (e.g., via + * MI_LOAD_REGISTER_IMMEDIATE instructions placed in a batch buffer) + * which allows different userspace drivers/clients to select + * different settings, or to change those settings on the fly in + * response to runtime needs. This option was known by name + * "FtrPerCtxtPreemptionGranularityControl" at one time, although + * that name is somewhat misleading as other non-granularity + * preemption settings are also impacted by this decision. + * + * On Linux, our policy has always been to let userspace drivers + * control preemption granularity/settings (Option 2). This was + * originally mandatory on gen9 to prevent ABI breakage (old gen9 + * userspace developed before object-level preemption was enabled would + * not behave well if i915 were to go with Option 1 and enable that + * preemption in a global manner). On gen9 each context would have + * object-level preemption disabled by default (see + * WaDisable3DMidCmdPreemption in gen9_ctx_workarounds_init), but + * userspace drivers could opt-in to object-level preemption as they + * saw fit. For post-gen9 platforms, we continue to utilize Option 2; + * even though it is no longer necessary for ABI compatibility when + * enabling a new platform, it does ensure that userspace will be able + * to implement any workarounds that show up requiring temporary + * adjustments to preemption behavior at runtime. + * + * Notes/Workarounds: + * - Wa_14015141709: On DG2 and early steppings of MTL, + * CS_CHICKEN1[0] does not disable object-level preemption as + * it is supposed to (nor does CS_DEBUG_MODE1[0] if we had been + * using Option 1). Effectively this means userspace is unable + * to disable object-level preemption on these platforms/steppings + * despite the setting here. + * + * - Wa_16013994831: May require that userspace program + * CS_CHICKEN1[10] when certain runtime conditions are true. + * Userspace requires Option 2 to be in effect for their update of + * CS_CHICKEN1[10] to be effective. + * + * Other workarounds may appear in the future that will also require + * Option 2 behavior to allow proper userspace implementation. + */ + if (GRAPHICS_VER(i915) >= 9) wa_masked_en(wal, GEN7_FF_SLICE_CS_CHICKEN1, GEN9_FFSC_PERCTX_PREEMPT_CTRL); - } if (IS_SKYLAKE(i915) || IS_KABYLAKE(i915) || @@ -2420,36 +2635,36 @@ rcs_engine_wa_init(struct intel_engine_cs *engine, struct i915_wa_list *wal) GEN9_PREEMPT_GPGPU_SYNC_SWITCH_DISABLE); /* WaEnableLbsSlaRetryTimerDecrement:skl,bxt,kbl,glk,cfl */ - wa_write_or(wal, - BDW_SCRATCH1, - GEN9_LBS_SLA_RETRY_TIMER_DECREMENT_ENABLE); + wa_mcr_write_or(wal, + BDW_SCRATCH1, + GEN9_LBS_SLA_RETRY_TIMER_DECREMENT_ENABLE); /* WaProgramL3SqcReg1DefaultForPerf:bxt,glk */ if (IS_GEN9_LP(i915)) - wa_write_clr_set(wal, - GEN8_L3SQCREG1, - L3_PRIO_CREDITS_MASK, - L3_GENERAL_PRIO_CREDITS(62) | - L3_HIGH_PRIO_CREDITS(2)); + wa_mcr_write_clr_set(wal, + GEN8_L3SQCREG1, + L3_PRIO_CREDITS_MASK, + L3_GENERAL_PRIO_CREDITS(62) | + L3_HIGH_PRIO_CREDITS(2)); /* WaOCLCoherentLineFlush:skl,bxt,kbl,cfl */ - wa_write_or(wal, - GEN8_L3SQCREG4, - GEN8_LQSC_FLUSH_COHERENT_LINES); + wa_mcr_write_or(wal, + GEN8_L3SQCREG4, + GEN8_LQSC_FLUSH_COHERENT_LINES); /* Disable atomics in L3 to prevent unrecoverable hangs */ wa_write_clr_set(wal, GEN9_SCRATCH_LNCF1, GEN9_LNCF_NONIA_COHERENT_ATOMICS_ENABLE, 0); - wa_write_clr_set(wal, GEN8_L3SQCREG4, - GEN8_LQSQ_NONIA_COHERENT_ATOMICS_ENABLE, 0); - wa_write_clr_set(wal, GEN9_SCRATCH1, - EVICTION_PERF_FIX_ENABLE, 0); + wa_mcr_write_clr_set(wal, GEN8_L3SQCREG4, + GEN8_LQSQ_NONIA_COHERENT_ATOMICS_ENABLE, 0); + wa_mcr_write_clr_set(wal, GEN9_SCRATCH1, + EVICTION_PERF_FIX_ENABLE, 0); } if (IS_HASWELL(i915)) { /* WaSampleCChickenBitEnable:hsw */ wa_masked_en(wal, - HALF_SLICE_CHICKEN3, HSW_SAMPLE_C_PERFORMANCE); + HSW_HALF_SLICE_CHICKEN3, HSW_SAMPLE_C_PERFORMANCE); wa_masked_dis(wal, CACHE_MODE_0_GEN7, @@ -2657,7 +2872,7 @@ ccs_engine_wa_init(struct intel_engine_cs *engine, struct i915_wa_list *wal) { if (IS_PVC_CT_STEP(engine->i915, STEP_A0, STEP_C0)) { /* Wa_14014999345:pvc */ - wa_masked_en(wal, GEN10_CACHE_MODE_SS, DISABLE_ECC); + wa_mcr_masked_en(wal, GEN10_CACHE_MODE_SS, DISABLE_ECC); } } @@ -2683,8 +2898,8 @@ add_render_compute_tuning_settings(struct drm_i915_private *i915, } if (IS_DG2(i915)) { - wa_write_or(wal, XEHP_L3SCQREG7, BLEND_FILL_CACHING_OPT_DIS); - wa_write_clr_set(wal, RT_CTRL, STACKID_CTRL, STACKID_CTRL_512); + wa_mcr_write_or(wal, XEHP_L3SCQREG7, BLEND_FILL_CACHING_OPT_DIS); + wa_mcr_write_clr_set(wal, RT_CTRL, STACKID_CTRL, STACKID_CTRL_512); /* * This is also listed as Wa_22012654132 for certain DG2 @@ -2695,10 +2910,10 @@ add_render_compute_tuning_settings(struct drm_i915_private *i915, * back for verification on DG2 (due to Wa_14012342262), so * we need to explicitly skip the readback. */ - wa_add(wal, GEN10_CACHE_MODE_SS, 0, - _MASKED_BIT_ENABLE(ENABLE_PREFETCH_INTO_IC), - 0 /* write-only, so skip validation */, - true); + wa_mcr_add(wal, GEN10_CACHE_MODE_SS, 0, + _MASKED_BIT_ENABLE(ENABLE_PREFETCH_INTO_IC), + 0 /* write-only, so skip validation */, + true); } /* @@ -2707,8 +2922,8 @@ add_render_compute_tuning_settings(struct drm_i915_private *i915, * platforms. */ if (INTEL_INFO(i915)->tuning_thread_rr_after_dep) - wa_masked_field_set(wal, GEN9_ROW_CHICKEN4, THREAD_EX_ARB_MODE, - THREAD_EX_ARB_MODE_RR_AFTER_DEP); + wa_mcr_masked_field_set(wal, GEN9_ROW_CHICKEN4, THREAD_EX_ARB_MODE, + THREAD_EX_ARB_MODE_RR_AFTER_DEP); } /* @@ -2734,30 +2949,30 @@ general_render_compute_wa_init(struct intel_engine_cs *engine, struct i915_wa_li if (IS_XEHPSDV(i915)) { /* Wa_1409954639 */ - wa_masked_en(wal, - GEN8_ROW_CHICKEN, - SYSTOLIC_DOP_CLOCK_GATING_DIS); + wa_mcr_masked_en(wal, + GEN8_ROW_CHICKEN, + SYSTOLIC_DOP_CLOCK_GATING_DIS); /* Wa_1607196519 */ - wa_masked_en(wal, - GEN9_ROW_CHICKEN4, - GEN12_DISABLE_GRF_CLEAR); + wa_mcr_masked_en(wal, + GEN9_ROW_CHICKEN4, + GEN12_DISABLE_GRF_CLEAR); /* Wa_14010670810:xehpsdv */ - wa_write_or(wal, XEHP_L3NODEARBCFG, XEHP_LNESPARE); + wa_mcr_write_or(wal, XEHP_L3NODEARBCFG, XEHP_LNESPARE); /* Wa_14010449647:xehpsdv */ - wa_masked_en(wal, GEN7_HALF_SLICE_CHICKEN1, - GEN7_PSD_SINGLE_PORT_DISPATCH_ENABLE); + wa_mcr_masked_en(wal, GEN8_HALF_SLICE_CHICKEN1, + GEN7_PSD_SINGLE_PORT_DISPATCH_ENABLE); /* Wa_18011725039:xehpsdv */ if (IS_XEHPSDV_GRAPHICS_STEP(i915, STEP_A1, STEP_B0)) { - wa_masked_dis(wal, MLTICTXCTL, TDONRENDER); - wa_write_or(wal, L3SQCREG1_CCS0, FLUSHALLNONCOH); + wa_mcr_masked_dis(wal, MLTICTXCTL, TDONRENDER); + wa_mcr_write_or(wal, L3SQCREG1_CCS0, FLUSHALLNONCOH); } /* Wa_14012362059:xehpsdv */ - wa_write_or(wal, GEN12_MERT_MOD_CTRL, FORCE_MISS_FTLB); + wa_mcr_write_or(wal, XEHP_MERT_MOD_CTRL, FORCE_MISS_FTLB); /* Wa_14014368820:xehpsdv */ wa_write_or(wal, GEN12_GAMCNTRL_CTRL, INVALIDATION_BROADCAST_MODE_DIS | @@ -2766,26 +2981,37 @@ general_render_compute_wa_init(struct intel_engine_cs *engine, struct i915_wa_li if (IS_DG2(i915) || IS_PONTEVECCHIO(i915)) { /* Wa_14015227452:dg2,pvc */ - wa_masked_en(wal, GEN9_ROW_CHICKEN4, XEHP_DIS_BBL_SYSPIPE); + wa_mcr_masked_en(wal, GEN9_ROW_CHICKEN4, XEHP_DIS_BBL_SYSPIPE); /* Wa_22014226127:dg2,pvc */ - wa_write_or(wal, LSC_CHICKEN_BIT_0, DISABLE_D8_D16_COASLESCE); + wa_mcr_write_or(wal, LSC_CHICKEN_BIT_0, DISABLE_D8_D16_COASLESCE); /* Wa_16015675438:dg2,pvc */ wa_masked_en(wal, FF_SLICE_CS_CHICKEN2, GEN12_PERF_FIX_BALANCING_CFE_DISABLE); /* Wa_18018781329:dg2,pvc */ - wa_write_or(wal, RENDER_MOD_CTRL, FORCE_MISS_FTLB); - wa_write_or(wal, COMP_MOD_CTRL, FORCE_MISS_FTLB); - wa_write_or(wal, VDBX_MOD_CTRL, FORCE_MISS_FTLB); - wa_write_or(wal, VEBX_MOD_CTRL, FORCE_MISS_FTLB); + wa_mcr_write_or(wal, RENDER_MOD_CTRL, FORCE_MISS_FTLB); + wa_mcr_write_or(wal, COMP_MOD_CTRL, FORCE_MISS_FTLB); + wa_mcr_write_or(wal, VDBX_MOD_CTRL, FORCE_MISS_FTLB); + wa_mcr_write_or(wal, VEBX_MOD_CTRL, FORCE_MISS_FTLB); + } + + if (IS_DG2(i915)) { + /* + * Wa_16011620976:dg2_g11 + * Wa_22015475538:dg2 + */ + wa_mcr_write_or(wal, LSC_CHICKEN_BIT_0_UDW, DIS_CHAIN_2XSIMD8); + + /* Wa_18017747507:dg2 */ + wa_masked_en(wal, VFG_PREEMPTION_CHICKEN, POLYGON_TRIFAN_LINELOOP_DISABLE); } } static void engine_init_workarounds(struct intel_engine_cs *engine, struct i915_wa_list *wal) { - if (I915_SELFTEST_ONLY(GRAPHICS_VER(engine->i915) < 4)) + if (GRAPHICS_VER(engine->i915) < 4) return; engine_fake_wa_init(engine, wal); @@ -2810,17 +3036,14 @@ void intel_engine_init_workarounds(struct intel_engine_cs *engine) { struct i915_wa_list *wal = &engine->wa_list; - if (GRAPHICS_VER(engine->i915) < 4) - return; - - wa_init_start(wal, "engine", engine->name); + wa_init_start(wal, engine->gt, "engine", engine->name); engine_init_workarounds(engine, wal); wa_init_finish(wal); } void intel_engine_apply_workarounds(struct intel_engine_cs *engine) { - wa_list_apply(engine->gt, &engine->wa_list); + wa_list_apply(&engine->wa_list); } static const struct i915_range mcr_ranges_gen8[] = { @@ -2964,9 +3187,7 @@ retry: goto err_vma; } - err = i915_request_await_object(rq, vma->obj, true); - if (err == 0) - err = i915_vma_move_to_active(vma, rq, EXEC_OBJECT_WRITE); + err = i915_vma_move_to_active(vma, rq, EXEC_OBJECT_WRITE); if (err == 0) err = wa_list_srm(rq, wal, vma); @@ -2994,7 +3215,7 @@ retry: if (mcr_range(rq->engine->i915, i915_mmio_reg_offset(wa->reg))) continue; - if (!wa_verify(wa, results[i], wal->name, from)) + if (!wa_verify(wal->gt, wa, results[i], wal->name, from)) err = -ENXIO; } diff --git a/drivers/gpu/drm/i915/gt/intel_workarounds_types.h b/drivers/gpu/drm/i915/gt/intel_workarounds_types.h index 8a4b6de4e754..e14188120e66 100644 --- a/drivers/gpu/drm/i915/gt/intel_workarounds_types.h +++ b/drivers/gpu/drm/i915/gt/intel_workarounds_types.h @@ -10,15 +10,23 @@ #include "i915_reg_defs.h" +struct intel_gt; + struct i915_wa { - i915_reg_t reg; + union { + i915_reg_t reg; + i915_mcr_reg_t mcr_reg; + }; u32 clr; u32 set; u32 read; - bool masked_reg; + + u32 masked_reg:1; + u32 is_mcr:1; }; struct i915_wa_list { + struct intel_gt *gt; const char *name; const char *engine_name; struct i915_wa *list; diff --git a/drivers/gpu/drm/i915/gt/selftest_engine_cs.c b/drivers/gpu/drm/i915/gt/selftest_engine_cs.c index 1b75f478d1b8..881b64f3e7b9 100644 --- a/drivers/gpu/drm/i915/gt/selftest_engine_cs.c +++ b/drivers/gpu/drm/i915/gt/selftest_engine_cs.c @@ -39,6 +39,16 @@ static int perf_end(struct intel_gt *gt) return igt_flush_test(gt->i915); } +static i915_reg_t timestamp_reg(struct intel_engine_cs *engine) +{ + struct drm_i915_private *i915 = engine->i915; + + if (GRAPHICS_VER(i915) == 5 || IS_G4X(i915)) + return RING_TIMESTAMP_UDW(engine->mmio_base); + else + return RING_TIMESTAMP(engine->mmio_base); +} + static int write_timestamp(struct i915_request *rq, int slot) { struct intel_timeline *tl = @@ -55,7 +65,7 @@ static int write_timestamp(struct i915_request *rq, int slot) if (GRAPHICS_VER(rq->engine->i915) >= 8) cmd++; *cs++ = cmd; - *cs++ = i915_mmio_reg_offset(RING_TIMESTAMP(rq->engine->mmio_base)); + *cs++ = i915_mmio_reg_offset(timestamp_reg(rq->engine)); *cs++ = tl->hwsp_offset + slot * sizeof(u32); *cs++ = 0; @@ -125,7 +135,7 @@ static int perf_mi_bb_start(void *arg) enum intel_engine_id id; int err = 0; - if (GRAPHICS_VER(gt->i915) < 7) /* for per-engine CS_TIMESTAMP */ + if (GRAPHICS_VER(gt->i915) < 4) /* Any CS_TIMESTAMP? */ return 0; perf_begin(gt); @@ -135,6 +145,9 @@ static int perf_mi_bb_start(void *arg) u32 cycles[COUNT]; int i; + if (GRAPHICS_VER(engine->i915) < 7 && engine->id != RCS0) + continue; + intel_engine_pm_get(engine); batch = create_empty_batch(ce); @@ -249,7 +262,7 @@ static int perf_mi_noop(void *arg) enum intel_engine_id id; int err = 0; - if (GRAPHICS_VER(gt->i915) < 7) /* for per-engine CS_TIMESTAMP */ + if (GRAPHICS_VER(gt->i915) < 4) /* Any CS_TIMESTAMP? */ return 0; perf_begin(gt); @@ -259,6 +272,9 @@ static int perf_mi_noop(void *arg) u32 cycles[COUNT]; int i; + if (GRAPHICS_VER(engine->i915) < 7 && engine->id != RCS0) + continue; + intel_engine_pm_get(engine); base = create_empty_batch(ce); diff --git a/drivers/gpu/drm/i915/gt/selftest_engine_pm.c b/drivers/gpu/drm/i915/gt/selftest_engine_pm.c index 0dcb3ed44a73..87c94314cf67 100644 --- a/drivers/gpu/drm/i915/gt/selftest_engine_pm.c +++ b/drivers/gpu/drm/i915/gt/selftest_engine_pm.c @@ -317,7 +317,7 @@ static int live_engine_busy_stats(void *arg) ENGINE_TRACE(engine, "measuring busy time\n"); preempt_disable(); de = intel_engine_get_busy_time(engine, &t[0]); - mdelay(10); + mdelay(100); de = ktime_sub(intel_engine_get_busy_time(engine, &t[1]), de); preempt_enable(); dt = ktime_sub(t[1], t[0]); diff --git a/drivers/gpu/drm/i915/gt/selftest_execlists.c b/drivers/gpu/drm/i915/gt/selftest_execlists.c index 1e08b2473b99..ab2e9a6a2452 100644 --- a/drivers/gpu/drm/i915/gt/selftest_execlists.c +++ b/drivers/gpu/drm/i915/gt/selftest_execlists.c @@ -85,8 +85,6 @@ static int wait_for_reset(struct intel_engine_cs *engine, break; } while (time_before(jiffies, timeout)); - flush_scheduled_work(); - if (rq->fence.error != -EIO) { pr_err("%s: hanging request %llx:%lld not reset\n", engine->name, @@ -2766,9 +2764,7 @@ static int create_gang(struct intel_engine_cs *engine, i915_request_get(rq); i915_vma_lock(vma); - err = i915_request_await_object(rq, vma->obj, false); - if (!err) - err = i915_vma_move_to_active(vma, rq, 0); + err = i915_vma_move_to_active(vma, rq, 0); if (!err) err = rq->engine->emit_bb_start(rq, vma->node.start, @@ -3182,15 +3178,11 @@ create_gpr_client(struct intel_engine_cs *engine, } i915_vma_lock(vma); - err = i915_request_await_object(rq, vma->obj, false); - if (!err) - err = i915_vma_move_to_active(vma, rq, 0); + err = i915_vma_move_to_active(vma, rq, 0); i915_vma_unlock(vma); i915_vma_lock(batch); if (!err) - err = i915_request_await_object(rq, batch->obj, false); - if (!err) err = i915_vma_move_to_active(batch, rq, 0); if (!err) err = rq->engine->emit_bb_start(rq, @@ -3475,12 +3467,14 @@ static int random_priority(struct rnd_state *rnd) struct preempt_smoke { struct intel_gt *gt; + struct kthread_work work; struct i915_gem_context **contexts; struct intel_engine_cs *engine; struct drm_i915_gem_object *batch; unsigned int ncontext; struct rnd_state prng; unsigned long count; + int result; }; static struct i915_gem_context *smoke_context(struct preempt_smoke *smoke) @@ -3521,9 +3515,7 @@ static int smoke_submit(struct preempt_smoke *smoke, if (vma) { i915_vma_lock(vma); - err = i915_request_await_object(rq, vma->obj, false); - if (!err) - err = i915_vma_move_to_active(vma, rq, 0); + err = i915_vma_move_to_active(vma, rq, 0); if (!err) err = rq->engine->emit_bb_start(rq, vma->node.start, @@ -3540,34 +3532,31 @@ unpin: return err; } -static int smoke_crescendo_thread(void *arg) +static void smoke_crescendo_work(struct kthread_work *work) { - struct preempt_smoke *smoke = arg; + struct preempt_smoke *smoke = container_of(work, typeof(*smoke), work); IGT_TIMEOUT(end_time); unsigned long count; count = 0; do { struct i915_gem_context *ctx = smoke_context(smoke); - int err; - err = smoke_submit(smoke, - ctx, count % I915_PRIORITY_MAX, - smoke->batch); - if (err) - return err; + smoke->result = smoke_submit(smoke, ctx, + count % I915_PRIORITY_MAX, + smoke->batch); count++; - } while (count < smoke->ncontext && !__igt_timeout(end_time, NULL)); + } while (!smoke->result && count < smoke->ncontext && + !__igt_timeout(end_time, NULL)); smoke->count = count; - return 0; } static int smoke_crescendo(struct preempt_smoke *smoke, unsigned int flags) #define BATCH BIT(0) { - struct task_struct *tsk[I915_NUM_ENGINES] = {}; + struct kthread_worker *worker[I915_NUM_ENGINES] = {}; struct preempt_smoke *arg; struct intel_engine_cs *engine; enum intel_engine_id id; @@ -3578,6 +3567,8 @@ static int smoke_crescendo(struct preempt_smoke *smoke, unsigned int flags) if (!arg) return -ENOMEM; + memset(arg, 0, I915_NUM_ENGINES * sizeof(*arg)); + for_each_engine(engine, smoke->gt, id) { arg[id] = *smoke; arg[id].engine = engine; @@ -3585,31 +3576,28 @@ static int smoke_crescendo(struct preempt_smoke *smoke, unsigned int flags) arg[id].batch = NULL; arg[id].count = 0; - tsk[id] = kthread_run(smoke_crescendo_thread, arg, - "igt/smoke:%d", id); - if (IS_ERR(tsk[id])) { - err = PTR_ERR(tsk[id]); + worker[id] = kthread_create_worker(0, "igt/smoke:%d", id); + if (IS_ERR(worker[id])) { + err = PTR_ERR(worker[id]); break; } - get_task_struct(tsk[id]); - } - yield(); /* start all threads before we kthread_stop() */ + kthread_init_work(&arg[id].work, smoke_crescendo_work); + kthread_queue_work(worker[id], &arg[id].work); + } count = 0; for_each_engine(engine, smoke->gt, id) { - int status; - - if (IS_ERR_OR_NULL(tsk[id])) + if (IS_ERR_OR_NULL(worker[id])) continue; - status = kthread_stop(tsk[id]); - if (status && !err) - err = status; + kthread_flush_work(&arg[id].work); + if (arg[id].result && !err) + err = arg[id].result; count += arg[id].count; - put_task_struct(tsk[id]); + kthread_destroy_worker(worker[id]); } pr_info("Submitted %lu crescendo:%x requests across %d engines and %d contexts\n", diff --git a/drivers/gpu/drm/i915/gt/selftest_gt_pm.c b/drivers/gpu/drm/i915/gt/selftest_gt_pm.c index be94f863bdef..b46425aeb2f0 100644 --- a/drivers/gpu/drm/i915/gt/selftest_gt_pm.c +++ b/drivers/gpu/drm/i915/gt/selftest_gt_pm.c @@ -36,6 +36,19 @@ static int cmp_u32(const void *A, const void *B) return 0; } +static u32 read_timestamp(struct intel_engine_cs *engine) +{ + struct drm_i915_private *i915 = engine->i915; + + /* On i965 the first read tends to give a stale value */ + ENGINE_READ_FW(engine, RING_TIMESTAMP); + + if (GRAPHICS_VER(i915) == 5 || IS_G4X(i915)) + return ENGINE_READ_FW(engine, RING_TIMESTAMP_UDW); + else + return ENGINE_READ_FW(engine, RING_TIMESTAMP); +} + static void measure_clocks(struct intel_engine_cs *engine, u32 *out_cycles, ktime_t *out_dt) { @@ -45,13 +58,13 @@ static void measure_clocks(struct intel_engine_cs *engine, for (i = 0; i < 5; i++) { local_irq_disable(); - cycles[i] = -ENGINE_READ_FW(engine, RING_TIMESTAMP); + cycles[i] = -read_timestamp(engine); dt[i] = ktime_get(); udelay(1000); dt[i] = ktime_sub(ktime_get(), dt[i]); - cycles[i] += ENGINE_READ_FW(engine, RING_TIMESTAMP); + cycles[i] += read_timestamp(engine); local_irq_enable(); } @@ -78,25 +91,6 @@ static int live_gt_clocks(void *arg) if (GRAPHICS_VER(gt->i915) < 4) /* Any CS_TIMESTAMP? */ return 0; - if (GRAPHICS_VER(gt->i915) == 5) - /* - * XXX CS_TIMESTAMP low dword is dysfunctional? - * - * Ville's experiments indicate the high dword still works, - * but at a correspondingly reduced frequency. - */ - return 0; - - if (GRAPHICS_VER(gt->i915) == 4) - /* - * XXX CS_TIMESTAMP appears gibberish - * - * Ville's experiments indicate that it mostly appears 'stuck' - * in that we see the register report the same cycle count - * for a couple of reads. - */ - return 0; - intel_gt_pm_get(gt); intel_uncore_forcewake_get(gt->uncore, FORCEWAKE_ALL); diff --git a/drivers/gpu/drm/i915/gt/selftest_hangcheck.c b/drivers/gpu/drm/i915/gt/selftest_hangcheck.c index 7f3bb1d34dfb..bc05ef48c194 100644 --- a/drivers/gpu/drm/i915/gt/selftest_hangcheck.c +++ b/drivers/gpu/drm/i915/gt/selftest_hangcheck.c @@ -99,22 +99,6 @@ static u64 hws_address(const struct i915_vma *hws, return hws->node.start + offset_in_page(sizeof(u32)*rq->fence.context); } -static int move_to_active(struct i915_vma *vma, - struct i915_request *rq, - unsigned int flags) -{ - int err; - - i915_vma_lock(vma); - err = i915_request_await_object(rq, vma->obj, - flags & EXEC_OBJECT_WRITE); - if (err == 0) - err = i915_vma_move_to_active(vma, rq, flags); - i915_vma_unlock(vma); - - return err; -} - static struct i915_request * hang_create_request(struct hang *h, struct intel_engine_cs *engine) { @@ -175,11 +159,11 @@ hang_create_request(struct hang *h, struct intel_engine_cs *engine) goto unpin_hws; } - err = move_to_active(vma, rq, 0); + err = igt_vma_move_to_active_unlocked(vma, rq, 0); if (err) goto cancel_rq; - err = move_to_active(hws, rq, 0); + err = igt_vma_move_to_active_unlocked(hws, rq, 0); if (err) goto cancel_rq; @@ -866,10 +850,13 @@ static int igt_reset_active_engine(void *arg) } struct active_engine { - struct task_struct *task; + struct kthread_worker *worker; + struct kthread_work work; struct intel_engine_cs *engine; unsigned long resets; unsigned int flags; + bool stop; + int result; }; #define TEST_ACTIVE BIT(0) @@ -900,10 +887,10 @@ static int active_request_put(struct i915_request *rq) return err; } -static int active_engine(void *data) +static void active_engine(struct kthread_work *work) { I915_RND_STATE(prng); - struct active_engine *arg = data; + struct active_engine *arg = container_of(work, typeof(*arg), work); struct intel_engine_cs *engine = arg->engine; struct i915_request *rq[8] = {}; struct intel_context *ce[ARRAY_SIZE(rq)]; @@ -913,16 +900,17 @@ static int active_engine(void *data) for (count = 0; count < ARRAY_SIZE(ce); count++) { ce[count] = intel_context_create(engine); if (IS_ERR(ce[count])) { - err = PTR_ERR(ce[count]); - pr_err("[%s] Create context #%ld failed: %d!\n", engine->name, count, err); + arg->result = PTR_ERR(ce[count]); + pr_err("[%s] Create context #%ld failed: %d!\n", + engine->name, count, arg->result); while (--count) intel_context_put(ce[count]); - return err; + return; } } count = 0; - while (!kthread_should_stop()) { + while (!READ_ONCE(arg->stop)) { unsigned int idx = count++ & (ARRAY_SIZE(rq) - 1); struct i915_request *old = rq[idx]; struct i915_request *new; @@ -967,7 +955,7 @@ static int active_engine(void *data) intel_context_put(ce[count]); } - return err; + arg->result = err; } static int __igt_reset_engines(struct intel_gt *gt, @@ -1022,7 +1010,7 @@ static int __igt_reset_engines(struct intel_gt *gt, memset(threads, 0, sizeof(*threads) * I915_NUM_ENGINES); for_each_engine(other, gt, tmp) { - struct task_struct *tsk; + struct kthread_worker *worker; threads[tmp].resets = i915_reset_engine_count(global, other); @@ -1036,19 +1024,21 @@ static int __igt_reset_engines(struct intel_gt *gt, threads[tmp].engine = other; threads[tmp].flags = flags; - tsk = kthread_run(active_engine, &threads[tmp], - "igt/%s", other->name); - if (IS_ERR(tsk)) { - err = PTR_ERR(tsk); - pr_err("[%s] Thread spawn failed: %d!\n", engine->name, err); + worker = kthread_create_worker(0, "igt/%s", + other->name); + if (IS_ERR(worker)) { + err = PTR_ERR(worker); + pr_err("[%s] Worker create failed: %d!\n", + engine->name, err); goto unwind; } - threads[tmp].task = tsk; - get_task_struct(tsk); - } + threads[tmp].worker = worker; - yield(); /* start all threads before we begin */ + kthread_init_work(&threads[tmp].work, active_engine); + kthread_queue_work(threads[tmp].worker, + &threads[tmp].work); + } st_engine_heartbeat_disable_no_pm(engine); GEM_BUG_ON(test_and_set_bit(I915_RESET_ENGINE + id, @@ -1197,17 +1187,20 @@ unwind: for_each_engine(other, gt, tmp) { int ret; - if (!threads[tmp].task) + if (!threads[tmp].worker) continue; - ret = kthread_stop(threads[tmp].task); + WRITE_ONCE(threads[tmp].stop, true); + kthread_flush_work(&threads[tmp].work); + ret = READ_ONCE(threads[tmp].result); if (ret) { pr_err("kthread for other engine %s failed, err=%d\n", other->name, ret); if (!err) err = ret; } - put_task_struct(threads[tmp].task); + + kthread_destroy_worker(threads[tmp].worker); /* GuC based resets are not logged per engine */ if (!using_guc) { @@ -1510,18 +1503,9 @@ static int __igt_reset_evict_vma(struct intel_gt *gt, } } - i915_vma_lock(arg.vma); - err = i915_request_await_object(rq, arg.vma->obj, - flags & EXEC_OBJECT_WRITE); - if (err == 0) { - err = i915_vma_move_to_active(arg.vma, rq, flags); - if (err) - pr_err("[%s] Move to active failed: %d!\n", engine->name, err); - } else { - pr_err("[%s] Request await failed: %d!\n", engine->name, err); - } - - i915_vma_unlock(arg.vma); + err = igt_vma_move_to_active_unlocked(arg.vma, rq, flags); + if (err) + pr_err("[%s] Move to active failed: %d!\n", engine->name, err); if (flags & EXEC_OBJECT_NEEDS_FENCE) i915_vma_unpin_fence(arg.vma); diff --git a/drivers/gpu/drm/i915/gt/selftest_lrc.c b/drivers/gpu/drm/i915/gt/selftest_lrc.c index 82d3f8058995..7c56ffd2c659 100644 --- a/drivers/gpu/drm/i915/gt/selftest_lrc.c +++ b/drivers/gpu/drm/i915/gt/selftest_lrc.c @@ -452,9 +452,7 @@ retry: *cs++ = i915_ggtt_offset(scratch) + RING_TAIL_IDX * sizeof(u32); *cs++ = 0; - err = i915_request_await_object(rq, scratch->obj, true); - if (!err) - err = i915_vma_move_to_active(scratch, rq, EXEC_OBJECT_WRITE); + err = i915_vma_move_to_active(scratch, rq, EXEC_OBJECT_WRITE); i915_request_get(rq); i915_request_add(rq); @@ -602,9 +600,7 @@ __gpr_read(struct intel_context *ce, struct i915_vma *scratch, u32 *slot) } i915_vma_lock(scratch); - err = i915_request_await_object(rq, scratch->obj, true); - if (!err) - err = i915_vma_move_to_active(scratch, rq, EXEC_OBJECT_WRITE); + err = i915_vma_move_to_active(scratch, rq, EXEC_OBJECT_WRITE); i915_vma_unlock(scratch); i915_request_get(rq); @@ -1053,21 +1049,6 @@ store_context(struct intel_context *ce, struct i915_vma *scratch) return batch; } -static int move_to_active(struct i915_request *rq, - struct i915_vma *vma, - unsigned int flags) -{ - int err; - - i915_vma_lock(vma); - err = i915_request_await_object(rq, vma->obj, flags); - if (!err) - err = i915_vma_move_to_active(vma, rq, flags); - i915_vma_unlock(vma); - - return err; -} - static struct i915_request * record_registers(struct intel_context *ce, struct i915_vma *before, @@ -1093,19 +1074,19 @@ record_registers(struct intel_context *ce, if (IS_ERR(rq)) goto err_after; - err = move_to_active(rq, before, EXEC_OBJECT_WRITE); + err = igt_vma_move_to_active_unlocked(before, rq, EXEC_OBJECT_WRITE); if (err) goto err_rq; - err = move_to_active(rq, b_before, 0); + err = igt_vma_move_to_active_unlocked(b_before, rq, 0); if (err) goto err_rq; - err = move_to_active(rq, after, EXEC_OBJECT_WRITE); + err = igt_vma_move_to_active_unlocked(after, rq, EXEC_OBJECT_WRITE); if (err) goto err_rq; - err = move_to_active(rq, b_after, 0); + err = igt_vma_move_to_active_unlocked(b_after, rq, 0); if (err) goto err_rq; @@ -1243,7 +1224,7 @@ static int poison_registers(struct intel_context *ce, u32 poison, u32 *sema) goto err_batch; } - err = move_to_active(rq, batch, 0); + err = igt_vma_move_to_active_unlocked(batch, rq, 0); if (err) goto err_rq; diff --git a/drivers/gpu/drm/i915/gt/selftest_migrate.c b/drivers/gpu/drm/i915/gt/selftest_migrate.c index 2b0c87999949..0dc5309c90a4 100644 --- a/drivers/gpu/drm/i915/gt/selftest_migrate.c +++ b/drivers/gpu/drm/i915/gt/selftest_migrate.c @@ -6,6 +6,7 @@ #include <linux/sort.h> #include "gem/i915_gem_internal.h" +#include "gem/i915_gem_lmem.h" #include "selftests/i915_random.h" diff --git a/drivers/gpu/drm/i915/gt/selftest_mocs.c b/drivers/gpu/drm/i915/gt/selftest_mocs.c index c1d861333c44..f27cc28608d4 100644 --- a/drivers/gpu/drm/i915/gt/selftest_mocs.c +++ b/drivers/gpu/drm/i915/gt/selftest_mocs.c @@ -7,6 +7,7 @@ #include "gt/intel_gpu_commands.h" #include "i915_selftest.h" +#include "gem/selftests/igt_gem_utils.h" #include "gem/selftests/mock_context.h" #include "selftests/igt_reset.h" #include "selftests/igt_spinner.h" @@ -228,9 +229,7 @@ static int check_mocs_engine(struct live_mocs *arg, return PTR_ERR(rq); i915_vma_lock(vma); - err = i915_request_await_object(rq, vma->obj, true); - if (!err) - err = i915_vma_move_to_active(vma, rq, EXEC_OBJECT_WRITE); + err = i915_vma_move_to_active(vma, rq, EXEC_OBJECT_WRITE); i915_vma_unlock(vma); /* Read the mocs tables back using SRM */ diff --git a/drivers/gpu/drm/i915/gt/selftest_rc6.c b/drivers/gpu/drm/i915/gt/selftest_rc6.c index 8c70b7e12074..2ceeadecc639 100644 --- a/drivers/gpu/drm/i915/gt/selftest_rc6.c +++ b/drivers/gpu/drm/i915/gt/selftest_rc6.c @@ -19,11 +19,11 @@ static u64 rc6_residency(struct intel_rc6 *rc6) /* XXX VLV_GT_MEDIA_RC6? */ - result = intel_rc6_residency_ns(rc6, GEN6_GT_GFX_RC6); + result = intel_rc6_residency_ns(rc6, INTEL_RC6_RES_RC6); if (HAS_RC6p(rc6_to_i915(rc6))) - result += intel_rc6_residency_ns(rc6, GEN6_GT_GFX_RC6p); + result += intel_rc6_residency_ns(rc6, INTEL_RC6_RES_RC6p); if (HAS_RC6pp(rc6_to_i915(rc6))) - result += intel_rc6_residency_ns(rc6, GEN6_GT_GFX_RC6pp); + result += intel_rc6_residency_ns(rc6, INTEL_RC6_RES_RC6pp); return result; } diff --git a/drivers/gpu/drm/i915/gt/selftest_rps.c b/drivers/gpu/drm/i915/gt/selftest_rps.c index cfb4708dd62e..39f1b7564170 100644 --- a/drivers/gpu/drm/i915/gt/selftest_rps.c +++ b/drivers/gpu/drm/i915/gt/selftest_rps.c @@ -652,9 +652,7 @@ int live_rps_frequency_cs(void *arg) goto err_vma; } - err = i915_request_await_object(rq, vma->obj, false); - if (!err) - err = i915_vma_move_to_active(vma, rq, 0); + err = i915_vma_move_to_active(vma, rq, 0); if (!err) err = rq->engine->emit_bb_start(rq, vma->node.start, @@ -793,9 +791,7 @@ int live_rps_frequency_srm(void *arg) goto err_vma; } - err = i915_request_await_object(rq, vma->obj, false); - if (!err) - err = i915_vma_move_to_active(vma, rq, 0); + err = i915_vma_move_to_active(vma, rq, 0); if (!err) err = rq->engine->emit_bb_start(rq, vma->node.start, @@ -1107,21 +1103,27 @@ static u64 __measure_power(int duration_ms) return div64_u64(1000 * 1000 * dE, dt); } -static u64 measure_power_at(struct intel_rps *rps, int *freq) +static u64 measure_power(struct intel_rps *rps, int *freq) { u64 x[5]; int i; - *freq = rps_set_check(rps, *freq); for (i = 0; i < 5; i++) x[i] = __measure_power(5); - *freq = (*freq + read_cagf(rps)) / 2; + + *freq = (*freq + intel_rps_read_actual_frequency(rps)) / 2; /* A simple triangle filter for better result stability */ sort(x, 5, sizeof(*x), cmp_u64, NULL); return div_u64(x[1] + 2 * x[2] + x[3], 4); } +static u64 measure_power_at(struct intel_rps *rps, int *freq) +{ + *freq = rps_set_check(rps, *freq); + return measure_power(rps, freq); +} + int live_rps_power(void *arg) { struct intel_gt *gt = arg; diff --git a/drivers/gpu/drm/i915/gt/selftest_slpc.c b/drivers/gpu/drm/i915/gt/selftest_slpc.c index f8a1d27df272..bd44ce73a504 100644 --- a/drivers/gpu/drm/i915/gt/selftest_slpc.c +++ b/drivers/gpu/drm/i915/gt/selftest_slpc.c @@ -11,7 +11,16 @@ enum test_type { VARY_MIN, VARY_MAX, - MAX_GRANTED + MAX_GRANTED, + SLPC_POWER, + TILE_INTERACTION, +}; + +struct slpc_thread { + struct kthread_worker *worker; + struct kthread_work work; + struct intel_gt *gt; + int result; }; static int slpc_set_min_freq(struct intel_guc_slpc *slpc, u32 freq) @@ -41,6 +50,39 @@ static int slpc_set_max_freq(struct intel_guc_slpc *slpc, u32 freq) return ret; } +static int slpc_set_freq(struct intel_gt *gt, u32 freq) +{ + int err; + struct intel_guc_slpc *slpc = >->uc.guc.slpc; + + err = slpc_set_max_freq(slpc, freq); + if (err) { + pr_err("Unable to update max freq"); + return err; + } + + err = slpc_set_min_freq(slpc, freq); + if (err) { + pr_err("Unable to update min freq"); + return err; + } + + return err; +} + +static u64 measure_power_at_freq(struct intel_gt *gt, int *freq, u64 *power) +{ + int err = 0; + + err = slpc_set_freq(gt, *freq); + if (err) + return err; + *freq = intel_rps_read_actual_frequency(>->rps); + *power = measure_power(>->rps, freq); + + return err; +} + static int vary_max_freq(struct intel_guc_slpc *slpc, struct intel_rps *rps, u32 *max_act_freq) { @@ -113,6 +155,58 @@ static int vary_min_freq(struct intel_guc_slpc *slpc, struct intel_rps *rps, return err; } +static int slpc_power(struct intel_gt *gt, struct intel_engine_cs *engine) +{ + struct intel_guc_slpc *slpc = >->uc.guc.slpc; + struct { + u64 power; + int freq; + } min, max; + int err = 0; + + /* + * Our fundamental assumption is that running at lower frequency + * actually saves power. Let's see if our RAPL measurement supports + * that theory. + */ + if (!librapl_supported(gt->i915)) + return 0; + + min.freq = slpc->min_freq; + err = measure_power_at_freq(gt, &min.freq, &min.power); + + if (err) + return err; + + max.freq = slpc->rp0_freq; + err = measure_power_at_freq(gt, &max.freq, &max.power); + + if (err) + return err; + + pr_info("%s: min:%llumW @ %uMHz, max:%llumW @ %uMHz\n", + engine->name, + min.power, min.freq, + max.power, max.freq); + + if (10 * min.freq >= 9 * max.freq) { + pr_notice("Could not control frequency, ran at [%uMHz, %uMhz]\n", + min.freq, max.freq); + } + + if (11 * min.power > 10 * max.power) { + pr_err("%s: did not conserve power when setting lower frequency!\n", + engine->name); + err = -EINVAL; + } + + /* Restore min/max frequencies */ + slpc_set_max_freq(slpc, slpc->rp0_freq); + slpc_set_min_freq(slpc, slpc->min_freq); + + return err; +} + static int max_granted_freq(struct intel_guc_slpc *slpc, struct intel_rps *rps, u32 *max_act_freq) { struct intel_gt *gt = rps_to_gt(rps); @@ -126,7 +220,8 @@ static int max_granted_freq(struct intel_guc_slpc *slpc, struct intel_rps *rps, *max_act_freq = intel_rps_read_actual_frequency(rps); if (*max_act_freq != slpc->rp0_freq) { /* Check if there was some throttling by pcode */ - perf_limit_reasons = intel_uncore_read(gt->uncore, GT0_PERF_LIMIT_REASONS); + perf_limit_reasons = intel_uncore_read(gt->uncore, + intel_gt_perf_limit_reasons_reg(gt)); /* If not, this is an error */ if (!(perf_limit_reasons & GT0_PERF_LIMIT_REASONS_MASK)) { @@ -153,6 +248,11 @@ static int run_test(struct intel_gt *gt, int test_type) if (!intel_uc_uses_guc_slpc(>->uc)) return 0; + if (slpc->min_freq == slpc->rp0_freq) { + pr_err("Min/Max are fused to the same value\n"); + return -EINVAL; + } + if (igt_spinner_init(&spin, gt)) return -ENOMEM; @@ -167,17 +267,14 @@ static int run_test(struct intel_gt *gt, int test_type) } /* - * FIXME: With efficient frequency enabled, GuC can request - * frequencies higher than the SLPC max. While this is fixed - * in GuC, we level set these tests with RPn as min. + * Set min frequency to RPn so that we can test the whole + * range of RPn-RP0. This also turns off efficient freq + * usage and makes results more predictable. */ err = slpc_set_min_freq(slpc, slpc->min_freq); - if (err) + if (err) { + pr_err("Unable to update min freq!"); return err; - - if (slpc->min_freq == slpc->rp0_freq) { - pr_err("Min/Max are fused to the same value\n"); - return -EINVAL; } intel_gt_pm_wait_for_idle(gt); @@ -222,9 +319,10 @@ static int run_test(struct intel_gt *gt, int test_type) break; case MAX_GRANTED: + case TILE_INTERACTION: /* Media engines have a different RP0 */ - if (engine->class == VIDEO_DECODE_CLASS || - engine->class == VIDEO_ENHANCEMENT_CLASS) { + if (gt->type != GT_MEDIA && (engine->class == VIDEO_DECODE_CLASS || + engine->class == VIDEO_ENHANCEMENT_CLASS)) { igt_spinner_end(&spin); st_engine_heartbeat_enable(engine); err = 0; @@ -233,17 +331,24 @@ static int run_test(struct intel_gt *gt, int test_type) err = max_granted_freq(slpc, rps, &max_act_freq); break; - } - pr_info("Max actual frequency for %s was %d\n", - engine->name, max_act_freq); + case SLPC_POWER: + err = slpc_power(gt, engine); + break; + } - /* Actual frequency should rise above min */ - if (max_act_freq <= slpc_min_freq) { - pr_err("Actual freq did not rise above min\n"); - pr_err("Perf Limit Reasons: 0x%x\n", - intel_uncore_read(gt->uncore, GT0_PERF_LIMIT_REASONS)); - err = -EINVAL; + if (test_type != SLPC_POWER) { + pr_info("Max actual frequency for %s was %d\n", + engine->name, max_act_freq); + + /* Actual frequency should rise above min */ + if (max_act_freq <= slpc->min_freq) { + pr_err("Actual freq did not rise above min\n"); + pr_err("Perf Limit Reasons: 0x%x\n", + intel_uncore_read(gt->uncore, + intel_gt_perf_limit_reasons_reg(gt))); + err = -EINVAL; + } } igt_spinner_end(&spin); @@ -270,26 +375,116 @@ static int run_test(struct intel_gt *gt, int test_type) static int live_slpc_vary_min(void *arg) { struct drm_i915_private *i915 = arg; - struct intel_gt *gt = to_gt(i915); + struct intel_gt *gt; + unsigned int i; + int ret; + + for_each_gt(gt, i915, i) { + ret = run_test(gt, VARY_MIN); + if (ret) + return ret; + } - return run_test(gt, VARY_MIN); + return ret; } static int live_slpc_vary_max(void *arg) { struct drm_i915_private *i915 = arg; - struct intel_gt *gt = to_gt(i915); + struct intel_gt *gt; + unsigned int i; + int ret; + + for_each_gt(gt, i915, i) { + ret = run_test(gt, VARY_MAX); + if (ret) + return ret; + } - return run_test(gt, VARY_MAX); + return ret; } /* check if pcode can grant RP0 */ static int live_slpc_max_granted(void *arg) { struct drm_i915_private *i915 = arg; - struct intel_gt *gt = to_gt(i915); + struct intel_gt *gt; + unsigned int i; + int ret; + + for_each_gt(gt, i915, i) { + ret = run_test(gt, MAX_GRANTED); + if (ret) + return ret; + } + + return ret; +} + +static int live_slpc_power(void *arg) +{ + struct drm_i915_private *i915 = arg; + struct intel_gt *gt; + unsigned int i; + int ret; + + for_each_gt(gt, i915, i) { + ret = run_test(gt, SLPC_POWER); + if (ret) + return ret; + } + + return ret; +} + +static void slpc_spinner_thread(struct kthread_work *work) +{ + struct slpc_thread *thread = container_of(work, typeof(*thread), work); + + thread->result = run_test(thread->gt, TILE_INTERACTION); +} + +static int live_slpc_tile_interaction(void *arg) +{ + struct drm_i915_private *i915 = arg; + struct intel_gt *gt; + struct slpc_thread *threads; + int i = 0, ret = 0; + + threads = kcalloc(I915_MAX_GT, sizeof(*threads), GFP_KERNEL); + if (!threads) + return -ENOMEM; + + for_each_gt(gt, i915, i) { + threads[i].worker = kthread_create_worker(0, "igt/slpc_parallel:%d", gt->info.id); + + if (IS_ERR(threads[i].worker)) { + ret = PTR_ERR(threads[i].worker); + break; + } + + threads[i].gt = gt; + kthread_init_work(&threads[i].work, slpc_spinner_thread); + kthread_queue_work(threads[i].worker, &threads[i].work); + } + + for_each_gt(gt, i915, i) { + int status; + + if (IS_ERR_OR_NULL(threads[i].worker)) + continue; - return run_test(gt, MAX_GRANTED); + kthread_flush_work(&threads[i].work); + status = READ_ONCE(threads[i].result); + if (status && !ret) { + pr_err("%s GT %d failed ", __func__, gt->info.id); + ret = status; + } + kthread_destroy_worker(threads[i].worker); + } + + kfree(threads); + return ret; } int intel_slpc_live_selftests(struct drm_i915_private *i915) @@ -298,10 +493,17 @@ int intel_slpc_live_selftests(struct drm_i915_private *i915) SUBTEST(live_slpc_vary_max), SUBTEST(live_slpc_vary_min), SUBTEST(live_slpc_max_granted), + SUBTEST(live_slpc_power), + SUBTEST(live_slpc_tile_interaction), }; - if (intel_gt_is_wedged(to_gt(i915))) - return 0; + struct intel_gt *gt; + unsigned int i; + + for_each_gt(gt, i915, i) { + if (intel_gt_is_wedged(gt)) + return 0; + } return i915_live_subtests(tests, i915); } diff --git a/drivers/gpu/drm/i915/gt/selftest_workarounds.c b/drivers/gpu/drm/i915/gt/selftest_workarounds.c index 67a9aab801dd..96e3861706d6 100644 --- a/drivers/gpu/drm/i915/gt/selftest_workarounds.c +++ b/drivers/gpu/drm/i915/gt/selftest_workarounds.c @@ -66,14 +66,14 @@ reference_lists_init(struct intel_gt *gt, struct wa_lists *lists) memset(lists, 0, sizeof(*lists)); - wa_init_start(&lists->gt_wa_list, "GT_REF", "global"); + wa_init_start(&lists->gt_wa_list, gt, "GT_REF", "global"); gt_init_workarounds(gt, &lists->gt_wa_list); wa_init_finish(&lists->gt_wa_list); for_each_engine(engine, gt, id) { struct i915_wa_list *wal = &lists->engine[id].wa_list; - wa_init_start(wal, "REF", engine->name); + wa_init_start(wal, gt, "REF", engine->name); engine_init_workarounds(engine, wal); wa_init_finish(wal); @@ -139,9 +139,7 @@ read_nonprivs(struct intel_context *ce) } i915_vma_lock(vma); - err = i915_request_await_object(rq, vma->obj, true); - if (err == 0) - err = i915_vma_move_to_active(vma, rq, EXEC_OBJECT_WRITE); + err = i915_vma_move_to_active(vma, rq, EXEC_OBJECT_WRITE); i915_vma_unlock(vma); if (err) goto err_req; @@ -632,16 +630,12 @@ retry: goto err_request; } - err = i915_request_await_object(rq, batch->obj, false); - if (err == 0) - err = i915_vma_move_to_active(batch, rq, 0); + err = i915_vma_move_to_active(batch, rq, 0); if (err) goto err_request; - err = i915_request_await_object(rq, scratch->obj, true); - if (err == 0) - err = i915_vma_move_to_active(scratch, rq, - EXEC_OBJECT_WRITE); + err = i915_vma_move_to_active(scratch, rq, + EXEC_OBJECT_WRITE); if (err) goto err_request; @@ -860,9 +854,7 @@ static int read_whitelisted_registers(struct intel_context *ce, return PTR_ERR(rq); i915_vma_lock(results); - err = i915_request_await_object(rq, results->obj, true); - if (err == 0) - err = i915_vma_move_to_active(results, rq, EXEC_OBJECT_WRITE); + err = i915_vma_move_to_active(results, rq, EXEC_OBJECT_WRITE); i915_vma_unlock(results); if (err) goto err_req; @@ -944,9 +936,7 @@ static int scrub_whitelisted_registers(struct intel_context *ce) } i915_vma_lock(batch); - err = i915_request_await_object(rq, batch->obj, false); - if (err == 0) - err = i915_vma_move_to_active(batch, rq, 0); + err = i915_vma_move_to_active(batch, rq, 0); i915_vma_unlock(batch); if (err) goto err_request; @@ -991,7 +981,7 @@ static bool pardon_reg(struct drm_i915_private *i915, i915_reg_t reg) /* Alas, we must pardon some whitelists. Mistakes already made */ static const struct regmask pardon[] = { { GEN9_CTX_PREEMPT_REG, 9 }, - { GEN8_L3SQCREG4, 9 }, + { _MMIO(0xb118), 9 }, /* GEN8_L3SQCREG4 */ }; return find_reg(i915, reg, pardon, ARRAY_SIZE(pardon)); diff --git a/drivers/gpu/drm/i915/gt/sysfs_engines.c b/drivers/gpu/drm/i915/gt/sysfs_engines.c index 967031056202..f2d9858d827c 100644 --- a/drivers/gpu/drm/i915/gt/sysfs_engines.c +++ b/drivers/gpu/drm/i915/gt/sysfs_engines.c @@ -144,7 +144,7 @@ max_spin_store(struct kobject *kobj, struct kobj_attribute *attr, const char *buf, size_t count) { struct intel_engine_cs *engine = kobj_to_engine(kobj); - unsigned long long duration; + unsigned long long duration, clamped; int err; /* @@ -168,7 +168,8 @@ max_spin_store(struct kobject *kobj, struct kobj_attribute *attr, if (err) return err; - if (duration > jiffies_to_nsecs(2)) + clamped = intel_clamp_max_busywait_duration_ns(engine, duration); + if (duration != clamped) return -EINVAL; WRITE_ONCE(engine->props.max_busywait_duration_ns, duration); @@ -203,7 +204,7 @@ timeslice_store(struct kobject *kobj, struct kobj_attribute *attr, const char *buf, size_t count) { struct intel_engine_cs *engine = kobj_to_engine(kobj); - unsigned long long duration; + unsigned long long duration, clamped; int err; /* @@ -218,7 +219,8 @@ timeslice_store(struct kobject *kobj, struct kobj_attribute *attr, if (err) return err; - if (duration > jiffies_to_msecs(MAX_SCHEDULE_TIMEOUT)) + clamped = intel_clamp_timeslice_duration_ms(engine, duration); + if (duration != clamped) return -EINVAL; WRITE_ONCE(engine->props.timeslice_duration_ms, duration); @@ -256,7 +258,7 @@ stop_store(struct kobject *kobj, struct kobj_attribute *attr, const char *buf, size_t count) { struct intel_engine_cs *engine = kobj_to_engine(kobj); - unsigned long long duration; + unsigned long long duration, clamped; int err; /* @@ -272,7 +274,8 @@ stop_store(struct kobject *kobj, struct kobj_attribute *attr, if (err) return err; - if (duration > jiffies_to_msecs(MAX_SCHEDULE_TIMEOUT)) + clamped = intel_clamp_stop_timeout_ms(engine, duration); + if (duration != clamped) return -EINVAL; WRITE_ONCE(engine->props.stop_timeout_ms, duration); @@ -306,7 +309,7 @@ preempt_timeout_store(struct kobject *kobj, struct kobj_attribute *attr, const char *buf, size_t count) { struct intel_engine_cs *engine = kobj_to_engine(kobj); - unsigned long long timeout; + unsigned long long timeout, clamped; int err; /* @@ -322,7 +325,8 @@ preempt_timeout_store(struct kobject *kobj, struct kobj_attribute *attr, if (err) return err; - if (timeout > jiffies_to_msecs(MAX_SCHEDULE_TIMEOUT)) + clamped = intel_clamp_preempt_timeout_ms(engine, timeout); + if (timeout != clamped) return -EINVAL; WRITE_ONCE(engine->props.preempt_timeout_ms, timeout); @@ -362,7 +366,7 @@ heartbeat_store(struct kobject *kobj, struct kobj_attribute *attr, const char *buf, size_t count) { struct intel_engine_cs *engine = kobj_to_engine(kobj); - unsigned long long delay; + unsigned long long delay, clamped; int err; /* @@ -379,7 +383,8 @@ heartbeat_store(struct kobject *kobj, struct kobj_attribute *attr, if (err) return err; - if (delay >= jiffies_to_msecs(MAX_SCHEDULE_TIMEOUT)) + clamped = intel_clamp_heartbeat_interval_ms(engine, delay); + if (delay != clamped) return -EINVAL; err = intel_engine_set_heartbeat(engine, delay); diff --git a/drivers/gpu/drm/i915/gt/uc/abi/guc_actions_abi.h b/drivers/gpu/drm/i915/gt/uc/abi/guc_actions_abi.h index 29ef8afc8c2e..f359bef046e0 100644 --- a/drivers/gpu/drm/i915/gt/uc/abi/guc_actions_abi.h +++ b/drivers/gpu/drm/i915/gt/uc/abi/guc_actions_abi.h @@ -117,6 +117,7 @@ enum intel_guc_action { INTEL_GUC_ACTION_ENTER_S_STATE = 0x501, INTEL_GUC_ACTION_EXIT_S_STATE = 0x502, INTEL_GUC_ACTION_GLOBAL_SCHED_POLICY_CHANGE = 0x506, + INTEL_GUC_ACTION_UPDATE_SCHEDULING_POLICIES_KLV = 0x509, INTEL_GUC_ACTION_SCHED_CONTEXT = 0x1000, INTEL_GUC_ACTION_SCHED_CONTEXT_MODE_SET = 0x1001, INTEL_GUC_ACTION_SCHED_CONTEXT_MODE_DONE = 0x1002, diff --git a/drivers/gpu/drm/i915/gt/uc/abi/guc_actions_slpc_abi.h b/drivers/gpu/drm/i915/gt/uc/abi/guc_actions_slpc_abi.h index 4c840a2639dc..811add10c30d 100644 --- a/drivers/gpu/drm/i915/gt/uc/abi/guc_actions_slpc_abi.h +++ b/drivers/gpu/drm/i915/gt/uc/abi/guc_actions_slpc_abi.h @@ -128,6 +128,15 @@ enum slpc_media_ratio_mode { SLPC_MEDIA_RATIO_MODE_FIXED_ONE_TO_TWO = 2, }; +enum slpc_gucrc_mode { + SLPC_GUCRC_MODE_HW = 0, + SLPC_GUCRC_MODE_GUCRC_NO_RC6 = 1, + SLPC_GUCRC_MODE_GUCRC_STATIC_TIMEOUT = 2, + SLPC_GUCRC_MODE_GUCRC_DYNAMIC_HYSTERESIS = 3, + + SLPC_GUCRC_MODE_MAX, +}; + enum slpc_event_id { SLPC_EVENT_RESET = 0, SLPC_EVENT_SHUTDOWN = 1, diff --git a/drivers/gpu/drm/i915/gt/uc/abi/guc_klvs_abi.h b/drivers/gpu/drm/i915/gt/uc/abi/guc_klvs_abi.h index 4a59478c3b5c..58012edd4eb0 100644 --- a/drivers/gpu/drm/i915/gt/uc/abi/guc_klvs_abi.h +++ b/drivers/gpu/drm/i915/gt/uc/abi/guc_klvs_abi.h @@ -82,9 +82,16 @@ #define GUC_KLV_SELF_CFG_G2H_CTB_SIZE_LEN 1u /* + * Global scheduling policy update keys. + */ +enum { + GUC_SCHEDULING_POLICIES_KLV_ID_RENDER_COMPUTE_YIELD = 0x1001, +}; + +/* * Per context scheduling policy update keys. */ -enum { +enum { GUC_CONTEXT_POLICIES_KLV_ID_EXECUTION_QUANTUM = 0x2001, GUC_CONTEXT_POLICIES_KLV_ID_PREEMPTION_TIMEOUT = 0x2002, GUC_CONTEXT_POLICIES_KLV_ID_SCHEDULING_PRIORITY = 0x2003, diff --git a/drivers/gpu/drm/i915/gt/uc/intel_guc.c b/drivers/gpu/drm/i915/gt/uc/intel_guc.c index bac06e3d6f2c..52aede324788 100644 --- a/drivers/gpu/drm/i915/gt/uc/intel_guc.c +++ b/drivers/gpu/drm/i915/gt/uc/intel_guc.c @@ -98,6 +98,8 @@ static void gen9_enable_guc_interrupts(struct intel_guc *guc) gt->pm_guc_events); gen6_gt_pm_enable_irq(gt, gt->pm_guc_events); spin_unlock_irq(gt->irq_lock); + + guc->interrupts.enabled = true; } static void gen9_disable_guc_interrupts(struct intel_guc *guc) @@ -105,6 +107,7 @@ static void gen9_disable_guc_interrupts(struct intel_guc *guc) struct intel_gt *gt = guc_to_gt(guc); assert_rpm_wakelock_held(>->i915->runtime_pm); + guc->interrupts.enabled = false; spin_lock_irq(gt->irq_lock); @@ -116,39 +119,39 @@ static void gen9_disable_guc_interrupts(struct intel_guc *guc) gen9_reset_guc_interrupts(guc); } +static bool __gen11_reset_guc_interrupts(struct intel_gt *gt) +{ + u32 irq = gt->type == GT_MEDIA ? MTL_MGUC : GEN11_GUC; + + lockdep_assert_held(gt->irq_lock); + return gen11_gt_reset_one_iir(gt, 0, irq); +} + static void gen11_reset_guc_interrupts(struct intel_guc *guc) { struct intel_gt *gt = guc_to_gt(guc); spin_lock_irq(gt->irq_lock); - gen11_gt_reset_one_iir(gt, 0, GEN11_GUC); + __gen11_reset_guc_interrupts(gt); spin_unlock_irq(gt->irq_lock); } static void gen11_enable_guc_interrupts(struct intel_guc *guc) { struct intel_gt *gt = guc_to_gt(guc); - u32 events = REG_FIELD_PREP(ENGINE1_MASK, GUC_INTR_GUC2HOST); spin_lock_irq(gt->irq_lock); - WARN_ON_ONCE(gen11_gt_reset_one_iir(gt, 0, GEN11_GUC)); - intel_uncore_write(gt->uncore, - GEN11_GUC_SG_INTR_ENABLE, events); - intel_uncore_write(gt->uncore, - GEN11_GUC_SG_INTR_MASK, ~events); + __gen11_reset_guc_interrupts(gt); spin_unlock_irq(gt->irq_lock); + + guc->interrupts.enabled = true; } static void gen11_disable_guc_interrupts(struct intel_guc *guc) { struct intel_gt *gt = guc_to_gt(guc); - spin_lock_irq(gt->irq_lock); - - intel_uncore_write(gt->uncore, GEN11_GUC_SG_INTR_MASK, ~0); - intel_uncore_write(gt->uncore, GEN11_GUC_SG_INTR_ENABLE, 0); - - spin_unlock_irq(gt->irq_lock); + guc->interrupts.enabled = false; intel_synchronize_irq(gt->i915); gen11_reset_guc_interrupts(guc); @@ -156,7 +159,8 @@ static void gen11_disable_guc_interrupts(struct intel_guc *guc) void intel_guc_init_early(struct intel_guc *guc) { - struct drm_i915_private *i915 = guc_to_gt(guc)->i915; + struct intel_gt *gt = guc_to_gt(guc); + struct drm_i915_private *i915 = gt->i915; intel_uc_fw_init_early(&guc->fw, INTEL_UC_FW_TYPE_GUC); intel_guc_ct_init_early(&guc->ct); @@ -168,12 +172,17 @@ void intel_guc_init_early(struct intel_guc *guc) mutex_init(&guc->send_mutex); spin_lock_init(&guc->irq_lock); if (GRAPHICS_VER(i915) >= 11) { - guc->notify_reg = GEN11_GUC_HOST_INTERRUPT; guc->interrupts.reset = gen11_reset_guc_interrupts; guc->interrupts.enable = gen11_enable_guc_interrupts; guc->interrupts.disable = gen11_disable_guc_interrupts; - guc->send_regs.base = - i915_mmio_reg_offset(GEN11_SOFT_SCRATCH(0)); + if (gt->type == GT_MEDIA) { + guc->notify_reg = MEDIA_GUC_HOST_INTERRUPT; + guc->send_regs.base = i915_mmio_reg_offset(MEDIA_SOFT_SCRATCH(0)); + } else { + guc->notify_reg = GEN11_GUC_HOST_INTERRUPT; + guc->send_regs.base = i915_mmio_reg_offset(GEN11_SOFT_SCRATCH(0)); + } + guc->send_regs.count = GEN11_SOFT_SCRATCH_COUNT; } else { @@ -441,6 +450,7 @@ err_log: err_fw: intel_uc_fw_fini(&guc->fw); out: + intel_uc_fw_change_status(&guc->fw, INTEL_UC_FIRMWARE_INIT_FAIL); i915_probe_error(gt->i915, "failed with %d\n", ret); return ret; } @@ -870,14 +880,14 @@ void intel_guc_load_status(struct intel_guc *guc, struct drm_printer *p) u32 status = intel_uncore_read(uncore, GUC_STATUS); u32 i; - drm_printf(p, "\nGuC status 0x%08x:\n", status); + drm_printf(p, "GuC status 0x%08x:\n", status); drm_printf(p, "\tBootrom status = 0x%x\n", (status & GS_BOOTROM_MASK) >> GS_BOOTROM_SHIFT); drm_printf(p, "\tuKernel status = 0x%x\n", (status & GS_UKERNEL_MASK) >> GS_UKERNEL_SHIFT); drm_printf(p, "\tMIA Core status = 0x%x\n", (status & GS_MIA_MASK) >> GS_MIA_SHIFT); - drm_puts(p, "\nScratch registers:\n"); + drm_puts(p, "Scratch registers:\n"); for (i = 0; i < 16; i++) { drm_printf(p, "\t%2d: \t0x%x\n", i, intel_uncore_read(uncore, SOFT_SCRATCH(i))); diff --git a/drivers/gpu/drm/i915/gt/uc/intel_guc.h b/drivers/gpu/drm/i915/gt/uc/intel_guc.h index 804133df1ac9..1bb3f9829286 100644 --- a/drivers/gpu/drm/i915/gt/uc/intel_guc.h +++ b/drivers/gpu/drm/i915/gt/uc/intel_guc.h @@ -78,6 +78,7 @@ struct intel_guc { /** @interrupts: pointers to GuC interrupt-managing functions. */ struct { + bool enabled; void (*reset)(struct intel_guc *guc); void (*enable)(struct intel_guc *guc); void (*disable)(struct intel_guc *guc); @@ -113,6 +114,10 @@ struct intel_guc { */ struct list_head guc_id_list; /** + * @guc_ids_in_use: Number single-lrc guc_ids in use + */ + unsigned int guc_ids_in_use; + /** * @destroyed_contexts: list of contexts waiting to be destroyed * (deregistered with the GuC) */ @@ -132,6 +137,16 @@ struct intel_guc { * @reset_fail_mask: mask of engines that failed to reset */ intel_engine_mask_t reset_fail_mask; + /** + * @sched_disable_delay_ms: schedule disable delay, in ms, for + * contexts + */ + unsigned int sched_disable_delay_ms; + /** + * @sched_disable_gucid_threshold: threshold of min remaining available + * guc_ids before we start bypassing the schedule disable delay + */ + unsigned int sched_disable_gucid_threshold; } submission_state; /** @@ -316,9 +331,11 @@ retry: return err; } +/* Only call this from the interrupt handler code */ static inline void intel_guc_to_host_event_handler(struct intel_guc *guc) { - intel_guc_ct_event_handler(&guc->ct); + if (guc->interrupts.enabled) + intel_guc_ct_event_handler(&guc->ct); } /* GuC addresses above GUC_GGTT_TOP also don't map through the GTT */ @@ -466,4 +483,6 @@ void intel_guc_write_barrier(struct intel_guc *guc); void intel_guc_dump_time_info(struct intel_guc *guc, struct drm_printer *p); +int intel_guc_sched_disable_gucid_threshold_max(struct intel_guc *guc); + #endif diff --git a/drivers/gpu/drm/i915/gt/uc/intel_guc_ads.c b/drivers/gpu/drm/i915/gt/uc/intel_guc_ads.c index 74cbe8eaf531..a7f737c4792e 100644 --- a/drivers/gpu/drm/i915/gt/uc/intel_guc_ads.c +++ b/drivers/gpu/drm/i915/gt/uc/intel_guc_ads.c @@ -5,6 +5,7 @@ #include <linux/bsearch.h> +#include "gem/i915_gem_lmem.h" #include "gt/intel_engine_regs.h" #include "gt/intel_gt.h" #include "gt/intel_gt_mcr.h" @@ -277,24 +278,16 @@ __mmio_reg_add(struct temp_regset *regset, struct guc_mmio_reg *reg) return slot; } -#define GUC_REGSET_STEERING(group, instance) ( \ - FIELD_PREP(GUC_REGSET_STEERING_GROUP, (group)) | \ - FIELD_PREP(GUC_REGSET_STEERING_INSTANCE, (instance)) | \ - GUC_REGSET_NEEDS_STEERING \ -) - static long __must_check guc_mmio_reg_add(struct intel_gt *gt, struct temp_regset *regset, - i915_reg_t reg, u32 flags) + u32 offset, u32 flags) { u32 count = regset->storage_used - (regset->registers - regset->storage); - u32 offset = i915_mmio_reg_offset(reg); struct guc_mmio_reg entry = { .offset = offset, .flags = flags, }; struct guc_mmio_reg *slot; - u8 group, inst; /* * The mmio list is built using separate lists within the driver. @@ -306,17 +299,6 @@ static long __must_check guc_mmio_reg_add(struct intel_gt *gt, sizeof(entry), guc_mmio_reg_cmp)) return 0; - /* - * The GuC doesn't have a default steering, so we need to explicitly - * steer all registers that need steering. However, we do not keep track - * of all the steering ranges, only of those that have a chance of using - * a non-default steering from the i915 pov. Instead of adding such - * tracking, it is easier to just program the default steering for all - * regs that don't need a non-default one. - */ - intel_gt_mcr_get_nonterminated_steering(gt, reg, &group, &inst); - entry.flags |= GUC_REGSET_STEERING(group, inst); - slot = __mmio_reg_add(regset, &entry); if (IS_ERR(slot)) return PTR_ERR(slot); @@ -335,6 +317,38 @@ static long __must_check guc_mmio_reg_add(struct intel_gt *gt, #define GUC_MMIO_REG_ADD(gt, regset, reg, masked) \ guc_mmio_reg_add(gt, \ regset, \ + i915_mmio_reg_offset(reg), \ + (masked) ? GUC_REGSET_MASKED : 0) + +#define GUC_REGSET_STEERING(group, instance) ( \ + FIELD_PREP(GUC_REGSET_STEERING_GROUP, (group)) | \ + FIELD_PREP(GUC_REGSET_STEERING_INSTANCE, (instance)) | \ + GUC_REGSET_NEEDS_STEERING \ +) + +static long __must_check guc_mcr_reg_add(struct intel_gt *gt, + struct temp_regset *regset, + i915_mcr_reg_t reg, u32 flags) +{ + u8 group, inst; + + /* + * The GuC doesn't have a default steering, so we need to explicitly + * steer all registers that need steering. However, we do not keep track + * of all the steering ranges, only of those that have a chance of using + * a non-default steering from the i915 pov. Instead of adding such + * tracking, it is easier to just program the default steering for all + * regs that don't need a non-default one. + */ + intel_gt_mcr_get_nonterminated_steering(gt, reg, &group, &inst); + flags |= GUC_REGSET_STEERING(group, inst); + + return guc_mmio_reg_add(gt, regset, i915_mmio_reg_offset(reg), flags); +} + +#define GUC_MCR_REG_ADD(gt, regset, reg, masked) \ + guc_mcr_reg_add(gt, \ + regset, \ (reg), \ (masked) ? GUC_REGSET_MASKED : 0) @@ -372,8 +386,21 @@ static int guc_mmio_regset_init(struct temp_regset *regset, false); /* add in local MOCS registers */ - for (i = 0; i < GEN9_LNCFCMOCS_REG_COUNT; i++) - ret |= GUC_MMIO_REG_ADD(gt, regset, GEN9_LNCFCMOCS(i), false); + for (i = 0; i < LNCFCMOCS_REG_COUNT; i++) + if (GRAPHICS_VER_FULL(engine->i915) >= IP_VER(12, 50)) + ret |= GUC_MCR_REG_ADD(gt, regset, XEHP_LNCFCMOCS(i), false); + else + ret |= GUC_MMIO_REG_ADD(gt, regset, GEN9_LNCFCMOCS(i), false); + + if (GRAPHICS_VER(engine->i915) >= 12) { + ret |= GUC_MMIO_REG_ADD(gt, regset, EU_PERF_CNTL0, false); + ret |= GUC_MMIO_REG_ADD(gt, regset, EU_PERF_CNTL1, false); + ret |= GUC_MMIO_REG_ADD(gt, regset, EU_PERF_CNTL2, false); + ret |= GUC_MMIO_REG_ADD(gt, regset, EU_PERF_CNTL3, false); + ret |= GUC_MMIO_REG_ADD(gt, regset, EU_PERF_CNTL4, false); + ret |= GUC_MMIO_REG_ADD(gt, regset, EU_PERF_CNTL5, false); + ret |= GUC_MMIO_REG_ADD(gt, regset, EU_PERF_CNTL6, false); + } return ret ? -1 : 0; } @@ -461,6 +488,11 @@ static void fill_engine_enable_masks(struct intel_gt *gt, info_map_write(info_map, engine_enabled_masks[GUC_BLITTER_CLASS], BCS_MASK(gt)); info_map_write(info_map, engine_enabled_masks[GUC_VIDEO_CLASS], VDBOX_MASK(gt)); info_map_write(info_map, engine_enabled_masks[GUC_VIDEOENHANCE_CLASS], VEBOX_MASK(gt)); + + /* The GSC engine is an instance (6) of OTHER_CLASS */ + if (gt->engine[GSC0]) + info_map_write(info_map, engine_enabled_masks[GUC_GSC_OTHER_CLASS], + BIT(gt->engine[GSC0]->instance)); } #define LR_HW_CONTEXT_SIZE (80 * sizeof(u32)) @@ -502,9 +534,6 @@ static int guc_prep_golden_context(struct intel_guc *guc) } for (engine_class = 0; engine_class <= MAX_ENGINE_CLASS; ++engine_class) { - if (engine_class == OTHER_CLASS) - continue; - guc_class = engine_class_to_guc_class(engine_class); if (!info_map_read(&info_map, engine_enabled_masks[guc_class])) @@ -582,9 +611,6 @@ static void guc_init_golden_context(struct intel_guc *guc) addr_ggtt = intel_guc_ggtt_offset(guc, guc->ads_vma) + offset; for (engine_class = 0; engine_class <= MAX_ENGINE_CLASS; ++engine_class) { - if (engine_class == OTHER_CLASS) - continue; - guc_class = engine_class_to_guc_class(engine_class); if (!ads_blob_read(guc, system_info.engine_enabled_masks[guc_class])) continue; diff --git a/drivers/gpu/drm/i915/gt/uc/intel_guc_capture.c b/drivers/gpu/drm/i915/gt/uc/intel_guc_capture.c index 8f1165146013..1c1b85073b4b 100644 --- a/drivers/gpu/drm/i915/gt/uc/intel_guc_capture.c +++ b/drivers/gpu/drm/i915/gt/uc/intel_guc_capture.c @@ -132,6 +132,11 @@ static const struct __guc_mmio_reg_descr xe_lpd_blt_inst_regs[] = { COMMON_BASE_ENGINE_INSTANCE, }; +/* XE_LPD - GSC Per-Engine-Instance */ +static const struct __guc_mmio_reg_descr xe_lpd_gsc_inst_regs[] = { + COMMON_BASE_ENGINE_INSTANCE, +}; + /* GEN9 - Global */ static const struct __guc_mmio_reg_descr default_global_regs[] = { COMMON_BASE_GLOBAL, @@ -165,16 +170,20 @@ static const struct __guc_mmio_reg_descr empty_regs_list[] = { } /* List of lists */ -static struct __guc_mmio_reg_descr_group default_lists[] = { +static const struct __guc_mmio_reg_descr_group default_lists[] = { MAKE_REGLIST(default_global_regs, PF, GLOBAL, 0), MAKE_REGLIST(default_rc_class_regs, PF, ENGINE_CLASS, GUC_RENDER_CLASS), MAKE_REGLIST(xe_lpd_rc_inst_regs, PF, ENGINE_INSTANCE, GUC_RENDER_CLASS), + MAKE_REGLIST(default_rc_class_regs, PF, ENGINE_CLASS, GUC_COMPUTE_CLASS), + MAKE_REGLIST(xe_lpd_rc_inst_regs, PF, ENGINE_INSTANCE, GUC_COMPUTE_CLASS), MAKE_REGLIST(empty_regs_list, PF, ENGINE_CLASS, GUC_VIDEO_CLASS), MAKE_REGLIST(xe_lpd_vd_inst_regs, PF, ENGINE_INSTANCE, GUC_VIDEO_CLASS), MAKE_REGLIST(empty_regs_list, PF, ENGINE_CLASS, GUC_VIDEOENHANCE_CLASS), MAKE_REGLIST(xe_lpd_vec_inst_regs, PF, ENGINE_INSTANCE, GUC_VIDEOENHANCE_CLASS), MAKE_REGLIST(empty_regs_list, PF, ENGINE_CLASS, GUC_BLITTER_CLASS), MAKE_REGLIST(xe_lpd_blt_inst_regs, PF, ENGINE_INSTANCE, GUC_BLITTER_CLASS), + MAKE_REGLIST(empty_regs_list, PF, ENGINE_CLASS, GUC_GSC_OTHER_CLASS), + MAKE_REGLIST(xe_lpd_gsc_inst_regs, PF, ENGINE_INSTANCE, GUC_GSC_OTHER_CLASS), {} }; @@ -182,12 +191,16 @@ static const struct __guc_mmio_reg_descr_group xe_lpd_lists[] = { MAKE_REGLIST(xe_lpd_global_regs, PF, GLOBAL, 0), MAKE_REGLIST(xe_lpd_rc_class_regs, PF, ENGINE_CLASS, GUC_RENDER_CLASS), MAKE_REGLIST(xe_lpd_rc_inst_regs, PF, ENGINE_INSTANCE, GUC_RENDER_CLASS), + MAKE_REGLIST(xe_lpd_rc_class_regs, PF, ENGINE_CLASS, GUC_COMPUTE_CLASS), + MAKE_REGLIST(xe_lpd_rc_inst_regs, PF, ENGINE_INSTANCE, GUC_COMPUTE_CLASS), MAKE_REGLIST(empty_regs_list, PF, ENGINE_CLASS, GUC_VIDEO_CLASS), MAKE_REGLIST(xe_lpd_vd_inst_regs, PF, ENGINE_INSTANCE, GUC_VIDEO_CLASS), MAKE_REGLIST(xe_lpd_vec_class_regs, PF, ENGINE_CLASS, GUC_VIDEOENHANCE_CLASS), MAKE_REGLIST(xe_lpd_vec_inst_regs, PF, ENGINE_INSTANCE, GUC_VIDEOENHANCE_CLASS), MAKE_REGLIST(empty_regs_list, PF, ENGINE_CLASS, GUC_BLITTER_CLASS), MAKE_REGLIST(xe_lpd_blt_inst_regs, PF, ENGINE_INSTANCE, GUC_BLITTER_CLASS), + MAKE_REGLIST(empty_regs_list, PF, ENGINE_CLASS, GUC_GSC_OTHER_CLASS), + MAKE_REGLIST(xe_lpd_gsc_inst_regs, PF, ENGINE_INSTANCE, GUC_GSC_OTHER_CLASS), {} }; @@ -240,19 +253,19 @@ static void guc_capture_free_extlists(struct __guc_mmio_reg_descr_group *reglist struct __ext_steer_reg { const char *name; - i915_reg_t reg; + i915_mcr_reg_t reg; }; static const struct __ext_steer_reg xe_extregs[] = { - {"GEN7_SAMPLER_INSTDONE", GEN7_SAMPLER_INSTDONE}, - {"GEN7_ROW_INSTDONE", GEN7_ROW_INSTDONE} + {"GEN8_SAMPLER_INSTDONE", GEN8_SAMPLER_INSTDONE}, + {"GEN8_ROW_INSTDONE", GEN8_ROW_INSTDONE} }; static void __fill_ext_reg(struct __guc_mmio_reg_descr *ext, const struct __ext_steer_reg *extlist, int slice_id, int subslice_id) { - ext->reg = extlist->reg; + ext->reg = _MMIO(i915_mmio_reg_offset(extlist->reg)); ext->flags = FIELD_PREP(GUC_REGSET_STEERING_GROUP, slice_id); ext->flags |= FIELD_PREP(GUC_REGSET_STEERING_INSTANCE, subslice_id); ext->regname = extlist->name; @@ -419,6 +432,46 @@ guc_capture_get_device_reglist(struct intel_guc *guc) return default_lists; } +static const char * +__stringify_type(u32 type) +{ + switch (type) { + case GUC_CAPTURE_LIST_TYPE_GLOBAL: + return "Global"; + case GUC_CAPTURE_LIST_TYPE_ENGINE_CLASS: + return "Class"; + case GUC_CAPTURE_LIST_TYPE_ENGINE_INSTANCE: + return "Instance"; + default: + break; + } + + return "unknown"; +} + +static const char * +__stringify_engclass(u32 class) +{ + switch (class) { + case GUC_RENDER_CLASS: + return "Render"; + case GUC_VIDEO_CLASS: + return "Video"; + case GUC_VIDEOENHANCE_CLASS: + return "VideoEnhance"; + case GUC_BLITTER_CLASS: + return "Blitter"; + case GUC_COMPUTE_CLASS: + return "Compute"; + case GUC_GSC_OTHER_CLASS: + return "GSC-Other"; + default: + break; + } + + return "unknown"; +} + static int guc_capture_list_init(struct intel_guc *guc, u32 owner, u32 type, u32 classid, struct guc_mmio_reg *ptr, u16 num_entries) @@ -482,32 +535,55 @@ guc_cap_list_num_regs(struct intel_guc_state_capture *gc, u32 owner, u32 type, u return num_regs; } -int -intel_guc_capture_getlistsize(struct intel_guc *guc, u32 owner, u32 type, u32 classid, - size_t *size) +static int +guc_capture_getlistsize(struct intel_guc *guc, u32 owner, u32 type, u32 classid, + size_t *size, bool is_purpose_est) { struct intel_guc_state_capture *gc = guc->capture; + struct drm_i915_private *i915 = guc_to_gt(guc)->i915; struct __guc_capture_ads_cache *cache = &gc->ads_cache[owner][type][classid]; int num_regs; - if (!gc->reglists) + if (!gc->reglists) { + drm_warn(&i915->drm, "GuC-capture: No reglist on this device\n"); return -ENODEV; + } if (cache->is_valid) { *size = cache->size; return cache->status; } + if (!is_purpose_est && owner == GUC_CAPTURE_LIST_INDEX_PF && + !guc_capture_get_one_list(gc->reglists, owner, type, classid)) { + if (type == GUC_CAPTURE_LIST_TYPE_GLOBAL) + drm_warn(&i915->drm, "Missing GuC-Err-Cap reglist Global!\n"); + else + drm_warn(&i915->drm, "Missing GuC-Err-Cap reglist %s(%u):%s(%u)!\n", + __stringify_type(type), type, + __stringify_engclass(classid), classid); + return -ENODATA; + } + num_regs = guc_cap_list_num_regs(gc, owner, type, classid); + /* intentional empty lists can exist depending on hw config */ if (!num_regs) return -ENODATA; - *size = PAGE_ALIGN((sizeof(struct guc_debug_capture_list)) + - (num_regs * sizeof(struct guc_mmio_reg))); + if (size) + *size = PAGE_ALIGN((sizeof(struct guc_debug_capture_list)) + + (num_regs * sizeof(struct guc_mmio_reg))); return 0; } +int +intel_guc_capture_getlistsize(struct intel_guc *guc, u32 owner, u32 type, u32 classid, + size_t *size) +{ + return guc_capture_getlistsize(guc, owner, type, classid, size, false); +} + static void guc_capture_create_prealloc_nodes(struct intel_guc *guc); int @@ -606,7 +682,7 @@ guc_capture_output_min_size_est(struct intel_guc *guc) struct intel_gt *gt = guc_to_gt(guc); struct intel_engine_cs *engine; enum intel_engine_id id; - int worst_min_size = 0, num_regs = 0; + int worst_min_size = 0; size_t tmp = 0; if (!guc->capture) @@ -627,21 +703,19 @@ guc_capture_output_min_size_est(struct intel_guc *guc) worst_min_size += sizeof(struct guc_state_capture_group_header_t) + (3 * sizeof(struct guc_state_capture_header_t)); - if (!intel_guc_capture_getlistsize(guc, 0, GUC_CAPTURE_LIST_TYPE_GLOBAL, 0, &tmp)) - num_regs += tmp; + if (!guc_capture_getlistsize(guc, 0, GUC_CAPTURE_LIST_TYPE_GLOBAL, 0, &tmp, true)) + worst_min_size += tmp; - if (!intel_guc_capture_getlistsize(guc, 0, GUC_CAPTURE_LIST_TYPE_ENGINE_CLASS, - engine->class, &tmp)) { - num_regs += tmp; + if (!guc_capture_getlistsize(guc, 0, GUC_CAPTURE_LIST_TYPE_ENGINE_CLASS, + engine->class, &tmp, true)) { + worst_min_size += tmp; } - if (!intel_guc_capture_getlistsize(guc, 0, GUC_CAPTURE_LIST_TYPE_ENGINE_INSTANCE, - engine->class, &tmp)) { - num_regs += tmp; + if (!guc_capture_getlistsize(guc, 0, GUC_CAPTURE_LIST_TYPE_ENGINE_INSTANCE, + engine->class, &tmp, true)) { + worst_min_size += tmp; } } - worst_min_size += (num_regs * sizeof(struct guc_mmio_reg)); - return worst_min_size; } @@ -658,15 +732,23 @@ static void check_guc_capture_size(struct intel_guc *guc) int spare_size = min_size * GUC_CAPTURE_OVERBUFFER_MULTIPLIER; u32 buffer_size = intel_guc_log_section_size_capture(&guc->log); + /* + * NOTE: min_size is much smaller than the capture region allocation (DG2: <80K vs 1MB) + * Additionally, its based on space needed to fit all engines getting reset at once + * within the same G2H handler task slot. This is very unlikely. However, if GuC really + * does run out of space for whatever reason, we will see an separate warning message + * when processing the G2H event capture-notification, search for: + * INTEL_GUC_STATE_CAPTURE_EVENT_STATUS_NOSPACE. + */ if (min_size < 0) drm_warn(&i915->drm, "Failed to calculate GuC error state capture buffer minimum size: %d!\n", min_size); else if (min_size > buffer_size) - drm_warn(&i915->drm, "GuC error state capture buffer is too small: %d < %d\n", + drm_warn(&i915->drm, "GuC error state capture buffer maybe small: %d < %d\n", buffer_size, min_size); else if (spare_size > buffer_size) - drm_notice(&i915->drm, "GuC error state capture buffer maybe too small: %d < %d (min = %d)\n", - buffer_size, spare_size, min_size); + drm_dbg(&i915->drm, "GuC error state capture buffer lacks spare size: %d < %d (min = %d)\n", + buffer_size, spare_size, min_size); } /* diff --git a/drivers/gpu/drm/i915/gt/uc/intel_guc_debugfs.c b/drivers/gpu/drm/i915/gt/uc/intel_guc_debugfs.c index 25f09a420561..7269eb0bbedf 100644 --- a/drivers/gpu/drm/i915/gt/uc/intel_guc_debugfs.c +++ b/drivers/gpu/drm/i915/gt/uc/intel_guc_debugfs.c @@ -71,12 +71,73 @@ static bool intel_eval_slpc_support(void *data) return intel_guc_slpc_is_used(guc); } +static int guc_sched_disable_delay_ms_get(void *data, u64 *val) +{ + struct intel_guc *guc = data; + + if (!intel_guc_submission_is_used(guc)) + return -ENODEV; + + *val = (u64)guc->submission_state.sched_disable_delay_ms; + + return 0; +} + +static int guc_sched_disable_delay_ms_set(void *data, u64 val) +{ + struct intel_guc *guc = data; + + if (!intel_guc_submission_is_used(guc)) + return -ENODEV; + + /* clamp to a practical limit, 1 minute is reasonable for a longest delay */ + guc->submission_state.sched_disable_delay_ms = min_t(u64, val, 60000); + + return 0; +} +DEFINE_SIMPLE_ATTRIBUTE(guc_sched_disable_delay_ms_fops, + guc_sched_disable_delay_ms_get, + guc_sched_disable_delay_ms_set, "%lld\n"); + +static int guc_sched_disable_gucid_threshold_get(void *data, u64 *val) +{ + struct intel_guc *guc = data; + + if (!intel_guc_submission_is_used(guc)) + return -ENODEV; + + *val = guc->submission_state.sched_disable_gucid_threshold; + return 0; +} + +static int guc_sched_disable_gucid_threshold_set(void *data, u64 val) +{ + struct intel_guc *guc = data; + + if (!intel_guc_submission_is_used(guc)) + return -ENODEV; + + if (val > intel_guc_sched_disable_gucid_threshold_max(guc)) + guc->submission_state.sched_disable_gucid_threshold = + intel_guc_sched_disable_gucid_threshold_max(guc); + else + guc->submission_state.sched_disable_gucid_threshold = val; + + return 0; +} +DEFINE_SIMPLE_ATTRIBUTE(guc_sched_disable_gucid_threshold_fops, + guc_sched_disable_gucid_threshold_get, + guc_sched_disable_gucid_threshold_set, "%lld\n"); + void intel_guc_debugfs_register(struct intel_guc *guc, struct dentry *root) { static const struct intel_gt_debugfs_file files[] = { { "guc_info", &guc_info_fops, NULL }, { "guc_registered_contexts", &guc_registered_contexts_fops, NULL }, { "guc_slpc_info", &guc_slpc_info_fops, &intel_eval_slpc_support}, + { "guc_sched_disable_delay_ms", &guc_sched_disable_delay_ms_fops, NULL }, + { "guc_sched_disable_gucid_threshold", &guc_sched_disable_gucid_threshold_fops, + NULL }, }; if (!intel_guc_is_supported(guc)) diff --git a/drivers/gpu/drm/i915/gt/uc/intel_guc_fw.c b/drivers/gpu/drm/i915/gt/uc/intel_guc_fw.c index a0372735cddb..5b86b2e286e0 100644 --- a/drivers/gpu/drm/i915/gt/uc/intel_guc_fw.c +++ b/drivers/gpu/drm/i915/gt/uc/intel_guc_fw.c @@ -10,12 +10,15 @@ */ #include "gt/intel_gt.h" +#include "gt/intel_gt_mcr.h" #include "gt/intel_gt_regs.h" #include "intel_guc_fw.h" #include "i915_drv.h" -static void guc_prepare_xfer(struct intel_uncore *uncore) +static void guc_prepare_xfer(struct intel_gt *gt) { + struct intel_uncore *uncore = gt->uncore; + u32 shim_flags = GUC_ENABLE_READ_CACHE_LOGIC | GUC_ENABLE_READ_CACHE_FOR_SRAM_DATA | GUC_ENABLE_READ_CACHE_FOR_WOPCM_DATA | @@ -35,8 +38,9 @@ static void guc_prepare_xfer(struct intel_uncore *uncore) if (GRAPHICS_VER(uncore->i915) == 9) { /* DOP Clock Gating Enable for GuC clocks */ - intel_uncore_rmw(uncore, GEN7_MISCCPCTL, - 0, GEN8_DOP_CLOCK_GATE_GUC_ENABLE); + intel_gt_mcr_multicast_write(gt, GEN8_MISCCPCTL, + GEN8_DOP_CLOCK_GATE_GUC_ENABLE | + intel_gt_mcr_read_any(gt, GEN8_MISCCPCTL)); /* allows for 5us (in 10ns units) before GT can go to RC6 */ intel_uncore_write(uncore, GUC_ARAT_C6DIS, 0x1FF); @@ -168,7 +172,7 @@ int intel_guc_fw_upload(struct intel_guc *guc) struct intel_uncore *uncore = gt->uncore; int ret; - guc_prepare_xfer(uncore); + guc_prepare_xfer(gt); /* * Note that GuC needs the CSS header plus uKernel code to be copied diff --git a/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h b/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h index 323b055e5db9..4ae5fc2f6002 100644 --- a/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h +++ b/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h @@ -47,7 +47,8 @@ #define GUC_VIDEOENHANCE_CLASS 2 #define GUC_BLITTER_CLASS 3 #define GUC_COMPUTE_CLASS 4 -#define GUC_LAST_ENGINE_CLASS GUC_COMPUTE_CLASS +#define GUC_GSC_OTHER_CLASS 5 +#define GUC_LAST_ENGINE_CLASS GUC_GSC_OTHER_CLASS #define GUC_MAX_ENGINE_CLASSES 16 #define GUC_MAX_INSTANCES_PER_CLASS 32 @@ -169,6 +170,7 @@ static u8 engine_class_guc_class_map[] = { [COPY_ENGINE_CLASS] = GUC_BLITTER_CLASS, [VIDEO_DECODE_CLASS] = GUC_VIDEO_CLASS, [VIDEO_ENHANCEMENT_CLASS] = GUC_VIDEOENHANCE_CLASS, + [OTHER_CLASS] = GUC_GSC_OTHER_CLASS, [COMPUTE_CLASS] = GUC_COMPUTE_CLASS, }; @@ -178,12 +180,13 @@ static u8 guc_class_engine_class_map[] = { [GUC_VIDEO_CLASS] = VIDEO_DECODE_CLASS, [GUC_VIDEOENHANCE_CLASS] = VIDEO_ENHANCEMENT_CLASS, [GUC_COMPUTE_CLASS] = COMPUTE_CLASS, + [GUC_GSC_OTHER_CLASS] = OTHER_CLASS, }; static inline u8 engine_class_to_guc_class(u8 class) { BUILD_BUG_ON(ARRAY_SIZE(engine_class_guc_class_map) != MAX_ENGINE_CLASS + 1); - GEM_BUG_ON(class > MAX_ENGINE_CLASS || class == OTHER_CLASS); + GEM_BUG_ON(class > MAX_ENGINE_CLASS); return engine_class_guc_class_map[class]; } @@ -290,6 +293,25 @@ struct guc_update_context_policy { struct guc_klv_generic_dw_t klv[GUC_CONTEXT_POLICIES_KLV_NUM_IDS]; } __packed; +/* Format of the UPDATE_SCHEDULING_POLICIES H2G data packet */ +struct guc_update_scheduling_policy_header { + u32 action; +} __packed; + +/* + * Can't dynmically allocate memory for the scheduling policy KLV because + * it will be sent from within the reset path. Need a fixed size lump on + * the stack instead :(. + * + * Currently, there is only one KLV defined, which has 1 word of KL + 2 words of V. + */ +#define MAX_SCHEDULING_POLICY_SIZE 3 + +struct guc_update_scheduling_policy { + struct guc_update_scheduling_policy_header header; + u32 data[MAX_SCHEDULING_POLICY_SIZE]; +} __packed; + #define GUC_POWER_UNSPECIFIED 0 #define GUC_POWER_D0 1 #define GUC_POWER_D1 2 @@ -298,6 +320,9 @@ struct guc_update_context_policy { /* Scheduling policy settings */ +#define GLOBAL_SCHEDULE_POLICY_RC_YIELD_DURATION 100 /* in ms */ +#define GLOBAL_SCHEDULE_POLICY_RC_YIELD_RATIO 50 /* in percent */ + #define GLOBAL_POLICY_MAX_NUM_WI 15 /* Don't reset an engine upon preemption failure */ @@ -305,6 +330,27 @@ struct guc_update_context_policy { #define GLOBAL_POLICY_DEFAULT_DPC_PROMOTE_TIME_US 500000 +/* + * GuC converts the timeout to clock ticks internally. Different platforms have + * different GuC clocks. Thus, the maximum value before overflow is platform + * dependent. Current worst case scenario is about 110s. So, the spec says to + * limit to 100s to be safe. + */ +#define GUC_POLICY_MAX_EXEC_QUANTUM_US (100 * 1000 * 1000UL) +#define GUC_POLICY_MAX_PREEMPT_TIMEOUT_US (100 * 1000 * 1000UL) + +static inline u32 guc_policy_max_exec_quantum_ms(void) +{ + BUILD_BUG_ON(GUC_POLICY_MAX_EXEC_QUANTUM_US >= UINT_MAX); + return GUC_POLICY_MAX_EXEC_QUANTUM_US / 1000; +} + +static inline u32 guc_policy_max_preempt_timeout_ms(void) +{ + BUILD_BUG_ON(GUC_POLICY_MAX_PREEMPT_TIMEOUT_US >= UINT_MAX); + return GUC_POLICY_MAX_PREEMPT_TIMEOUT_US / 1000; +} + struct guc_policies { u32 submission_queue_depth[GUC_MAX_ENGINE_CLASSES]; /* In micro seconds. How much time to allow before DPC processing is diff --git a/drivers/gpu/drm/i915/gt/uc/intel_guc_log.c b/drivers/gpu/drm/i915/gt/uc/intel_guc_log.c index 55d3ef93e86f..68331c538b0a 100644 --- a/drivers/gpu/drm/i915/gt/uc/intel_guc_log.c +++ b/drivers/gpu/drm/i915/gt/uc/intel_guc_log.c @@ -16,15 +16,15 @@ #if defined(CONFIG_DRM_I915_DEBUG_GUC) #define GUC_LOG_DEFAULT_CRASH_BUFFER_SIZE SZ_2M #define GUC_LOG_DEFAULT_DEBUG_BUFFER_SIZE SZ_16M -#define GUC_LOG_DEFAULT_CAPTURE_BUFFER_SIZE SZ_4M +#define GUC_LOG_DEFAULT_CAPTURE_BUFFER_SIZE SZ_1M #elif defined(CONFIG_DRM_I915_DEBUG_GEM) #define GUC_LOG_DEFAULT_CRASH_BUFFER_SIZE SZ_1M #define GUC_LOG_DEFAULT_DEBUG_BUFFER_SIZE SZ_2M -#define GUC_LOG_DEFAULT_CAPTURE_BUFFER_SIZE SZ_4M +#define GUC_LOG_DEFAULT_CAPTURE_BUFFER_SIZE SZ_1M #else #define GUC_LOG_DEFAULT_CRASH_BUFFER_SIZE SZ_8K #define GUC_LOG_DEFAULT_DEBUG_BUFFER_SIZE SZ_64K -#define GUC_LOG_DEFAULT_CAPTURE_BUFFER_SIZE SZ_2M +#define GUC_LOG_DEFAULT_CAPTURE_BUFFER_SIZE SZ_1M #endif static void guc_log_copy_debuglogs_for_relay(struct intel_guc_log *log); diff --git a/drivers/gpu/drm/i915/gt/uc/intel_guc_rc.c b/drivers/gpu/drm/i915/gt/uc/intel_guc_rc.c index 8f8dd05835c5..b5855091cf6a 100644 --- a/drivers/gpu/drm/i915/gt/uc/intel_guc_rc.c +++ b/drivers/gpu/drm/i915/gt/uc/intel_guc_rc.c @@ -11,9 +11,20 @@ static bool __guc_rc_supported(struct intel_guc *guc) { + struct intel_gt *gt = guc_to_gt(guc); + + /* + * Wa_14017073508: mtl + * Do not enable gucrc to avoid additional interrupts which + * may disrupt pcode wa. + */ + if (IS_MTL_GRAPHICS_STEP(gt->i915, P, STEP_A0, STEP_B0) && + gt->type == GT_MEDIA) + return false; + /* GuC RC is unavailable for pre-Gen12 */ return guc->submission_supported && - GRAPHICS_VER(guc_to_gt(guc)->i915) >= 12; + GRAPHICS_VER(gt->i915) >= 12; } static bool __guc_rc_selected(struct intel_guc *guc) diff --git a/drivers/gpu/drm/i915/gt/uc/intel_guc_reg.h b/drivers/gpu/drm/i915/gt/uc/intel_guc_reg.h index a7092f711e9c..9915de32e894 100644 --- a/drivers/gpu/drm/i915/gt/uc/intel_guc_reg.h +++ b/drivers/gpu/drm/i915/gt/uc/intel_guc_reg.h @@ -36,6 +36,7 @@ #define SOFT_SCRATCH_COUNT 16 #define GEN11_SOFT_SCRATCH(n) _MMIO(0x190240 + (n) * 4) +#define MEDIA_SOFT_SCRATCH(n) _MMIO(0x190310 + (n) * 4) #define GEN11_SOFT_SCRATCH_COUNT 4 #define UOS_RSA_SCRATCH(i) _MMIO(0xc200 + (i) * 4) @@ -101,6 +102,7 @@ #define GUC_SEND_INTERRUPT _MMIO(0xc4c8) #define GUC_SEND_TRIGGER (1<<0) #define GEN11_GUC_HOST_INTERRUPT _MMIO(0x1901f0) +#define MEDIA_GUC_HOST_INTERRUPT _MMIO(0x190304) #define GEN12_GUC_SEM_INTR_ENABLES _MMIO(0xc71c) #define GUC_SEM_INTR_ROUTE_TO_GUC BIT(31) diff --git a/drivers/gpu/drm/i915/gt/uc/intel_guc_slpc.c b/drivers/gpu/drm/i915/gt/uc/intel_guc_slpc.c index fdd895f73f9f..63464933cbce 100644 --- a/drivers/gpu/drm/i915/gt/uc/intel_guc_slpc.c +++ b/drivers/gpu/drm/i915/gt/uc/intel_guc_slpc.c @@ -137,6 +137,17 @@ static int guc_action_slpc_set_param(struct intel_guc *guc, u8 id, u32 value) return ret > 0 ? -EPROTO : ret; } +static int guc_action_slpc_unset_param(struct intel_guc *guc, u8 id) +{ + u32 request[] = { + GUC_ACTION_HOST2GUC_PC_SLPC_REQUEST, + SLPC_EVENT(SLPC_EVENT_PARAMETER_UNSET, 1), + id, + }; + + return intel_guc_send(guc, request, ARRAY_SIZE(request)); +} + static bool slpc_is_running(struct intel_guc_slpc *slpc) { return slpc_get_state(slpc) == SLPC_GLOBAL_STATE_RUNNING; @@ -190,6 +201,15 @@ static int slpc_set_param(struct intel_guc_slpc *slpc, u8 id, u32 value) return ret; } +static int slpc_unset_param(struct intel_guc_slpc *slpc, u8 id) +{ + struct intel_guc *guc = slpc_to_guc(slpc); + + GEM_BUG_ON(id >= SLPC_MAX_PARAM); + + return guc_action_slpc_unset_param(guc, id); +} + static int slpc_force_min_freq(struct intel_guc_slpc *slpc, u32 freq) { struct drm_i915_private *i915 = slpc_to_i915(slpc); @@ -263,6 +283,7 @@ int intel_guc_slpc_init(struct intel_guc_slpc *slpc) slpc->max_freq_softlimit = 0; slpc->min_freq_softlimit = 0; + slpc->min_is_rpmax = false; slpc->boost_freq = 0; atomic_set(&slpc->num_waiters, 0); @@ -588,6 +609,39 @@ static int slpc_set_softlimits(struct intel_guc_slpc *slpc) return 0; } +static bool is_slpc_min_freq_rpmax(struct intel_guc_slpc *slpc) +{ + struct drm_i915_private *i915 = slpc_to_i915(slpc); + int slpc_min_freq; + int ret; + + ret = intel_guc_slpc_get_min_freq(slpc, &slpc_min_freq); + if (ret) { + drm_err(&i915->drm, + "Failed to get min freq: (%d)\n", + ret); + return false; + } + + if (slpc_min_freq == SLPC_MAX_FREQ_MHZ) + return true; + else + return false; +} + +static void update_server_min_softlimit(struct intel_guc_slpc *slpc) +{ + /* For server parts, SLPC min will be at RPMax. + * Use min softlimit to clamp it to RP0 instead. + */ + if (!slpc->min_freq_softlimit && + is_slpc_min_freq_rpmax(slpc)) { + slpc->min_is_rpmax = true; + slpc->min_freq_softlimit = slpc->rp0_freq; + (slpc_to_gt(slpc))->defaults.min_freq = slpc->min_freq_softlimit; + } +} + static int slpc_use_fused_rp0(struct intel_guc_slpc *slpc) { /* Force SLPC to used platform rp0 */ @@ -610,6 +664,52 @@ static void slpc_get_rp_values(struct intel_guc_slpc *slpc) slpc->boost_freq = slpc->rp0_freq; } +/** + * intel_guc_slpc_override_gucrc_mode() - override GUCRC mode + * @slpc: pointer to intel_guc_slpc. + * @mode: new value of the mode. + * + * This function will override the GUCRC mode. + * + * Return: 0 on success, non-zero error code on failure. + */ +int intel_guc_slpc_override_gucrc_mode(struct intel_guc_slpc *slpc, u32 mode) +{ + int ret; + struct drm_i915_private *i915 = slpc_to_i915(slpc); + intel_wakeref_t wakeref; + + if (mode >= SLPC_GUCRC_MODE_MAX) + return -EINVAL; + + with_intel_runtime_pm(&i915->runtime_pm, wakeref) { + ret = slpc_set_param(slpc, SLPC_PARAM_PWRGATE_RC_MODE, mode); + if (ret) + drm_err(&i915->drm, + "Override gucrc mode %d failed %d\n", + mode, ret); + } + + return ret; +} + +int intel_guc_slpc_unset_gucrc_mode(struct intel_guc_slpc *slpc) +{ + struct drm_i915_private *i915 = slpc_to_i915(slpc); + intel_wakeref_t wakeref; + int ret = 0; + + with_intel_runtime_pm(&i915->runtime_pm, wakeref) { + ret = slpc_unset_param(slpc, SLPC_PARAM_PWRGATE_RC_MODE); + if (ret) + drm_err(&i915->drm, + "Unsetting gucrc mode failed %d\n", + ret); + } + + return ret; +} + /* * intel_guc_slpc_enable() - Start SLPC * @slpc: pointer to intel_guc_slpc. @@ -647,6 +747,9 @@ int intel_guc_slpc_enable(struct intel_guc_slpc *slpc) slpc_get_rp_values(slpc); + /* Handle the case where min=max=RPmax */ + update_server_min_softlimit(slpc); + /* Set SLPC max limit to RP0 */ ret = slpc_use_fused_rp0(slpc); if (unlikely(ret)) { diff --git a/drivers/gpu/drm/i915/gt/uc/intel_guc_slpc.h b/drivers/gpu/drm/i915/gt/uc/intel_guc_slpc.h index 82a98f78f96c..17ed515f6a85 100644 --- a/drivers/gpu/drm/i915/gt/uc/intel_guc_slpc.h +++ b/drivers/gpu/drm/i915/gt/uc/intel_guc_slpc.h @@ -9,6 +9,8 @@ #include "intel_guc_submission.h" #include "intel_guc_slpc_types.h" +#define SLPC_MAX_FREQ_MHZ 4250 + struct intel_gt; struct drm_printer; @@ -42,5 +44,7 @@ int intel_guc_slpc_set_media_ratio_mode(struct intel_guc_slpc *slpc, u32 val); void intel_guc_pm_intrmsk_enable(struct intel_gt *gt); void intel_guc_slpc_boost(struct intel_guc_slpc *slpc); void intel_guc_slpc_dec_waiters(struct intel_guc_slpc *slpc); +int intel_guc_slpc_unset_gucrc_mode(struct intel_guc_slpc *slpc); +int intel_guc_slpc_override_gucrc_mode(struct intel_guc_slpc *slpc, u32 mode); #endif diff --git a/drivers/gpu/drm/i915/gt/uc/intel_guc_slpc_types.h b/drivers/gpu/drm/i915/gt/uc/intel_guc_slpc_types.h index 73d208123528..a6ef53b04e04 100644 --- a/drivers/gpu/drm/i915/gt/uc/intel_guc_slpc_types.h +++ b/drivers/gpu/drm/i915/gt/uc/intel_guc_slpc_types.h @@ -19,6 +19,9 @@ struct intel_guc_slpc { bool supported; bool selected; + /* Indicates this is a server part */ + bool min_is_rpmax; + /* platform frequency limits */ u32 min_freq; u32 rp0_freq; diff --git a/drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c b/drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c index 1db59eeb34db..0a42f1807f52 100644 --- a/drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c +++ b/drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c @@ -6,6 +6,7 @@ #include <linux/circ_buf.h> #include "gem/i915_gem_context.h" +#include "gem/i915_gem_lmem.h" #include "gt/gen8_engine_cs.h" #include "gt/intel_breadcrumbs.h" #include "gt/intel_context.h" @@ -29,6 +30,7 @@ #include "intel_guc_submission.h" #include "i915_drv.h" +#include "i915_reg.h" #include "i915_trace.h" /** @@ -65,7 +67,13 @@ * corresponding G2H returns indicating the scheduling disable operation has * completed it is safe to unpin the context. While a disable is in flight it * isn't safe to resubmit the context so a fence is used to stall all future - * requests of that context until the G2H is returned. + * requests of that context until the G2H is returned. Because this interaction + * with the GuC takes a non-zero amount of time we delay the disabling of + * scheduling after the pin count goes to zero by a configurable period of time + * (see SCHED_DISABLE_DELAY_MS). The thought is this gives the user a window of + * time to resubmit something on the context before doing this costly operation. + * This delay is only done if the context isn't closed and the guc_id usage is + * less than a threshold (see NUM_SCHED_DISABLE_GUC_IDS_THRESHOLD). * * Context deregistration: * Before a context can be destroyed or if we steal its guc_id we must @@ -163,7 +171,8 @@ guc_create_parallel(struct intel_engine_cs **engines, #define SCHED_STATE_PENDING_ENABLE BIT(5) #define SCHED_STATE_REGISTERED BIT(6) #define SCHED_STATE_POLICY_REQUIRED BIT(7) -#define SCHED_STATE_BLOCKED_SHIFT 8 +#define SCHED_STATE_CLOSED BIT(8) +#define SCHED_STATE_BLOCKED_SHIFT 9 #define SCHED_STATE_BLOCKED BIT(SCHED_STATE_BLOCKED_SHIFT) #define SCHED_STATE_BLOCKED_MASK (0xfff << SCHED_STATE_BLOCKED_SHIFT) @@ -173,12 +182,20 @@ static inline void init_sched_state(struct intel_context *ce) ce->guc_state.sched_state &= SCHED_STATE_BLOCKED_MASK; } +/* + * Kernel contexts can have SCHED_STATE_REGISTERED after suspend. + * A context close can race with the submission path, so SCHED_STATE_CLOSED + * can be set immediately before we try to register. + */ +#define SCHED_STATE_VALID_INIT \ + (SCHED_STATE_BLOCKED_MASK | \ + SCHED_STATE_CLOSED | \ + SCHED_STATE_REGISTERED) + __maybe_unused static bool sched_state_is_init(struct intel_context *ce) { - /* Kernel contexts can have SCHED_STATE_REGISTERED after suspend. */ - return !(ce->guc_state.sched_state & - ~(SCHED_STATE_BLOCKED_MASK | SCHED_STATE_REGISTERED)); + return !(ce->guc_state.sched_state & ~SCHED_STATE_VALID_INIT); } static inline bool @@ -319,6 +336,17 @@ static inline void clr_context_policy_required(struct intel_context *ce) ce->guc_state.sched_state &= ~SCHED_STATE_POLICY_REQUIRED; } +static inline bool context_close_done(struct intel_context *ce) +{ + return ce->guc_state.sched_state & SCHED_STATE_CLOSED; +} + +static inline void set_context_close_done(struct intel_context *ce) +{ + lockdep_assert_held(&ce->guc_state.lock); + ce->guc_state.sched_state |= SCHED_STATE_CLOSED; +} + static inline u32 context_blocked(struct intel_context *ce) { return (ce->guc_state.sched_state & SCHED_STATE_BLOCKED_MASK) >> @@ -343,25 +371,6 @@ static inline void decr_context_blocked(struct intel_context *ce) ce->guc_state.sched_state -= SCHED_STATE_BLOCKED; } -static inline bool context_has_committed_requests(struct intel_context *ce) -{ - return !!ce->guc_state.number_committed_requests; -} - -static inline void incr_context_committed_requests(struct intel_context *ce) -{ - lockdep_assert_held(&ce->guc_state.lock); - ++ce->guc_state.number_committed_requests; - GEM_BUG_ON(ce->guc_state.number_committed_requests < 0); -} - -static inline void decr_context_committed_requests(struct intel_context *ce) -{ - lockdep_assert_held(&ce->guc_state.lock); - --ce->guc_state.number_committed_requests; - GEM_BUG_ON(ce->guc_state.number_committed_requests < 0); -} - static struct intel_context * request_to_scheduling_context(struct i915_request *rq) { @@ -1067,6 +1076,12 @@ static void scrub_guc_desc_for_outstanding_g2h(struct intel_guc *guc) xa_unlock(&guc->context_lookup); + if (test_bit(CONTEXT_GUC_INIT, &ce->flags) && + (cancel_delayed_work(&ce->guc_state.sched_disable_delay_work))) { + /* successful cancel so jump straight to close it */ + intel_context_sched_disable_unpin(ce); + } + spin_lock(&ce->guc_state.lock); /* @@ -1387,7 +1402,9 @@ static void guc_timestamp_ping(struct work_struct *wrk) /* * Synchronize with gt reset to make sure the worker does not - * corrupt the engine/guc stats. + * corrupt the engine/guc stats. NB: can't actually block waiting + * for a reset to complete as the reset requires flushing out + * this worker thread if started. So waiting would deadlock. */ ret = intel_gt_reset_trylock(gt, &srcu); if (ret) @@ -1994,6 +2011,9 @@ static int new_guc_id(struct intel_guc *guc, struct intel_context *ce) if (unlikely(ret < 0)) return ret; + if (!intel_context_is_parent(ce)) + ++guc->submission_state.guc_ids_in_use; + ce->guc_id.id = ret; return 0; } @@ -2003,14 +2023,16 @@ static void __release_guc_id(struct intel_guc *guc, struct intel_context *ce) GEM_BUG_ON(intel_context_is_child(ce)); if (!context_guc_id_invalid(ce)) { - if (intel_context_is_parent(ce)) + if (intel_context_is_parent(ce)) { bitmap_release_region(guc->submission_state.guc_ids_bitmap, ce->guc_id.id, order_base_2(ce->parallel.number_children + 1)); - else + } else { + --guc->submission_state.guc_ids_in_use; ida_simple_remove(&guc->submission_state.guc_ids, ce->guc_id.id); + } clr_ctx_id_mapping(guc, ce->guc_id.id); set_context_guc_id_invalid(ce); } @@ -2429,6 +2451,10 @@ static int guc_context_policy_init_v70(struct intel_context *ce, bool loop) int ret; /* NB: For both of these, zero means disabled. */ + GEM_BUG_ON(overflows_type(engine->props.timeslice_duration_ms * 1000, + execution_quantum)); + GEM_BUG_ON(overflows_type(engine->props.preempt_timeout_ms * 1000, + preemption_timeout)); execution_quantum = engine->props.timeslice_duration_ms * 1000; preemption_timeout = engine->props.preempt_timeout_ms * 1000; @@ -2462,6 +2488,10 @@ static void guc_context_policy_init_v69(struct intel_engine_cs *engine, desc->policy_flags |= CONTEXT_POLICY_FLAG_PREEMPT_TO_IDLE_V69; /* NB: For both of these, zero means disabled. */ + GEM_BUG_ON(overflows_type(engine->props.timeslice_duration_ms * 1000, + desc->execution_quantum)); + GEM_BUG_ON(overflows_type(engine->props.preempt_timeout_ms * 1000, + desc->preemption_timeout)); desc->execution_quantum = engine->props.timeslice_duration_ms * 1000; desc->preemption_timeout = engine->props.preempt_timeout_ms * 1000; } @@ -2998,41 +3028,104 @@ guc_context_revoke(struct intel_context *ce, struct i915_request *rq, } } -static void guc_context_sched_disable(struct intel_context *ce) +static void do_sched_disable(struct intel_guc *guc, struct intel_context *ce, + unsigned long flags) + __releases(ce->guc_state.lock) { - struct intel_guc *guc = ce_to_guc(ce); - unsigned long flags; struct intel_runtime_pm *runtime_pm = &ce->engine->gt->i915->runtime_pm; intel_wakeref_t wakeref; u16 guc_id; + lockdep_assert_held(&ce->guc_state.lock); + guc_id = prep_context_pending_disable(ce); + + spin_unlock_irqrestore(&ce->guc_state.lock, flags); + + with_intel_runtime_pm(runtime_pm, wakeref) + __guc_context_sched_disable(guc, ce, guc_id); +} + +static bool bypass_sched_disable(struct intel_guc *guc, + struct intel_context *ce) +{ + lockdep_assert_held(&ce->guc_state.lock); GEM_BUG_ON(intel_context_is_child(ce)); + if (submission_disabled(guc) || context_guc_id_invalid(ce) || + !ctx_id_mapped(guc, ce->guc_id.id)) { + clr_context_enabled(ce); + return true; + } + + return !context_enabled(ce); +} + +static void __delay_sched_disable(struct work_struct *wrk) +{ + struct intel_context *ce = + container_of(wrk, typeof(*ce), guc_state.sched_disable_delay_work.work); + struct intel_guc *guc = ce_to_guc(ce); + unsigned long flags; + spin_lock_irqsave(&ce->guc_state.lock, flags); + if (bypass_sched_disable(guc, ce)) { + spin_unlock_irqrestore(&ce->guc_state.lock, flags); + intel_context_sched_disable_unpin(ce); + } else { + do_sched_disable(guc, ce, flags); + } +} + +static bool guc_id_pressure(struct intel_guc *guc, struct intel_context *ce) +{ /* - * We have to check if the context has been disabled by another thread, - * check if submssion has been disabled to seal a race with reset and - * finally check if any more requests have been committed to the - * context ensursing that a request doesn't slip through the - * 'context_pending_disable' fence. + * parent contexts are perma-pinned, if we are unpinning do schedule + * disable immediately. */ - if (unlikely(!context_enabled(ce) || submission_disabled(guc) || - context_has_committed_requests(ce))) { - clr_context_enabled(ce); + if (intel_context_is_parent(ce)) + return true; + + /* + * If we are beyond the threshold for avail guc_ids, do schedule disable immediately. + */ + return guc->submission_state.guc_ids_in_use > + guc->submission_state.sched_disable_gucid_threshold; +} + +static void guc_context_sched_disable(struct intel_context *ce) +{ + struct intel_guc *guc = ce_to_guc(ce); + u64 delay = guc->submission_state.sched_disable_delay_ms; + unsigned long flags; + + spin_lock_irqsave(&ce->guc_state.lock, flags); + + if (bypass_sched_disable(guc, ce)) { + spin_unlock_irqrestore(&ce->guc_state.lock, flags); + intel_context_sched_disable_unpin(ce); + } else if (!intel_context_is_closed(ce) && !guc_id_pressure(guc, ce) && + delay) { spin_unlock_irqrestore(&ce->guc_state.lock, flags); - goto unpin; + mod_delayed_work(system_unbound_wq, + &ce->guc_state.sched_disable_delay_work, + msecs_to_jiffies(delay)); + } else { + do_sched_disable(guc, ce, flags); } - guc_id = prep_context_pending_disable(ce); +} - spin_unlock_irqrestore(&ce->guc_state.lock, flags); +static void guc_context_close(struct intel_context *ce) +{ + unsigned long flags; - with_intel_runtime_pm(runtime_pm, wakeref) - __guc_context_sched_disable(guc, ce, guc_id); + if (test_bit(CONTEXT_GUC_INIT, &ce->flags) && + cancel_delayed_work(&ce->guc_state.sched_disable_delay_work)) + __delay_sched_disable(&ce->guc_state.sched_disable_delay_work.work); - return; -unpin: - intel_context_sched_disable_unpin(ce); + spin_lock_irqsave(&ce->guc_state.lock, flags); + set_context_close_done(ce); + spin_unlock_irqrestore(&ce->guc_state.lock, flags); } static inline void guc_lrc_desc_unpin(struct intel_context *ce) @@ -3071,7 +3164,6 @@ static void __guc_context_destroy(struct intel_context *ce) ce->guc_state.prio_count[GUC_CLIENT_PRIORITY_HIGH] || ce->guc_state.prio_count[GUC_CLIENT_PRIORITY_KMD_NORMAL] || ce->guc_state.prio_count[GUC_CLIENT_PRIORITY_NORMAL]); - GEM_BUG_ON(ce->guc_state.number_committed_requests); lrc_fini(ce); intel_context_fini(ce); @@ -3340,8 +3432,6 @@ static void remove_from_context(struct i915_request *rq) guc_prio_fini(rq, ce); - decr_context_committed_requests(ce); - spin_unlock_irq(&ce->guc_state.lock); atomic_dec(&ce->guc_id.ref); @@ -3351,6 +3441,8 @@ static void remove_from_context(struct i915_request *rq) static const struct intel_context_ops guc_context_ops = { .alloc = guc_context_alloc, + .close = guc_context_close, + .pre_pin = guc_context_pre_pin, .pin = guc_context_pin, .unpin = guc_context_unpin, @@ -3433,6 +3525,10 @@ static void guc_context_init(struct intel_context *ce) rcu_read_unlock(); ce->guc_state.prio = map_i915_prio_to_guc_prio(prio); + + INIT_DELAYED_WORK(&ce->guc_state.sched_disable_delay_work, + __delay_sched_disable); + set_bit(CONTEXT_GUC_INIT, &ce->flags); } @@ -3471,6 +3567,26 @@ static int guc_request_alloc(struct i915_request *rq) guc_context_init(ce); /* + * If the context gets closed while the execbuf is ongoing, the context + * close code will race with the below code to cancel the delayed work. + * If the context close wins the race and cancels the work, it will + * immediately call the sched disable (see guc_context_close), so there + * is a chance we can get past this check while the sched_disable code + * is being executed. To make sure that code completes before we check + * the status further down, we wait for the close process to complete. + * Else, this code path could send a request down thinking that the + * context is still in a schedule-enable mode while the GuC ends up + * dropping the request completely because the disable did go from the + * context_close path right to GuC just prior. In the event the CT is + * full, we could potentially need to wait up to 1.5 seconds. + */ + if (cancel_delayed_work_sync(&ce->guc_state.sched_disable_delay_work)) + intel_context_sched_disable_unpin(ce); + else if (intel_context_is_closed(ce)) + if (wait_for(context_close_done(ce), 1500)) + drm_warn(&guc_to_gt(guc)->i915->drm, + "timed out waiting on context sched close before realloc\n"); + /* * Call pin_guc_id here rather than in the pinning step as with * dma_resv, contexts can be repeatedly pinned / unpinned trashing the * guc_id and creating horrible race conditions. This is especially bad @@ -3524,7 +3640,6 @@ out: list_add_tail(&rq->guc_fence_link, &ce->guc_state.fences); } - incr_context_committed_requests(ce); spin_unlock_irqrestore(&ce->guc_state.lock, flags); return 0; @@ -3600,6 +3715,8 @@ static int guc_virtual_context_alloc(struct intel_context *ce) static const struct intel_context_ops virtual_guc_context_ops = { .alloc = guc_virtual_context_alloc, + .close = guc_context_close, + .pre_pin = guc_virtual_context_pre_pin, .pin = guc_virtual_context_pin, .unpin = guc_virtual_context_unpin, @@ -3689,6 +3806,8 @@ static void guc_child_context_destroy(struct kref *kref) static const struct intel_context_ops virtual_parent_context_ops = { .alloc = guc_virtual_context_alloc, + .close = guc_context_close, + .pre_pin = guc_context_pre_pin, .pin = guc_parent_context_pin, .unpin = guc_parent_context_unpin, @@ -3995,6 +4114,9 @@ static inline void guc_kernel_context_pin(struct intel_guc *guc, if (context_guc_id_invalid(ce)) pin_guc_id(guc, ce); + if (!test_bit(CONTEXT_GUC_INIT, &ce->flags)) + guc_context_init(ce); + try_context_registration(ce, true); } @@ -4093,7 +4215,7 @@ static void guc_default_vfuncs(struct intel_engine_cs *engine) engine->emit_bb_start = gen8_emit_bb_start; if (GRAPHICS_VER_FULL(engine->i915) >= IP_VER(12, 50)) - engine->emit_bb_start = gen125_emit_bb_start; + engine->emit_bb_start = xehp_emit_bb_start; } static void rcs_submission_override(struct intel_engine_cs *engine) @@ -4177,6 +4299,98 @@ int intel_guc_submission_setup(struct intel_engine_cs *engine) return 0; } +struct scheduling_policy { + /* internal data */ + u32 max_words, num_words; + u32 count; + /* API data */ + struct guc_update_scheduling_policy h2g; +}; + +static u32 __guc_scheduling_policy_action_size(struct scheduling_policy *policy) +{ + u32 *start = (void *)&policy->h2g; + u32 *end = policy->h2g.data + policy->num_words; + size_t delta = end - start; + + return delta; +} + +static struct scheduling_policy *__guc_scheduling_policy_start_klv(struct scheduling_policy *policy) +{ + policy->h2g.header.action = INTEL_GUC_ACTION_UPDATE_SCHEDULING_POLICIES_KLV; + policy->max_words = ARRAY_SIZE(policy->h2g.data); + policy->num_words = 0; + policy->count = 0; + + return policy; +} + +static void __guc_scheduling_policy_add_klv(struct scheduling_policy *policy, + u32 action, u32 *data, u32 len) +{ + u32 *klv_ptr = policy->h2g.data + policy->num_words; + + GEM_BUG_ON((policy->num_words + 1 + len) > policy->max_words); + *(klv_ptr++) = FIELD_PREP(GUC_KLV_0_KEY, action) | + FIELD_PREP(GUC_KLV_0_LEN, len); + memcpy(klv_ptr, data, sizeof(u32) * len); + policy->num_words += 1 + len; + policy->count++; +} + +static int __guc_action_set_scheduling_policies(struct intel_guc *guc, + struct scheduling_policy *policy) +{ + int ret; + + ret = intel_guc_send(guc, (u32 *)&policy->h2g, + __guc_scheduling_policy_action_size(policy)); + if (ret < 0) + return ret; + + if (ret != policy->count) { + drm_warn(&guc_to_gt(guc)->i915->drm, "GuC global scheduler policy processed %d of %d KLVs!", + ret, policy->count); + if (ret > policy->count) + return -EPROTO; + } + + return 0; +} + +static int guc_init_global_schedule_policy(struct intel_guc *guc) +{ + struct scheduling_policy policy; + struct intel_gt *gt = guc_to_gt(guc); + intel_wakeref_t wakeref; + int ret = 0; + + if (GET_UC_VER(guc) < MAKE_UC_VER(70, 3, 0)) + return 0; + + __guc_scheduling_policy_start_klv(&policy); + + with_intel_runtime_pm(>->i915->runtime_pm, wakeref) { + u32 yield[] = { + GLOBAL_SCHEDULE_POLICY_RC_YIELD_DURATION, + GLOBAL_SCHEDULE_POLICY_RC_YIELD_RATIO, + }; + + __guc_scheduling_policy_add_klv(&policy, + GUC_SCHEDULING_POLICIES_KLV_ID_RENDER_COMPUTE_YIELD, + yield, ARRAY_SIZE(yield)); + + ret = __guc_action_set_scheduling_policies(guc, &policy); + if (ret) + i915_probe_error(gt->i915, + "Failed to configure global scheduling policies: %pe!\n", + ERR_PTR(ret)); + } + + return ret; +} + void intel_guc_submission_enable(struct intel_guc *guc) { struct intel_gt *gt = guc_to_gt(guc); @@ -4189,6 +4403,7 @@ void intel_guc_submission_enable(struct intel_guc *guc) guc_init_lrc_mapping(guc); guc_init_engine_stats(guc); + guc_init_global_schedule_policy(guc); } void intel_guc_submission_disable(struct intel_guc *guc) @@ -4219,6 +4434,26 @@ static bool __guc_submission_selected(struct intel_guc *guc) return i915->params.enable_guc & ENABLE_GUC_SUBMISSION; } +int intel_guc_sched_disable_gucid_threshold_max(struct intel_guc *guc) +{ + return guc->submission_state.num_guc_ids - NUMBER_MULTI_LRC_GUC_ID(guc); +} + +/* + * This default value of 33 milisecs (+1 milisec round up) ensures 30fps or higher + * workloads are able to enjoy the latency reduction when delaying the schedule-disable + * operation. This matches the 30fps game-render + encode (real world) workload this + * knob was tested against. + */ +#define SCHED_DISABLE_DELAY_MS 34 + +/* + * A threshold of 75% is a reasonable starting point considering that real world apps + * generally don't get anywhere near this. + */ +#define NUM_SCHED_DISABLE_GUCIDS_DEFAULT_THRESHOLD(__guc) \ + (((intel_guc_sched_disable_gucid_threshold_max(guc)) * 3) / 4) + void intel_guc_submission_init_early(struct intel_guc *guc) { xa_init_flags(&guc->context_lookup, XA_FLAGS_LOCK_IRQ); @@ -4235,7 +4470,10 @@ void intel_guc_submission_init_early(struct intel_guc *guc) spin_lock_init(&guc->timestamp.lock); INIT_DELAYED_WORK(&guc->timestamp.work, guc_timestamp_ping); + guc->submission_state.sched_disable_delay_ms = SCHED_DISABLE_DELAY_MS; guc->submission_state.num_guc_ids = GUC_MAX_CONTEXT_ID; + guc->submission_state.sched_disable_gucid_threshold = + NUM_SCHED_DISABLE_GUCIDS_DEFAULT_THRESHOLD(guc); guc->submission_supported = __guc_submission_supported(guc); guc->submission_selected = __guc_submission_selected(guc); } @@ -4669,7 +4907,7 @@ void intel_guc_submission_print_info(struct intel_guc *guc, drm_printf(p, "GuC Number Outstanding Submission G2H: %u\n", atomic_read(&guc->outstanding_submission_g2h)); - drm_printf(p, "GuC tasklet count: %u\n\n", + drm_printf(p, "GuC tasklet count: %u\n", atomic_read(&sched_engine->tasklet.count)); spin_lock_irqsave(&sched_engine->lock, flags); @@ -4717,7 +4955,7 @@ static inline void guc_log_context(struct drm_printer *p, atomic_read(&ce->pin_count)); drm_printf(p, "\t\tGuC ID Ref Count: %u\n", atomic_read(&ce->guc_id.ref)); - drm_printf(p, "\t\tSchedule State: 0x%x\n\n", + drm_printf(p, "\t\tSchedule State: 0x%x\n", ce->guc_state.sched_state); } @@ -4746,7 +4984,7 @@ void intel_guc_submission_print_context_info(struct intel_guc *guc, READ_ONCE(*ce->parallel.guc.wq_head)); drm_printf(p, "\t\tWQI Tail: %u\n", READ_ONCE(*ce->parallel.guc.wq_tail)); - drm_printf(p, "\t\tWQI Status: %u\n\n", + drm_printf(p, "\t\tWQI Status: %u\n", READ_ONCE(*ce->parallel.guc.wq_status)); } @@ -4754,7 +4992,7 @@ void intel_guc_submission_print_context_info(struct intel_guc *guc, emit_bb_start_parent_no_preempt_mid_batch) { u8 i; - drm_printf(p, "\t\tChildren Go: %u\n\n", + drm_printf(p, "\t\tChildren Go: %u\n", get_children_go_value(ce)); for (i = 0; i < ce->parallel.number_children; ++i) drm_printf(p, "\t\tChildren Join: %u\n", diff --git a/drivers/gpu/drm/i915/gt/uc/intel_huc.c b/drivers/gpu/drm/i915/gt/uc/intel_huc.c index 3bb8838e325a..410905da8e97 100644 --- a/drivers/gpu/drm/i915/gt/uc/intel_huc.c +++ b/drivers/gpu/drm/i915/gt/uc/intel_huc.c @@ -10,6 +10,9 @@ #include "intel_huc.h" #include "i915_drv.h" +#include <linux/device/bus.h> +#include <linux/mei_aux.h> + /** * DOC: HuC * @@ -42,12 +45,240 @@ * HuC-specific commands. */ +/* + * MEI-GSC load is an async process. The probing of the exposed aux device + * (see intel_gsc.c) usually happens a few seconds after i915 probe, depending + * on when the kernel schedules it. Unless something goes terribly wrong, we're + * guaranteed for this to happen during boot, so the big timeout is a safety net + * that we never expect to need. + * MEI-PXP + HuC load usually takes ~300ms, but if the GSC needs to be resumed + * and/or reset, this can take longer. Note that the kernel might schedule + * other work between the i915 init/resume and the MEI one, which can add to + * the delay. + */ +#define GSC_INIT_TIMEOUT_MS 10000 +#define PXP_INIT_TIMEOUT_MS 5000 + +static int sw_fence_dummy_notify(struct i915_sw_fence *sf, + enum i915_sw_fence_notify state) +{ + return NOTIFY_DONE; +} + +static void __delayed_huc_load_complete(struct intel_huc *huc) +{ + if (!i915_sw_fence_done(&huc->delayed_load.fence)) + i915_sw_fence_complete(&huc->delayed_load.fence); +} + +static void delayed_huc_load_complete(struct intel_huc *huc) +{ + hrtimer_cancel(&huc->delayed_load.timer); + __delayed_huc_load_complete(huc); +} + +static void __gsc_init_error(struct intel_huc *huc) +{ + huc->delayed_load.status = INTEL_HUC_DELAYED_LOAD_ERROR; + __delayed_huc_load_complete(huc); +} + +static void gsc_init_error(struct intel_huc *huc) +{ + hrtimer_cancel(&huc->delayed_load.timer); + __gsc_init_error(huc); +} + +static void gsc_init_done(struct intel_huc *huc) +{ + hrtimer_cancel(&huc->delayed_load.timer); + + /* MEI-GSC init is done, now we wait for MEI-PXP to bind */ + huc->delayed_load.status = INTEL_HUC_WAITING_ON_PXP; + if (!i915_sw_fence_done(&huc->delayed_load.fence)) + hrtimer_start(&huc->delayed_load.timer, + ms_to_ktime(PXP_INIT_TIMEOUT_MS), + HRTIMER_MODE_REL); +} + +static enum hrtimer_restart huc_delayed_load_timer_callback(struct hrtimer *hrtimer) +{ + struct intel_huc *huc = container_of(hrtimer, struct intel_huc, delayed_load.timer); + + if (!intel_huc_is_authenticated(huc)) { + if (huc->delayed_load.status == INTEL_HUC_WAITING_ON_GSC) + drm_notice(&huc_to_gt(huc)->i915->drm, + "timed out waiting for MEI GSC init to load HuC\n"); + else if (huc->delayed_load.status == INTEL_HUC_WAITING_ON_PXP) + drm_notice(&huc_to_gt(huc)->i915->drm, + "timed out waiting for MEI PXP init to load HuC\n"); + else + MISSING_CASE(huc->delayed_load.status); + + __gsc_init_error(huc); + } + + return HRTIMER_NORESTART; +} + +static void huc_delayed_load_start(struct intel_huc *huc) +{ + ktime_t delay; + + GEM_BUG_ON(intel_huc_is_authenticated(huc)); + + /* + * On resume we don't have to wait for MEI-GSC to be re-probed, but we + * do need to wait for MEI-PXP to reset & re-bind + */ + switch (huc->delayed_load.status) { + case INTEL_HUC_WAITING_ON_GSC: + delay = ms_to_ktime(GSC_INIT_TIMEOUT_MS); + break; + case INTEL_HUC_WAITING_ON_PXP: + delay = ms_to_ktime(PXP_INIT_TIMEOUT_MS); + break; + default: + gsc_init_error(huc); + return; + } + + /* + * This fence is always complete unless we're waiting for the + * GSC device to come up to load the HuC. We arm the fence here + * and complete it when we confirm that the HuC is loaded from + * the PXP bind callback. + */ + GEM_BUG_ON(!i915_sw_fence_done(&huc->delayed_load.fence)); + i915_sw_fence_fini(&huc->delayed_load.fence); + i915_sw_fence_reinit(&huc->delayed_load.fence); + i915_sw_fence_await(&huc->delayed_load.fence); + i915_sw_fence_commit(&huc->delayed_load.fence); + + hrtimer_start(&huc->delayed_load.timer, delay, HRTIMER_MODE_REL); +} + +static int gsc_notifier(struct notifier_block *nb, unsigned long action, void *data) +{ + struct device *dev = data; + struct intel_huc *huc = container_of(nb, struct intel_huc, delayed_load.nb); + struct intel_gsc_intf *intf = &huc_to_gt(huc)->gsc.intf[0]; + + if (!intf->adev || &intf->adev->aux_dev.dev != dev) + return 0; + + switch (action) { + case BUS_NOTIFY_BOUND_DRIVER: /* mei driver bound to aux device */ + gsc_init_done(huc); + break; + + case BUS_NOTIFY_DRIVER_NOT_BOUND: /* mei driver fails to be bound */ + case BUS_NOTIFY_UNBIND_DRIVER: /* mei driver about to be unbound */ + drm_info(&huc_to_gt(huc)->i915->drm, + "mei driver not bound, disabling HuC load\n"); + gsc_init_error(huc); + break; + } + + return 0; +} + +void intel_huc_register_gsc_notifier(struct intel_huc *huc, struct bus_type *bus) +{ + int ret; + + if (!intel_huc_is_loaded_by_gsc(huc)) + return; + + huc->delayed_load.nb.notifier_call = gsc_notifier; + ret = bus_register_notifier(bus, &huc->delayed_load.nb); + if (ret) { + drm_err(&huc_to_gt(huc)->i915->drm, + "failed to register GSC notifier\n"); + huc->delayed_load.nb.notifier_call = NULL; + gsc_init_error(huc); + } +} + +void intel_huc_unregister_gsc_notifier(struct intel_huc *huc, struct bus_type *bus) +{ + if (!huc->delayed_load.nb.notifier_call) + return; + + delayed_huc_load_complete(huc); + + bus_unregister_notifier(bus, &huc->delayed_load.nb); + huc->delayed_load.nb.notifier_call = NULL; +} + +static void delayed_huc_load_init(struct intel_huc *huc) +{ + /* + * Initialize fence to be complete as this is expected to be complete + * unless there is a delayed HuC load in progress. + */ + i915_sw_fence_init(&huc->delayed_load.fence, + sw_fence_dummy_notify); + i915_sw_fence_commit(&huc->delayed_load.fence); + + hrtimer_init(&huc->delayed_load.timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL); + huc->delayed_load.timer.function = huc_delayed_load_timer_callback; +} + +static void delayed_huc_load_fini(struct intel_huc *huc) +{ + /* + * the fence is initialized in init_early, so we need to clean it up + * even if HuC loading is off. + */ + delayed_huc_load_complete(huc); + i915_sw_fence_fini(&huc->delayed_load.fence); +} + +static bool vcs_supported(struct intel_gt *gt) +{ + intel_engine_mask_t mask = gt->info.engine_mask; + + /* + * We reach here from i915_driver_early_probe for the primary GT before + * its engine mask is set, so we use the device info engine mask for it; + * this means we're not taking VCS fusing into account, but if the + * primary GT supports VCS engines we expect at least one of them to + * remain unfused so we're fine. + * For other GTs we expect the GT-specific mask to be set before we + * call this function. + */ + GEM_BUG_ON(!gt_is_root(gt) && !gt->info.engine_mask); + + if (gt_is_root(gt)) + mask = RUNTIME_INFO(gt->i915)->platform_engine_mask; + else + mask = gt->info.engine_mask; + + return __ENGINE_INSTANCES_MASK(mask, VCS0, I915_MAX_VCS); +} + void intel_huc_init_early(struct intel_huc *huc) { struct drm_i915_private *i915 = huc_to_gt(huc)->i915; + struct intel_gt *gt = huc_to_gt(huc); intel_uc_fw_init_early(&huc->fw, INTEL_UC_FW_TYPE_HUC); + /* + * we always init the fence as already completed, even if HuC is not + * supported. This way we don't have to distinguish between HuC not + * supported/disabled or already loaded, and can focus on if the load + * is currently in progress (fence not complete) or not, which is what + * we care about for stalling userspace submissions. + */ + delayed_huc_load_init(huc); + + if (!vcs_supported(gt)) { + intel_uc_fw_change_status(&huc->fw, INTEL_UC_FIRMWARE_NOT_SUPPORTED); + return; + } + if (GRAPHICS_VER(i915) >= 11) { huc->status.reg = GEN11_HUC_KERNEL_LOAD_INFO; huc->status.mask = HUC_LOAD_SUCCESSFUL; @@ -113,16 +344,59 @@ int intel_huc_init(struct intel_huc *huc) return 0; out: + intel_uc_fw_change_status(&huc->fw, INTEL_UC_FIRMWARE_INIT_FAIL); drm_info(&i915->drm, "HuC init failed with %d\n", err); return err; } void intel_huc_fini(struct intel_huc *huc) { + /* + * the fence is initialized in init_early, so we need to clean it up + * even if HuC loading is off. + */ + delayed_huc_load_fini(huc); + + if (intel_uc_fw_is_loadable(&huc->fw)) + intel_uc_fw_fini(&huc->fw); +} + +void intel_huc_suspend(struct intel_huc *huc) +{ if (!intel_uc_fw_is_loadable(&huc->fw)) return; - intel_uc_fw_fini(&huc->fw); + /* + * in the unlikely case that we're suspending before the GSC has + * completed its loading sequence, just stop waiting. We'll restart + * on resume. + */ + delayed_huc_load_complete(huc); +} + +int intel_huc_wait_for_auth_complete(struct intel_huc *huc) +{ + struct intel_gt *gt = huc_to_gt(huc); + int ret; + + ret = __intel_wait_for_register(gt->uncore, + huc->status.reg, + huc->status.mask, + huc->status.value, + 2, 50, NULL); + + /* mark the load process as complete even if the wait failed */ + delayed_huc_load_complete(huc); + + if (ret) { + drm_err(>->i915->drm, "HuC: Firmware not verified %d\n", ret); + intel_uc_fw_change_status(&huc->fw, INTEL_UC_FIRMWARE_LOAD_FAIL); + return ret; + } + + intel_uc_fw_change_status(&huc->fw, INTEL_UC_FIRMWARE_RUNNING); + drm_info(>->i915->drm, "HuC authenticated\n"); + return 0; } /** @@ -161,27 +435,18 @@ int intel_huc_auth(struct intel_huc *huc) } /* Check authentication status, it should be done by now */ - ret = __intel_wait_for_register(gt->uncore, - huc->status.reg, - huc->status.mask, - huc->status.value, - 2, 50, NULL); - if (ret) { - DRM_ERROR("HuC: Firmware not verified %d\n", ret); + ret = intel_huc_wait_for_auth_complete(huc); + if (ret) goto fail; - } - intel_uc_fw_change_status(&huc->fw, INTEL_UC_FIRMWARE_RUNNING); - drm_info(>->i915->drm, "HuC authenticated\n"); return 0; fail: i915_probe_error(gt->i915, "HuC: Authentication failed %d\n", ret); - intel_uc_fw_change_status(&huc->fw, INTEL_UC_FIRMWARE_LOAD_FAIL); return ret; } -static bool huc_is_authenticated(struct intel_huc *huc) +bool intel_huc_is_authenticated(struct intel_huc *huc) { struct intel_gt *gt = huc_to_gt(huc); intel_wakeref_t wakeref; @@ -200,13 +465,8 @@ static bool huc_is_authenticated(struct intel_huc *huc) * This function reads status register to verify if HuC * firmware was successfully loaded. * - * Returns: - * * -ENODEV if HuC is not present on this platform, - * * -EOPNOTSUPP if HuC firmware is disabled, - * * -ENOPKG if HuC firmware was not installed, - * * -ENOEXEC if HuC firmware is invalid or mismatched, - * * 0 if HuC firmware is not running, - * * 1 if HuC firmware is authenticated and running. + * The return values match what is expected for the I915_PARAM_HUC_STATUS + * getparam. */ int intel_huc_check_status(struct intel_huc *huc) { @@ -219,11 +479,21 @@ int intel_huc_check_status(struct intel_huc *huc) return -ENOPKG; case INTEL_UC_FIRMWARE_ERROR: return -ENOEXEC; + case INTEL_UC_FIRMWARE_INIT_FAIL: + return -ENOMEM; + case INTEL_UC_FIRMWARE_LOAD_FAIL: + return -EIO; default: break; } - return huc_is_authenticated(huc); + return intel_huc_is_authenticated(huc); +} + +static bool huc_has_delayed_load(struct intel_huc *huc) +{ + return intel_huc_is_loaded_by_gsc(huc) && + (huc->delayed_load.status != INTEL_HUC_DELAYED_LOAD_ERROR); } void intel_huc_update_auth_status(struct intel_huc *huc) @@ -231,9 +501,11 @@ void intel_huc_update_auth_status(struct intel_huc *huc) if (!intel_uc_fw_is_loadable(&huc->fw)) return; - if (huc_is_authenticated(huc)) + if (intel_huc_is_authenticated(huc)) intel_uc_fw_change_status(&huc->fw, INTEL_UC_FIRMWARE_RUNNING); + else if (huc_has_delayed_load(huc)) + huc_delayed_load_start(huc); } /** diff --git a/drivers/gpu/drm/i915/gt/uc/intel_huc.h b/drivers/gpu/drm/i915/gt/uc/intel_huc.h index d7e25b6e879e..52db03620c60 100644 --- a/drivers/gpu/drm/i915/gt/uc/intel_huc.h +++ b/drivers/gpu/drm/i915/gt/uc/intel_huc.h @@ -7,9 +7,21 @@ #define _INTEL_HUC_H_ #include "i915_reg_defs.h" +#include "i915_sw_fence.h" #include "intel_uc_fw.h" #include "intel_huc_fw.h" +#include <linux/notifier.h> +#include <linux/hrtimer.h> + +struct bus_type; + +enum intel_huc_delayed_load_status { + INTEL_HUC_WAITING_ON_GSC = 0, + INTEL_HUC_WAITING_ON_PXP, + INTEL_HUC_DELAYED_LOAD_ERROR, +}; + struct intel_huc { /* Generic uC firmware management */ struct intel_uc_fw fw; @@ -20,14 +32,27 @@ struct intel_huc { u32 mask; u32 value; } status; + + struct { + struct i915_sw_fence fence; + struct hrtimer timer; + struct notifier_block nb; + enum intel_huc_delayed_load_status status; + } delayed_load; }; void intel_huc_init_early(struct intel_huc *huc); int intel_huc_init(struct intel_huc *huc); void intel_huc_fini(struct intel_huc *huc); +void intel_huc_suspend(struct intel_huc *huc); int intel_huc_auth(struct intel_huc *huc); +int intel_huc_wait_for_auth_complete(struct intel_huc *huc); int intel_huc_check_status(struct intel_huc *huc); void intel_huc_update_auth_status(struct intel_huc *huc); +bool intel_huc_is_authenticated(struct intel_huc *huc); + +void intel_huc_register_gsc_notifier(struct intel_huc *huc, struct bus_type *bus); +void intel_huc_unregister_gsc_notifier(struct intel_huc *huc, struct bus_type *bus); static inline int intel_huc_sanitize(struct intel_huc *huc) { @@ -56,6 +81,12 @@ static inline bool intel_huc_is_loaded_by_gsc(const struct intel_huc *huc) return huc->fw.loaded_via_gsc; } +static inline bool intel_huc_wait_required(struct intel_huc *huc) +{ + return intel_huc_is_used(huc) && intel_huc_is_loaded_by_gsc(huc) && + !intel_huc_is_authenticated(huc); +} + void intel_huc_load_status(struct intel_huc *huc, struct drm_printer *p); #endif diff --git a/drivers/gpu/drm/i915/gt/uc/intel_huc_fw.c b/drivers/gpu/drm/i915/gt/uc/intel_huc_fw.c index 9d6ab1e01639..4f246416db17 100644 --- a/drivers/gpu/drm/i915/gt/uc/intel_huc_fw.c +++ b/drivers/gpu/drm/i915/gt/uc/intel_huc_fw.c @@ -3,9 +3,43 @@ * Copyright © 2014-2019 Intel Corporation */ +#include "gt/intel_gsc.h" #include "gt/intel_gt.h" +#include "intel_huc.h" #include "intel_huc_fw.h" #include "i915_drv.h" +#include "pxp/intel_pxp_huc.h" + +int intel_huc_fw_load_and_auth_via_gsc(struct intel_huc *huc) +{ + int ret; + + if (!intel_huc_is_loaded_by_gsc(huc)) + return -ENODEV; + + if (!intel_uc_fw_is_loadable(&huc->fw)) + return -ENOEXEC; + + /* + * If we abort a suspend, HuC might still be loaded when the mei + * component gets re-bound and this function called again. If so, just + * mark the HuC as loaded. + */ + if (intel_huc_is_authenticated(huc)) { + intel_uc_fw_change_status(&huc->fw, INTEL_UC_FIRMWARE_RUNNING); + return 0; + } + + GEM_WARN_ON(intel_uc_fw_is_loaded(&huc->fw)); + + ret = intel_pxp_huc_load_and_auth(&huc_to_gt(huc)->pxp); + if (ret) + return ret; + + intel_uc_fw_change_status(&huc->fw, INTEL_UC_FIRMWARE_TRANSFERRED); + + return intel_huc_wait_for_auth_complete(huc); +} /** * intel_huc_fw_upload() - load HuC uCode to device via DMA transfer diff --git a/drivers/gpu/drm/i915/gt/uc/intel_huc_fw.h b/drivers/gpu/drm/i915/gt/uc/intel_huc_fw.h index 12f264ee3e0b..db42e238b45f 100644 --- a/drivers/gpu/drm/i915/gt/uc/intel_huc_fw.h +++ b/drivers/gpu/drm/i915/gt/uc/intel_huc_fw.h @@ -8,6 +8,7 @@ struct intel_huc; +int intel_huc_fw_load_and_auth_via_gsc(struct intel_huc *huc); int intel_huc_fw_upload(struct intel_huc *huc); #endif diff --git a/drivers/gpu/drm/i915/gt/uc/intel_uc.c b/drivers/gpu/drm/i915/gt/uc/intel_uc.c index dbd048b77e19..2a508b137e90 100644 --- a/drivers/gpu/drm/i915/gt/uc/intel_uc.c +++ b/drivers/gpu/drm/i915/gt/uc/intel_uc.c @@ -357,8 +357,8 @@ static int uc_init_wopcm(struct intel_uc *uc) { struct intel_gt *gt = uc_to_gt(uc); struct intel_uncore *uncore = gt->uncore; - u32 base = intel_wopcm_guc_base(>->i915->wopcm); - u32 size = intel_wopcm_guc_size(>->i915->wopcm); + u32 base = intel_wopcm_guc_base(>->wopcm); + u32 size = intel_wopcm_guc_size(>->wopcm); u32 huc_agent = intel_uc_uses_huc(uc) ? HUC_LOADING_AGENT_GUC : 0; u32 mask; int err; @@ -636,8 +636,10 @@ void intel_uc_runtime_suspend(struct intel_uc *uc) { struct intel_guc *guc = &uc->guc; - if (!intel_guc_is_ready(guc)) + if (!intel_guc_is_ready(guc)) { + guc->interrupts.enabled = false; return; + } /* * Wait for any outstanding CTB before tearing down communication /w the @@ -657,8 +659,10 @@ void intel_uc_suspend(struct intel_uc *uc) intel_wakeref_t wakeref; int err; - if (!intel_guc_is_ready(guc)) + if (!intel_guc_is_ready(guc)) { + guc->interrupts.enabled = false; return; + } with_intel_runtime_pm(&uc_to_gt(uc)->i915->runtime_pm, wakeref) { err = intel_guc_suspend(guc); @@ -718,6 +722,7 @@ int intel_uc_runtime_resume(struct intel_uc *uc) static const struct intel_uc_ops uc_ops_off = { .init_hw = __uc_check_hw, + .fini = __uc_fini, /* to clean-up the init_early initialization */ }; static const struct intel_uc_ops uc_ops_on = { diff --git a/drivers/gpu/drm/i915/gt/uc/intel_uc_fw.c b/drivers/gpu/drm/i915/gt/uc/intel_uc_fw.c index b91ad4aede1f..0c80ba51a4bd 100644 --- a/drivers/gpu/drm/i915/gt/uc/intel_uc_fw.c +++ b/drivers/gpu/drm/i915/gt/uc/intel_uc_fw.c @@ -93,7 +93,8 @@ void intel_uc_fw_change_status(struct intel_uc_fw *uc_fw, fw_def(BROXTON, 0, guc_mmp(bxt, 70, 1, 1)) \ fw_def(SKYLAKE, 0, guc_mmp(skl, 70, 1, 1)) -#define INTEL_HUC_FIRMWARE_DEFS(fw_def, huc_raw, huc_mmp) \ +#define INTEL_HUC_FIRMWARE_DEFS(fw_def, huc_raw, huc_mmp, huc_gsc) \ + fw_def(DG2, 0, huc_gsc(dg2)) \ fw_def(ALDERLAKE_P, 0, huc_raw(tgl)) \ fw_def(ALDERLAKE_P, 0, huc_mmp(tgl, 7, 9, 3)) \ fw_def(ALDERLAKE_S, 0, huc_raw(tgl)) \ @@ -141,6 +142,9 @@ void intel_uc_fw_change_status(struct intel_uc_fw *uc_fw, #define MAKE_HUC_FW_PATH_BLANK(prefix_) \ __MAKE_UC_FW_PATH_BLANK(prefix_, "_huc") +#define MAKE_HUC_FW_PATH_GSC(prefix_) \ + __MAKE_UC_FW_PATH_BLANK(prefix_, "_huc_gsc") + #define MAKE_HUC_FW_PATH_MMP(prefix_, major_, minor_, patch_) \ __MAKE_UC_FW_PATH_MMP(prefix_, "_huc_", major_, minor_, patch_) @@ -153,7 +157,7 @@ void intel_uc_fw_change_status(struct intel_uc_fw *uc_fw, MODULE_FIRMWARE(uc_); INTEL_GUC_FIRMWARE_DEFS(INTEL_UC_MODULE_FW, MAKE_GUC_FW_PATH_MAJOR, MAKE_GUC_FW_PATH_MMP) -INTEL_HUC_FIRMWARE_DEFS(INTEL_UC_MODULE_FW, MAKE_HUC_FW_PATH_BLANK, MAKE_HUC_FW_PATH_MMP) +INTEL_HUC_FIRMWARE_DEFS(INTEL_UC_MODULE_FW, MAKE_HUC_FW_PATH_BLANK, MAKE_HUC_FW_PATH_MMP, MAKE_HUC_FW_PATH_GSC) /* * The next expansion of the table macros (in __uc_fw_auto_select below) provides @@ -168,6 +172,7 @@ struct __packed uc_fw_blob { u8 major; u8 minor; u8 patch; + bool loaded_via_gsc; }; #define UC_FW_BLOB_BASE(major_, minor_, patch_, path_) \ @@ -176,16 +181,16 @@ struct __packed uc_fw_blob { .patch = patch_, \ .path = path_, -#define UC_FW_BLOB_NEW(major_, minor_, patch_, path_) \ +#define UC_FW_BLOB_NEW(major_, minor_, patch_, gsc_, path_) \ { UC_FW_BLOB_BASE(major_, minor_, patch_, path_) \ - .legacy = false } + .legacy = false, .loaded_via_gsc = gsc_ } #define UC_FW_BLOB_OLD(major_, minor_, patch_, path_) \ { UC_FW_BLOB_BASE(major_, minor_, patch_, path_) \ .legacy = true } #define GUC_FW_BLOB(prefix_, major_, minor_) \ - UC_FW_BLOB_NEW(major_, minor_, 0, \ + UC_FW_BLOB_NEW(major_, minor_, 0, false, \ MAKE_GUC_FW_PATH_MAJOR(prefix_, major_, minor_)) #define GUC_FW_BLOB_MMP(prefix_, major_, minor_, patch_) \ @@ -193,12 +198,15 @@ struct __packed uc_fw_blob { MAKE_GUC_FW_PATH_MMP(prefix_, major_, minor_, patch_)) #define HUC_FW_BLOB(prefix_) \ - UC_FW_BLOB_NEW(0, 0, 0, MAKE_HUC_FW_PATH_BLANK(prefix_)) + UC_FW_BLOB_NEW(0, 0, 0, false, MAKE_HUC_FW_PATH_BLANK(prefix_)) #define HUC_FW_BLOB_MMP(prefix_, major_, minor_, patch_) \ UC_FW_BLOB_OLD(major_, minor_, patch_, \ MAKE_HUC_FW_PATH_MMP(prefix_, major_, minor_, patch_)) +#define HUC_FW_BLOB_GSC(prefix_) \ + UC_FW_BLOB_NEW(0, 0, 0, true, MAKE_HUC_FW_PATH_GSC(prefix_)) + struct __packed uc_fw_platform_requirement { enum intel_platform p; u8 rev; /* first platform rev using this FW */ @@ -224,7 +232,7 @@ __uc_fw_auto_select(struct drm_i915_private *i915, struct intel_uc_fw *uc_fw) INTEL_GUC_FIRMWARE_DEFS(MAKE_FW_LIST, GUC_FW_BLOB, GUC_FW_BLOB_MMP) }; static const struct uc_fw_platform_requirement blobs_huc[] = { - INTEL_HUC_FIRMWARE_DEFS(MAKE_FW_LIST, HUC_FW_BLOB, HUC_FW_BLOB_MMP) + INTEL_HUC_FIRMWARE_DEFS(MAKE_FW_LIST, HUC_FW_BLOB, HUC_FW_BLOB_MMP, HUC_FW_BLOB_GSC) }; static const struct fw_blobs_by_type blobs_all[INTEL_UC_FW_NUM_TYPES] = { [INTEL_UC_FW_TYPE_GUC] = { blobs_guc, ARRAY_SIZE(blobs_guc) }, @@ -272,6 +280,7 @@ __uc_fw_auto_select(struct drm_i915_private *i915, struct intel_uc_fw *uc_fw) uc_fw->file_wanted.path = blob->path; uc_fw->file_wanted.major_ver = blob->major; uc_fw->file_wanted.minor_ver = blob->minor; + uc_fw->loaded_via_gsc = blob->loaded_via_gsc; found = true; break; } @@ -469,10 +478,11 @@ static int check_gsc_manifest(const struct firmware *fw, return 0; } -static int check_ccs_header(struct drm_i915_private *i915, +static int check_ccs_header(struct intel_gt *gt, const struct firmware *fw, struct intel_uc_fw *uc_fw) { + struct drm_i915_private *i915 = gt->i915; struct uc_css_header *css; size_t size; @@ -514,10 +524,10 @@ static int check_ccs_header(struct drm_i915_private *i915, /* Sanity check whether this fw is not larger than whole WOPCM memory */ size = __intel_uc_fw_get_upload_size(uc_fw); - if (unlikely(size >= i915->wopcm.size)) { + if (unlikely(size >= gt->wopcm.size)) { drm_warn(&i915->drm, "%s firmware %s: invalid size: %zu > %zu\n", intel_uc_fw_type_repr(uc_fw->type), uc_fw->file_selected.path, - size, (size_t)i915->wopcm.size); + size, (size_t)gt->wopcm.size); return -E2BIG; } @@ -545,7 +555,8 @@ static int check_ccs_header(struct drm_i915_private *i915, */ int intel_uc_fw_fetch(struct intel_uc_fw *uc_fw) { - struct drm_i915_private *i915 = __uc_fw_to_gt(uc_fw)->i915; + struct intel_gt *gt = __uc_fw_to_gt(uc_fw); + struct drm_i915_private *i915 = gt->i915; struct intel_uc_fw_file file_ideal; struct device *dev = i915->drm.dev; struct drm_i915_gem_object *obj; @@ -553,7 +564,7 @@ int intel_uc_fw_fetch(struct intel_uc_fw *uc_fw) bool old_ver = false; int err; - GEM_BUG_ON(!i915->wopcm.size); + GEM_BUG_ON(!gt->wopcm.size); GEM_BUG_ON(!intel_uc_fw_is_enabled(uc_fw)); err = i915_inject_probe_error(i915, -ENXIO); @@ -566,6 +577,17 @@ int intel_uc_fw_fetch(struct intel_uc_fw *uc_fw) err = firmware_request_nowarn(&fw, uc_fw->file_selected.path, dev); memcpy(&file_ideal, &uc_fw->file_wanted, sizeof(file_ideal)); + if (!err && fw->size > INTEL_UC_RSVD_GGTT_PER_FW) { + drm_err(&i915->drm, + "%s firmware %s: size (%zuKB) exceeds max supported size (%uKB)\n", + intel_uc_fw_type_repr(uc_fw->type), uc_fw->file_selected.path, + fw->size / SZ_1K, INTEL_UC_RSVD_GGTT_PER_FW / SZ_1K); + + /* try to find another blob to load */ + release_firmware(fw); + err = -ENOENT; + } + /* Any error is terminal if overriding. Don't bother searching for older versions */ if (err && intel_uc_fw_is_overridden(uc_fw)) goto fail; @@ -595,7 +617,7 @@ int intel_uc_fw_fetch(struct intel_uc_fw *uc_fw) if (uc_fw->loaded_via_gsc) err = check_gsc_manifest(fw, uc_fw); else - err = check_ccs_header(i915, fw, uc_fw); + err = check_ccs_header(gt, fw, uc_fw); if (err) goto fail; @@ -668,14 +690,30 @@ fail: static u32 uc_fw_ggtt_offset(struct intel_uc_fw *uc_fw) { - struct i915_ggtt *ggtt = __uc_fw_to_gt(uc_fw)->ggtt; + struct intel_gt *gt = __uc_fw_to_gt(uc_fw); + struct i915_ggtt *ggtt = gt->ggtt; struct drm_mm_node *node = &ggtt->uc_fw; + u32 offset = uc_fw->type * INTEL_UC_RSVD_GGTT_PER_FW; + + /* + * The media GT shares the GGTT with the root GT, which means that + * we need to use different offsets for the binaries on the media GT. + * To keep the math simple, we use 8MB for the root tile and 8MB for + * the media one. This will need to be updated if we ever have more + * than 1 media GT. + */ + BUILD_BUG_ON(INTEL_UC_FW_NUM_TYPES * INTEL_UC_RSVD_GGTT_PER_FW > SZ_8M); + GEM_BUG_ON(gt->type == GT_MEDIA && gt->info.id > 1); + if (gt->type == GT_MEDIA) + offset += SZ_8M; GEM_BUG_ON(!drm_mm_node_allocated(node)); GEM_BUG_ON(upper_32_bits(node->start)); GEM_BUG_ON(upper_32_bits(node->start + node->size - 1)); + GEM_BUG_ON(offset + uc_fw->obj->base.size > node->size); + GEM_BUG_ON(uc_fw->obj->base.size > INTEL_UC_RSVD_GGTT_PER_FW); - return lower_32_bits(node->start); + return lower_32_bits(node->start + offset); } static void uc_fw_bind_ggtt(struct intel_uc_fw *uc_fw) @@ -690,7 +728,6 @@ static void uc_fw_bind_ggtt(struct intel_uc_fw *uc_fw) dummy->bi.pages = obj->mm.pages; GEM_BUG_ON(!i915_gem_object_has_pinned_pages(obj)); - GEM_BUG_ON(dummy->node_size > ggtt->uc_fw.size); /* uc_fw->obj cache domains were not controlled across suspend */ if (i915_gem_object_has_struct_page(obj)) @@ -904,7 +941,6 @@ int intel_uc_fw_init(struct intel_uc_fw *uc_fw) out_unpin: i915_gem_object_unpin_pages(uc_fw->obj); out: - intel_uc_fw_change_status(uc_fw, INTEL_UC_FIRMWARE_INIT_FAIL); return err; } diff --git a/drivers/gpu/drm/i915/gt/uc/intel_uc_fw.h b/drivers/gpu/drm/i915/gt/uc/intel_uc_fw.h index cb586f7df270..bc898ba5355d 100644 --- a/drivers/gpu/drm/i915/gt/uc/intel_uc_fw.h +++ b/drivers/gpu/drm/i915/gt/uc/intel_uc_fw.h @@ -6,6 +6,7 @@ #ifndef _INTEL_UC_FW_H_ #define _INTEL_UC_FW_H_ +#include <linux/sizes.h> #include <linux/types.h> #include "intel_uc_fw_abi.h" #include "intel_device_info.h" @@ -114,6 +115,19 @@ struct intel_uc_fw { (uc)->fw.file_selected.minor_ver, \ (uc)->fw.file_selected.patch_ver)) +/* + * When we load the uC binaries, we pin them in a reserved section at the top of + * the GGTT, which is ~18 MBs. On multi-GT systems where the GTs share the GGTT, + * we also need to make sure that each binary is pinned to a unique location + * during load, because the different GT can go through the FW load at the same + * time (see uc_fw_ggtt_offset() for details). + * Given that the available space is much greater than what is required by the + * binaries, to keep things simple instead of dynamically partitioning the + * reserved section to make space for all the blobs we can just reserve a static + * chunk for each binary. + */ +#define INTEL_UC_RSVD_GGTT_PER_FW SZ_2M + #ifdef CONFIG_DRM_I915_DEBUG_GUC void intel_uc_fw_change_status(struct intel_uc_fw *uc_fw, enum intel_uc_fw_status status); diff --git a/drivers/gpu/drm/i915/gt/uc/selftest_guc_hangcheck.c b/drivers/gpu/drm/i915/gt/uc/selftest_guc_hangcheck.c index 01f8cd3c3134..d91b58f70403 100644 --- a/drivers/gpu/drm/i915/gt/uc/selftest_guc_hangcheck.c +++ b/drivers/gpu/drm/i915/gt/uc/selftest_guc_hangcheck.c @@ -35,11 +35,14 @@ static int intel_hang_guc(void *arg) struct i915_request *rq; intel_wakeref_t wakeref; struct i915_gpu_error *global = >->i915->gpu_error; - struct intel_engine_cs *engine; + struct intel_engine_cs *engine = intel_selftest_find_any_engine(gt); unsigned int reset_count; u32 guc_status; u32 old_beat; + if (!engine) + return 0; + ctx = kernel_context(gt->i915, NULL); if (IS_ERR(ctx)) { drm_err(>->i915->drm, "Failed get kernel context: %ld\n", PTR_ERR(ctx)); @@ -48,14 +51,13 @@ static int intel_hang_guc(void *arg) wakeref = intel_runtime_pm_get(gt->uncore->rpm); - ce = intel_context_create(gt->engine[BCS0]); + ce = intel_context_create(engine); if (IS_ERR(ce)) { ret = PTR_ERR(ce); drm_err(>->i915->drm, "Failed to create spinner request: %d\n", ret); goto err; } - engine = ce->engine; reset_count = i915_reset_count(global); old_beat = engine->props.heartbeat_interval_ms; diff --git a/drivers/gpu/drm/i915/gvt/cfg_space.c b/drivers/gpu/drm/i915/gvt/cfg_space.c index eef3bba8a41b..9bafac1eaf48 100644 --- a/drivers/gpu/drm/i915/gvt/cfg_space.c +++ b/drivers/gpu/drm/i915/gvt/cfg_space.c @@ -244,7 +244,7 @@ static void emulate_pci_bar_write(struct intel_vgpu *vgpu, unsigned int offset, } /** - * intel_vgpu_emulate_cfg_read - emulate vGPU configuration space write + * intel_vgpu_emulate_cfg_write - emulate vGPU configuration space write * @vgpu: target vgpu * @offset: offset * @p_data: write data ptr @@ -354,9 +354,9 @@ void intel_vgpu_init_cfg_space(struct intel_vgpu *vgpu, memset(vgpu_cfg_space(vgpu) + INTEL_GVT_PCI_OPREGION, 0, 4); vgpu->cfg_space.bar[INTEL_GVT_PCI_BAR_GTTMMIO].size = - pci_resource_len(pdev, GTTMMADR_BAR); + pci_resource_len(pdev, GEN4_GTTMMADR_BAR); vgpu->cfg_space.bar[INTEL_GVT_PCI_BAR_APERTURE].size = - pci_resource_len(pdev, GTT_APERTURE_BAR); + pci_resource_len(pdev, GEN4_GMADR_BAR); memset(vgpu_cfg_space(vgpu) + PCI_ROM_ADDRESS, 0, 4); diff --git a/drivers/gpu/drm/i915/gvt/cmd_parser.c b/drivers/gpu/drm/i915/gvt/cmd_parser.c index de13f102d4fd..0ebf5fbf0e39 100644 --- a/drivers/gpu/drm/i915/gvt/cmd_parser.c +++ b/drivers/gpu/drm/i915/gvt/cmd_parser.c @@ -37,6 +37,7 @@ #include <linux/slab.h> #include "i915_drv.h" +#include "i915_reg.h" #include "gt/intel_engine_regs.h" #include "gt/intel_gpu_commands.h" #include "gt/intel_gt_regs.h" diff --git a/drivers/gpu/drm/i915/gvt/display.c b/drivers/gpu/drm/i915/gvt/display.c index c7722c818b4d..c033249e73f4 100644 --- a/drivers/gpu/drm/i915/gvt/display.c +++ b/drivers/gpu/drm/i915/gvt/display.c @@ -36,6 +36,8 @@ #include "i915_reg.h" #include "gvt.h" +#include "display/intel_dpio_phy.h" + static int get_edp_pipe(struct intel_vgpu *vgpu) { u32 data = vgpu_vreg(vgpu, _TRANS_DDI_FUNC_CTL_EDP); diff --git a/drivers/gpu/drm/i915/gvt/dmabuf.c b/drivers/gpu/drm/i915/gvt/dmabuf.c index 01e54b45c5c1..355f1c0e8664 100644 --- a/drivers/gpu/drm/i915/gvt/dmabuf.c +++ b/drivers/gpu/drm/i915/gvt/dmabuf.c @@ -88,7 +88,7 @@ static int vgpu_gem_get_pages( sg_dma_address(sg) = dma_addr; } - __i915_gem_object_set_pages(obj, st, PAGE_SIZE); + __i915_gem_object_set_pages(obj, st); out: if (ret) { dma_addr_t dma_addr; diff --git a/drivers/gpu/drm/i915/gvt/dmabuf.h b/drivers/gpu/drm/i915/gvt/dmabuf.h index 5f8f03fb1d1b..3dcdb6570eda 100644 --- a/drivers/gpu/drm/i915/gvt/dmabuf.h +++ b/drivers/gpu/drm/i915/gvt/dmabuf.h @@ -48,7 +48,7 @@ struct intel_vgpu_fb_info { struct intel_vgpu_dmabuf_obj *obj; }; -/** +/* * struct intel_vgpu_dmabuf_obj- Intel vGPU device buffer object */ struct intel_vgpu_dmabuf_obj { diff --git a/drivers/gpu/drm/i915/gvt/firmware.c b/drivers/gpu/drm/i915/gvt/firmware.c index 54fe442238c6..a683c22d5b64 100644 --- a/drivers/gpu/drm/i915/gvt/firmware.c +++ b/drivers/gpu/drm/i915/gvt/firmware.c @@ -104,7 +104,7 @@ static int expose_firmware_sysfs(struct intel_gvt *gvt) memcpy(p, gvt->firmware.mmio, info->mmio_size); - crc32_start = offsetof(struct gvt_firmware_header, crc32) + 4; + crc32_start = offsetof(struct gvt_firmware_header, version); h->crc32 = crc32_le(0, firmware + crc32_start, size - crc32_start); firmware_attr.size = size; diff --git a/drivers/gpu/drm/i915/gvt/gtt.c b/drivers/gpu/drm/i915/gvt/gtt.c index ce0eb03709c3..51e5e8fb505b 100644 --- a/drivers/gpu/drm/i915/gvt/gtt.c +++ b/drivers/gpu/drm/i915/gvt/gtt.c @@ -282,11 +282,6 @@ static inline int get_next_pt_type(int type) return gtt_type_table[type].next_pt_type; } -static inline int get_pt_type(int type) -{ - return gtt_type_table[type].pt_type; -} - static inline int get_entry_type(int type) { return gtt_type_table[type].entry_type; @@ -2785,7 +2780,7 @@ int intel_gvt_init_gtt(struct intel_gvt *gvt) * intel_gvt_clean_gtt - clean up mm components of a GVT device * @gvt: GVT device * - * This function is called at the driver unloading stage, to clean up the + * This function is called at the driver unloading stage, to clean up * the mm components of a GVT device. * */ diff --git a/drivers/gpu/drm/i915/gvt/gvt.h b/drivers/gpu/drm/i915/gvt/gvt.h index dbf8d7470b2c..62823c0e13ab 100644 --- a/drivers/gpu/drm/i915/gvt/gvt.h +++ b/drivers/gpu/drm/i915/gvt/gvt.h @@ -227,8 +227,6 @@ struct intel_vgpu { unsigned long nr_cache_entries; struct mutex cache_lock; - atomic_t released; - struct kvm_page_track_notifier_node track_node; #define NR_BKT (1 << 18) struct hlist_head ptable[NR_BKT]; diff --git a/drivers/gpu/drm/i915/gvt/handlers.c b/drivers/gpu/drm/i915/gvt/handlers.c index daac2050d77d..735fc83e7026 100644 --- a/drivers/gpu/drm/i915/gvt/handlers.c +++ b/drivers/gpu/drm/i915/gvt/handlers.c @@ -43,6 +43,7 @@ #include "intel_mchbar_regs.h" #include "display/intel_display_types.h" #include "display/intel_dmc_regs.h" +#include "display/intel_dpio_phy.h" #include "display/intel_fbc.h" #include "display/vlv_dsi_pll_regs.h" #include "gt/intel_gt_regs.h" @@ -734,7 +735,7 @@ static i915_reg_t force_nonpriv_white_list[] = { _MMIO(0x770c), _MMIO(0x83a8), _MMIO(0xb110), - GEN8_L3SQCREG4,//_MMIO(0xb118) + _MMIO(0xb118), _MMIO(0xe100), _MMIO(0xe18c), _MMIO(0xe48c), @@ -2257,7 +2258,7 @@ static int init_generic_mmio_info(struct intel_gvt *gvt) MMIO_DFH(_MMIO(0x2438), D_ALL, F_CMD_ACCESS, NULL, NULL); MMIO_DFH(_MMIO(0x243c), D_ALL, F_CMD_ACCESS, NULL, NULL); MMIO_DFH(_MMIO(0x7018), D_ALL, F_MODE_MASK | F_CMD_ACCESS, NULL, NULL); - MMIO_DFH(HALF_SLICE_CHICKEN3, D_ALL, F_MODE_MASK | F_CMD_ACCESS, NULL, NULL); + MMIO_DFH(HSW_HALF_SLICE_CHICKEN3, D_ALL, F_MODE_MASK | F_CMD_ACCESS, NULL, NULL); MMIO_DFH(GEN7_HALF_SLICE_CHICKEN1, D_ALL, F_MODE_MASK | F_CMD_ACCESS, NULL, NULL); /* display */ diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c index 714221f9a131..077892a9aa8f 100644 --- a/drivers/gpu/drm/i915/gvt/kvmgt.c +++ b/drivers/gpu/drm/i915/gvt/kvmgt.c @@ -683,7 +683,6 @@ static int intel_vgpu_open_device(struct vfio_device *vfio_dev) intel_gvt_activate_vgpu(vgpu); - atomic_set(&vgpu->released, 0); return 0; } @@ -705,9 +704,6 @@ static void intel_vgpu_close_device(struct vfio_device *vfio_dev) if (!vgpu->attached) return; - if (atomic_cmpxchg(&vgpu->released, 0, 1)) - return; - intel_gvt_release_vgpu(vgpu); debugfs_remove(debugfs_lookup(KVMGT_DEBUGFS_FILENAME, vgpu->debugfs)); diff --git a/drivers/gpu/drm/i915/gvt/mmio.c b/drivers/gpu/drm/i915/gvt/mmio.c index 9acc00505fde..5b5def6ddef7 100644 --- a/drivers/gpu/drm/i915/gvt/mmio.c +++ b/drivers/gpu/drm/i915/gvt/mmio.c @@ -37,6 +37,7 @@ #include "i915_reg.h" #include "gvt.h" +#include "display/intel_dpio_phy.h" #include "gt/intel_gt_regs.h" /** diff --git a/drivers/gpu/drm/i915/gvt/mmio_context.c b/drivers/gpu/drm/i915/gvt/mmio_context.c index 1c6e941c9666..490e8ae51228 100644 --- a/drivers/gpu/drm/i915/gvt/mmio_context.c +++ b/drivers/gpu/drm/i915/gvt/mmio_context.c @@ -34,6 +34,7 @@ */ #include "i915_drv.h" +#include "i915_reg.h" #include "gt/intel_context.h" #include "gt/intel_engine_regs.h" #include "gt/intel_gpu_commands.h" @@ -106,15 +107,15 @@ static struct engine_mmio gen9_engine_mmio_list[] __cacheline_aligned = { {RCS0, GEN8_CS_CHICKEN1, 0xffff, true}, /* 0x2580 */ {RCS0, COMMON_SLICE_CHICKEN2, 0xffff, true}, /* 0x7014 */ {RCS0, GEN9_CS_DEBUG_MODE1, 0xffff, false}, /* 0x20ec */ - {RCS0, GEN8_L3SQCREG4, 0, false}, /* 0xb118 */ - {RCS0, GEN9_SCRATCH1, 0, false}, /* 0xb11c */ + {RCS0, _MMIO(0xb118), 0, false}, /* GEN8_L3SQCREG4 */ + {RCS0, _MMIO(0xb11c), 0, false}, /* GEN9_SCRATCH1 */ {RCS0, GEN9_SCRATCH_LNCF1, 0, false}, /* 0xb008 */ {RCS0, GEN7_HALF_SLICE_CHICKEN1, 0xffff, true}, /* 0xe100 */ - {RCS0, HALF_SLICE_CHICKEN2, 0xffff, true}, /* 0xe180 */ - {RCS0, HALF_SLICE_CHICKEN3, 0xffff, true}, /* 0xe184 */ - {RCS0, GEN9_HALF_SLICE_CHICKEN5, 0xffff, true}, /* 0xe188 */ - {RCS0, GEN9_HALF_SLICE_CHICKEN7, 0xffff, true}, /* 0xe194 */ - {RCS0, GEN8_ROW_CHICKEN, 0xffff, true}, /* 0xe4f0 */ + {RCS0, _MMIO(0xe180), 0xffff, true}, /* HALF_SLICE_CHICKEN2 */ + {RCS0, _MMIO(0xe184), 0xffff, true}, /* GEN8_HALF_SLICE_CHICKEN3 */ + {RCS0, _MMIO(0xe188), 0xffff, true}, /* GEN9_HALF_SLICE_CHICKEN5 */ + {RCS0, _MMIO(0xe194), 0xffff, true}, /* GEN9_HALF_SLICE_CHICKEN7 */ + {RCS0, _MMIO(0xe4f0), 0xffff, true}, /* GEN8_ROW_CHICKEN */ {RCS0, TRVATTL3PTRDW(0), 0, true}, /* 0x4de0 */ {RCS0, TRVATTL3PTRDW(1), 0, true}, /* 0x4de4 */ {RCS0, TRNULLDETCT, 0, true}, /* 0x4de8 */ diff --git a/drivers/gpu/drm/i915/gvt/page_track.c b/drivers/gpu/drm/i915/gvt/page_track.c index 3375b51c75f1..df34e73cba41 100644 --- a/drivers/gpu/drm/i915/gvt/page_track.c +++ b/drivers/gpu/drm/i915/gvt/page_track.c @@ -120,7 +120,7 @@ int intel_vgpu_enable_page_track(struct intel_vgpu *vgpu, unsigned long gfn) } /** - * intel_vgpu_enable_page_track - cancel write-protection on guest page + * intel_vgpu_disable_page_track - cancel write-protection on guest page * @vgpu: a vGPU * @gfn: the gfn of guest page * diff --git a/drivers/gpu/drm/i915/gvt/scheduler.c b/drivers/gpu/drm/i915/gvt/scheduler.c index d6fe94cd0fdb..9cd8fcbf7cad 100644 --- a/drivers/gpu/drm/i915/gvt/scheduler.c +++ b/drivers/gpu/drm/i915/gvt/scheduler.c @@ -570,9 +570,8 @@ retry: if (gmadr_bytes == 8) bb->bb_start_cmd_va[2] = 0; - ret = i915_vma_move_to_active(bb->vma, - workload->req, - 0); + ret = i915_vma_move_to_active(bb->vma, workload->req, + __EXEC_OBJECT_NO_REQUEST_AWAIT); if (ret) goto err; diff --git a/drivers/gpu/drm/i915/gvt/vgpu.c b/drivers/gpu/drm/i915/gvt/vgpu.c index 56c71474008a..3c529c2705dd 100644 --- a/drivers/gpu/drm/i915/gvt/vgpu.c +++ b/drivers/gpu/drm/i915/gvt/vgpu.c @@ -158,7 +158,7 @@ void intel_gvt_clean_vgpu_types(struct intel_gvt *gvt) } /** - * intel_gvt_active_vgpu - activate a virtual GPU + * intel_gvt_activate_vgpu - activate a virtual GPU * @vgpu: virtual GPU * * This function is called when user wants to activate a virtual GPU. @@ -172,7 +172,7 @@ void intel_gvt_activate_vgpu(struct intel_vgpu *vgpu) } /** - * intel_gvt_deactive_vgpu - deactivate a virtual GPU + * intel_gvt_deactivate_vgpu - deactivate a virtual GPU * @vgpu: virtual GPU * * This function is called when user wants to deactivate a virtual GPU. @@ -295,7 +295,7 @@ out_free_vgpu: } /** - * intel_gvt_destroy_vgpu - destroy an idle virtual GPU + * intel_gvt_destroy_idle_vgpu - destroy an idle virtual GPU * @vgpu: virtual GPU * * This function is called when user wants to destroy an idle virtual GPU. diff --git a/drivers/gpu/drm/i915/i915_debugfs.c b/drivers/gpu/drm/i915/i915_debugfs.c index ae987e92251d..6c7ac73b69a5 100644 --- a/drivers/gpu/drm/i915/i915_debugfs.c +++ b/drivers/gpu/drm/i915/i915_debugfs.c @@ -688,8 +688,8 @@ i915_drop_caches_set(void *data, u64 val) unsigned int flags; int ret; - DRM_DEBUG("Dropping caches: 0x%08llx [0x%08llx]\n", - val, val & DROP_ALL); + drm_dbg(&i915->drm, "Dropping caches: 0x%08llx [0x%08llx]\n", + val, val & DROP_ALL); ret = gt_drop_caches(to_gt(i915), val); if (ret) diff --git a/drivers/gpu/drm/i915/i915_driver.c b/drivers/gpu/drm/i915/i915_driver.c index f2a15d8155f4..69103ae37779 100644 --- a/drivers/gpu/drm/i915/i915_driver.c +++ b/drivers/gpu/drm/i915/i915_driver.c @@ -81,6 +81,7 @@ #include "i915_drm_client.h" #include "i915_drv.h" #include "i915_getparam.h" +#include "i915_hwmon.h" #include "i915_ioc32.h" #include "i915_ioctl.h" #include "i915_irq.h" @@ -337,7 +338,8 @@ static int i915_driver_early_probe(struct drm_i915_private *dev_priv) if (i915_inject_probe_failure(dev_priv)) return -ENODEV; - intel_device_info_subplatform_init(dev_priv); + intel_device_info_runtime_init_early(dev_priv); + intel_step_init(dev_priv); intel_uncore_mmio_debug_init_early(dev_priv); @@ -370,8 +372,6 @@ static int i915_driver_early_probe(struct drm_i915_private *dev_priv) if (ret) goto err_ttm; - intel_wopcm_init_early(&dev_priv->wopcm); - ret = intel_root_gt_init_early(dev_priv); if (ret < 0) goto err_rootgt; @@ -739,7 +739,6 @@ static void i915_driver_hw_remove(struct drm_i915_private *dev_priv) */ static void i915_driver_register(struct drm_i915_private *dev_priv) { - struct drm_device *dev = &dev_priv->drm; struct intel_gt *gt; unsigned int i; @@ -749,7 +748,7 @@ static void i915_driver_register(struct drm_i915_private *dev_priv) intel_vgpu_register(dev_priv); /* Reveal our presence to userspace */ - if (drm_dev_register(dev, 0)) { + if (drm_dev_register(&dev_priv->drm, 0)) { drm_err(&dev_priv->drm, "Failed to register driver for userspace access!\n"); return; @@ -764,6 +763,8 @@ static void i915_driver_register(struct drm_i915_private *dev_priv) for_each_gt(gt, dev_priv, i) intel_gt_driver_register(gt); + i915_hwmon_register(dev_priv); + intel_display_driver_register(dev_priv); intel_power_domains_enable(dev_priv); @@ -796,6 +797,8 @@ static void i915_driver_unregister(struct drm_i915_private *dev_priv) for_each_gt(gt, dev_priv, i) intel_gt_driver_unregister(gt); + i915_hwmon_unregister(dev_priv); + i915_perf_unregister(dev_priv); i915_pmu_unregister(dev_priv); @@ -894,10 +897,6 @@ int i915_driver_probe(struct pci_dev *pdev, const struct pci_device_id *ent) if (IS_ERR(i915)) return PTR_ERR(i915); - /* Disable nuclear pageflip by default on pre-ILK */ - if (!i915->params.nuclear_pageflip && DISPLAY_VER(i915) < 5) - i915->drm.driver_features &= ~DRIVER_ATOMIC; - ret = pci_enable_device(pdev); if (ret) goto out_fini; @@ -1093,32 +1092,30 @@ static void i915_driver_postclose(struct drm_device *dev, struct drm_file *file) static void intel_suspend_encoders(struct drm_i915_private *dev_priv) { - struct drm_device *dev = &dev_priv->drm; struct intel_encoder *encoder; if (!HAS_DISPLAY(dev_priv)) return; - drm_modeset_lock_all(dev); - for_each_intel_encoder(dev, encoder) + drm_modeset_lock_all(&dev_priv->drm); + for_each_intel_encoder(&dev_priv->drm, encoder) if (encoder->suspend) encoder->suspend(encoder); - drm_modeset_unlock_all(dev); + drm_modeset_unlock_all(&dev_priv->drm); } static void intel_shutdown_encoders(struct drm_i915_private *dev_priv) { - struct drm_device *dev = &dev_priv->drm; struct intel_encoder *encoder; if (!HAS_DISPLAY(dev_priv)) return; - drm_modeset_lock_all(dev); - for_each_intel_encoder(dev, encoder) + drm_modeset_lock_all(&dev_priv->drm); + for_each_intel_encoder(&dev_priv->drm, encoder) if (encoder->shutdown) encoder->shutdown(encoder); - drm_modeset_unlock_all(dev); + drm_modeset_unlock_all(&dev_priv->drm); } void i915_driver_shutdown(struct drm_i915_private *i915) @@ -1662,7 +1659,8 @@ static int intel_runtime_suspend(struct device *kdev) intel_runtime_pm_enable_interrupts(dev_priv); - intel_gt_runtime_resume(to_gt(dev_priv)); + for_each_gt(gt, dev_priv, i) + intel_gt_runtime_resume(gt); enable_rpm_wakeref_asserts(rpm); diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h index bdc81db76dbd..a380db36d52c 100644 --- a/drivers/gpu/drm/i915/i915_drv.h +++ b/drivers/gpu/drm/i915/i915_drv.h @@ -40,7 +40,6 @@ #include "display/intel_display_core.h" #include "gem/i915_gem_context_types.h" -#include "gem/i915_gem_lmem.h" #include "gem/i915_gem_shrinker.h" #include "gem/i915_gem_stolen.h" @@ -63,7 +62,6 @@ #include "intel_runtime_pm.h" #include "intel_step.h" #include "intel_uncore.h" -#include "intel_wopcm.h" struct drm_i915_clock_gating_funcs; struct drm_i915_gem_object; @@ -75,9 +73,6 @@ struct intel_limit; struct intel_overlay_error_state; struct vlv_s0ix_state; -/* Threshold == 5 for long IRQs, 50 for short */ -#define HPD_STORM_DEFAULT_THRESHOLD 50 - #define I915_GEM_GPU_DOMAINS \ (I915_GEM_DOMAIN_RENDER | \ I915_GEM_DOMAIN_SAMPLER | \ @@ -239,8 +234,6 @@ struct drm_i915_private { struct intel_gvt *gvt; - struct intel_wopcm wopcm; - struct pci_dev *bridge_dev; struct rb_root uabi_engines; @@ -291,28 +284,13 @@ struct drm_i915_private { unsigned long gem_quirks; - struct drm_atomic_state *modeset_restore_state; - struct drm_modeset_acquire_ctx reset_ctx; - struct i915_gem_mm mm; - /* Kernel Modesetting */ - - struct list_head global_obj_list; - bool mchbar_need_disable; struct intel_l3_parity l3_parity; /* - * HTI (aka HDPORT) state read during initial hw readout. Most - * platforms don't have HTI, so this will just stay 0. Those that do - * will use this later to figure out which PLLs and PHYs are unavailable - * for driver usage. - */ - u32 hti_state; - - /* * edram size in MB. * Cannot be determined by PCIID. You must always read a register. */ @@ -353,6 +331,8 @@ struct drm_i915_private { struct i915_perf perf; + struct i915_hwmon *hwmon; + /* Abstract the submission mechanism (legacy ringbuffer or execlists) away */ struct intel_gt gt0; @@ -742,6 +722,10 @@ IS_SUBPLATFORM(const struct drm_i915_private *i915, #define IS_XEHPSDV_GRAPHICS_STEP(__i915, since, until) \ (IS_XEHPSDV(__i915) && IS_GRAPHICS_STEP(__i915, since, until)) +#define IS_MTL_GRAPHICS_STEP(__i915, variant, since, until) \ + (IS_SUBPLATFORM(__i915, INTEL_METEORLAKE, INTEL_SUBPLATFORM_##variant) && \ + IS_GRAPHICS_STEP(__i915, since, until)) + /* * DG2 hardware steppings are a bit unusual. The hardware design was forked to * create three variants (G10, G11, and G12) which each have distinct @@ -780,12 +764,15 @@ IS_SUBPLATFORM(const struct drm_i915_private *i915, #define __HAS_ENGINE(engine_mask, id) ((engine_mask) & BIT(id)) #define HAS_ENGINE(gt, id) __HAS_ENGINE((gt)->info.engine_mask, id) -#define ENGINE_INSTANCES_MASK(gt, first, count) ({ \ +#define __ENGINE_INSTANCES_MASK(mask, first, count) ({ \ unsigned int first__ = (first); \ unsigned int count__ = (count); \ - ((gt)->info.engine_mask & \ - GENMASK(first__ + count__ - 1, first__)) >> first__; \ + ((mask) & GENMASK(first__ + count__ - 1, first__)) >> first__; \ }) + +#define ENGINE_INSTANCES_MASK(gt, first, count) \ + __ENGINE_INSTANCES_MASK((gt)->info.engine_mask, first, count) + #define RCS_MASK(gt) \ ENGINE_INSTANCES_MASK(gt, RCS0, I915_MAX_RCS) #define BCS_MASK(gt) \ @@ -871,6 +858,7 @@ IS_SUBPLATFORM(const struct drm_i915_private *i915, #define HAS_DOUBLE_BUFFERED_M_N(dev_priv) (DISPLAY_VER(dev_priv) >= 9 || IS_BROADWELL(dev_priv)) #define HAS_CDCLK_CRAWL(dev_priv) (INTEL_INFO(dev_priv)->display.has_cdclk_crawl) +#define HAS_CDCLK_SQUASH(dev_priv) (INTEL_INFO(dev_priv)->display.has_cdclk_squash) #define HAS_DDI(dev_priv) (INTEL_INFO(dev_priv)->display.has_ddi) #define HAS_FPGA_DBG_UNCLAIMED(dev_priv) (INTEL_INFO(dev_priv)->display.has_fpga_dbg) #define HAS_PSR(dev_priv) (INTEL_INFO(dev_priv)->display.has_psr) @@ -900,19 +888,17 @@ IS_SUBPLATFORM(const struct drm_i915_private *i915, #define HAS_RUNTIME_PM(dev_priv) (INTEL_INFO(dev_priv)->has_runtime_pm) #define HAS_64BIT_RELOC(dev_priv) (INTEL_INFO(dev_priv)->has_64bit_reloc) +#define HAS_OA_BPC_REPORTING(dev_priv) \ + (INTEL_INFO(dev_priv)->has_oa_bpc_reporting) +#define HAS_OA_SLICE_CONTRIB_LIMITS(dev_priv) \ + (INTEL_INFO(dev_priv)->has_oa_slice_contrib_limits) + /* * Set this flag, when platform requires 64K GTT page sizes or larger for * device local memory access. */ #define HAS_64K_PAGES(dev_priv) (INTEL_INFO(dev_priv)->has_64k_pages) -/* - * Set this flag when platform doesn't allow both 64k pages and 4k pages in - * the same PT. this flag means we need to support compact PT layout for the - * ppGTT when using the 64K GTT pages. - */ -#define NEEDS_COMPACT_PT(dev_priv) (INTEL_INFO(dev_priv)->needs_compact_pt) - #define HAS_IPC(dev_priv) (INTEL_INFO(dev_priv)->display.has_ipc) #define HAS_REGION(i915, i) (RUNTIME_INFO(i915)->memory_regions & (i)) @@ -938,6 +924,8 @@ IS_SUBPLATFORM(const struct drm_i915_private *i915, #define HAS_GMCH(dev_priv) (INTEL_INFO(dev_priv)->display.has_gmch) +#define HAS_GMD_ID(i915) (INTEL_INFO(i915)->has_gmd_id) + #define HAS_LSPCON(dev_priv) (IS_DISPLAY_VER(dev_priv, 9, 10)) #define HAS_L3_CCS_READ(i915) (INTEL_INFO(i915)->has_l3_ccs_read) @@ -976,6 +964,9 @@ IS_SUBPLATFORM(const struct drm_i915_private *i915, #define HAS_ONE_EU_PER_FUSE_BIT(i915) (INTEL_INFO(i915)->has_one_eu_per_fuse_bit) +#define HAS_LMEMBAR_SMEM_STOLEN(i915) (!HAS_LMEM(i915) && \ + GRAPHICS_VER_FULL(i915) >= IP_VER(12, 70)) + /* intel_device_info.c */ static inline struct intel_device_info * mkwrite_device_info(struct drm_i915_private *dev_priv) @@ -983,16 +974,4 @@ mkwrite_device_info(struct drm_i915_private *dev_priv) return (struct intel_device_info *)INTEL_INFO(dev_priv); } -static inline enum i915_map_type -i915_coherent_map_type(struct drm_i915_private *i915, - struct drm_i915_gem_object *obj, bool always_coherent) -{ - if (i915_gem_object_is_lmem(obj)) - return I915_MAP_WC; - if (HAS_LLC(i915) || always_coherent) - return I915_MAP_WB; - else - return I915_MAP_WC; -} - #endif diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c index 2bdddb61ebd7..8468ca9885fd 100644 --- a/drivers/gpu/drm/i915/i915_gem.c +++ b/drivers/gpu/drm/i915/i915_gem.c @@ -843,7 +843,7 @@ void i915_gem_runtime_suspend(struct drm_i915_private *i915) __i915_gem_object_release_mmap_gtt(obj); list_for_each_entry_safe(obj, on, - &to_gt(i915)->lmem_userfault_list, userfault_link) + &i915->runtime_pm.lmem_userfault_list, userfault_link) i915_gem_object_runtime_pm_release_mmap_offset(obj); /* @@ -1128,6 +1128,8 @@ void i915_gem_drain_workqueue(struct drm_i915_private *i915) int i915_gem_init(struct drm_i915_private *dev_priv) { + struct intel_gt *gt; + unsigned int i; int ret; /* We need to fallback to 4K pages if host doesn't support huge gtt. */ @@ -1138,8 +1140,10 @@ int i915_gem_init(struct drm_i915_private *dev_priv) if (ret) return ret; - intel_uc_fetch_firmwares(&to_gt(dev_priv)->uc); - intel_wopcm_init(&dev_priv->wopcm); + for_each_gt(gt, dev_priv, i) { + intel_uc_fetch_firmwares(>->uc); + intel_wopcm_init(>->wopcm); + } ret = i915_init_ggtt(dev_priv); if (ret) { @@ -1158,9 +1162,11 @@ int i915_gem_init(struct drm_i915_private *dev_priv) */ intel_init_clock_gating(dev_priv); - ret = intel_gt_init(to_gt(dev_priv)); - if (ret) - goto err_unlock; + for_each_gt(gt, dev_priv, i) { + ret = intel_gt_init(gt); + if (ret) + goto err_unlock; + } return 0; @@ -1173,8 +1179,13 @@ int i915_gem_init(struct drm_i915_private *dev_priv) err_unlock: i915_gem_drain_workqueue(dev_priv); - if (ret != -EIO) - intel_uc_cleanup_firmwares(&to_gt(dev_priv)->uc); + if (ret != -EIO) { + for_each_gt(gt, dev_priv, i) { + intel_gt_driver_remove(gt); + intel_gt_driver_release(gt); + intel_uc_cleanup_firmwares(>->uc); + } + } if (ret == -EIO) { /* @@ -1182,10 +1193,12 @@ err_unlock: * as wedged. But we only want to do this when the GPU is angry, * for all other failure, such as an allocation failure, bail. */ - if (!intel_gt_is_wedged(to_gt(dev_priv))) { - i915_probe_error(dev_priv, - "Failed to initialize GPU, declaring it wedged!\n"); - intel_gt_set_wedged(to_gt(dev_priv)); + for_each_gt(gt, dev_priv, i) { + if (!intel_gt_is_wedged(gt)) { + i915_probe_error(dev_priv, + "Failed to initialize GPU, declaring it wedged!\n"); + intel_gt_set_wedged(gt); + } } /* Minimal basic recovery for KMS */ @@ -1213,23 +1226,27 @@ void i915_gem_driver_unregister(struct drm_i915_private *i915) void i915_gem_driver_remove(struct drm_i915_private *dev_priv) { - intel_wakeref_auto_fini(&to_gt(dev_priv)->userfault_wakeref); + struct intel_gt *gt; + unsigned int i; i915_gem_suspend_late(dev_priv); - intel_gt_driver_remove(to_gt(dev_priv)); + for_each_gt(gt, dev_priv, i) + intel_gt_driver_remove(gt); dev_priv->uabi_engines = RB_ROOT; /* Flush any outstanding unpin_work. */ i915_gem_drain_workqueue(dev_priv); - - i915_gem_drain_freed_objects(dev_priv); } void i915_gem_driver_release(struct drm_i915_private *dev_priv) { - intel_gt_driver_release(to_gt(dev_priv)); + struct intel_gt *gt; + unsigned int i; - intel_uc_cleanup_firmwares(&to_gt(dev_priv)->uc); + for_each_gt(gt, dev_priv, i) { + intel_gt_driver_release(gt); + intel_uc_cleanup_firmwares(>->uc); + } /* Flush any outstanding work, including i915_gem_context.release_work. */ i915_gem_drain_workqueue(dev_priv); @@ -1259,7 +1276,7 @@ void i915_gem_init_early(struct drm_i915_private *dev_priv) void i915_gem_cleanup_early(struct drm_i915_private *dev_priv) { - i915_gem_drain_freed_objects(dev_priv); + i915_gem_drain_workqueue(dev_priv); GEM_BUG_ON(!llist_empty(&dev_priv->mm.free_list)); GEM_BUG_ON(atomic_read(&dev_priv->mm.free_count)); drm_WARN_ON(&dev_priv->drm, dev_priv->mm.shrink_count); @@ -1271,7 +1288,7 @@ int i915_gem_open(struct drm_i915_private *i915, struct drm_file *file) struct i915_drm_client *client; int ret = -ENOMEM; - DRM_DEBUG("\n"); + drm_dbg(&i915->drm, "\n"); file_priv = kzalloc(sizeof(*file_priv), GFP_KERNEL); if (!file_priv) diff --git a/drivers/gpu/drm/i915/i915_getparam.c b/drivers/gpu/drm/i915/i915_getparam.c index 342c8ca6414e..61ef2d9cfa62 100644 --- a/drivers/gpu/drm/i915/i915_getparam.c +++ b/drivers/gpu/drm/i915/i915_getparam.c @@ -175,8 +175,11 @@ int i915_getparam_ioctl(struct drm_device *dev, void *data, case I915_PARAM_PERF_REVISION: value = i915_perf_ioctl_version(); break; + case I915_PARAM_OA_TIMESTAMP_FREQUENCY: + value = i915_perf_oa_timestamp_frequency(i915); + break; default: - DRM_DEBUG("Unknown parameter %d\n", param->param); + drm_dbg(&i915->drm, "Unknown parameter %d\n", param->param); return -EINVAL; } diff --git a/drivers/gpu/drm/i915/i915_gpu_error.c b/drivers/gpu/drm/i915/i915_gpu_error.c index 9ea2fe34e7d3..9d5d5a397b64 100644 --- a/drivers/gpu/drm/i915/i915_gpu_error.c +++ b/drivers/gpu/drm/i915/i915_gpu_error.c @@ -55,6 +55,7 @@ #include "i915_drv.h" #include "i915_gpu_error.h" #include "i915_memcpy.h" +#include "i915_reg.h" #include "i915_scatterlist.h" #include "i915_utils.h" @@ -1221,7 +1222,10 @@ static void engine_record_registers(struct intel_engine_coredump *ee) if (GRAPHICS_VER(i915) >= 6) { ee->rc_psmi = ENGINE_READ(engine, RING_PSMI_CTL); - if (GRAPHICS_VER(i915) >= 12) + if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) + ee->fault_reg = intel_gt_mcr_read_any(engine->gt, + XEHP_RING_FAULT_REG); + else if (GRAPHICS_VER(i915) >= 12) ee->fault_reg = intel_uncore_read(engine->uncore, GEN12_RING_FAULT_REG); else if (GRAPHICS_VER(i915) >= 8) @@ -1820,7 +1824,12 @@ static void gt_record_global_regs(struct intel_gt_coredump *gt) if (GRAPHICS_VER(i915) == 7) gt->err_int = intel_uncore_read(uncore, GEN7_ERR_INT); - if (GRAPHICS_VER(i915) >= 12) { + if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) { + gt->fault_data0 = intel_gt_mcr_read_any((struct intel_gt *)gt->_gt, + XEHP_FAULT_TLB_DATA0); + gt->fault_data1 = intel_gt_mcr_read_any((struct intel_gt *)gt->_gt, + XEHP_FAULT_TLB_DATA1); + } else if (GRAPHICS_VER(i915) >= 12) { gt->fault_data0 = intel_uncore_read(uncore, GEN12_FAULT_TLB_DATA0); gt->fault_data1 = intel_uncore_read(uncore, diff --git a/drivers/gpu/drm/i915/i915_hwmon.c b/drivers/gpu/drm/i915/i915_hwmon.c new file mode 100644 index 000000000000..c588a17f97e9 --- /dev/null +++ b/drivers/gpu/drm/i915/i915_hwmon.c @@ -0,0 +1,732 @@ +// SPDX-License-Identifier: MIT +/* + * Copyright © 2022 Intel Corporation + */ + +#include <linux/hwmon.h> +#include <linux/hwmon-sysfs.h> +#include <linux/types.h> + +#include "i915_drv.h" +#include "i915_hwmon.h" +#include "i915_reg.h" +#include "intel_mchbar_regs.h" +#include "intel_pcode.h" +#include "gt/intel_gt.h" +#include "gt/intel_gt_regs.h" + +/* + * SF_* - scale factors for particular quantities according to hwmon spec. + * - voltage - millivolts + * - power - microwatts + * - curr - milliamperes + * - energy - microjoules + * - time - milliseconds + */ +#define SF_VOLTAGE 1000 +#define SF_POWER 1000000 +#define SF_CURR 1000 +#define SF_ENERGY 1000000 +#define SF_TIME 1000 + +struct hwm_reg { + i915_reg_t gt_perf_status; + i915_reg_t pkg_power_sku_unit; + i915_reg_t pkg_power_sku; + i915_reg_t pkg_rapl_limit; + i915_reg_t energy_status_all; + i915_reg_t energy_status_tile; +}; + +struct hwm_energy_info { + u32 reg_val_prev; + long accum_energy; /* Accumulated energy for energy1_input */ +}; + +struct hwm_drvdata { + struct i915_hwmon *hwmon; + struct intel_uncore *uncore; + struct device *hwmon_dev; + struct hwm_energy_info ei; /* Energy info for energy1_input */ + char name[12]; + int gt_n; +}; + +struct i915_hwmon { + struct hwm_drvdata ddat; + struct hwm_drvdata ddat_gt[I915_MAX_GT]; + struct mutex hwmon_lock; /* counter overflow logic and rmw */ + struct hwm_reg rg; + int scl_shift_power; + int scl_shift_energy; + int scl_shift_time; +}; + +static void +hwm_locked_with_pm_intel_uncore_rmw(struct hwm_drvdata *ddat, + i915_reg_t reg, u32 clear, u32 set) +{ + struct i915_hwmon *hwmon = ddat->hwmon; + struct intel_uncore *uncore = ddat->uncore; + intel_wakeref_t wakeref; + + mutex_lock(&hwmon->hwmon_lock); + + with_intel_runtime_pm(uncore->rpm, wakeref) + intel_uncore_rmw(uncore, reg, clear, set); + + mutex_unlock(&hwmon->hwmon_lock); +} + +/* + * This function's return type of u64 allows for the case where the scaling + * of the field taken from the 32-bit register value might cause a result to + * exceed 32 bits. + */ +static u64 +hwm_field_read_and_scale(struct hwm_drvdata *ddat, i915_reg_t rgadr, + u32 field_msk, int nshift, u32 scale_factor) +{ + struct intel_uncore *uncore = ddat->uncore; + intel_wakeref_t wakeref; + u32 reg_value; + + with_intel_runtime_pm(uncore->rpm, wakeref) + reg_value = intel_uncore_read(uncore, rgadr); + + reg_value = REG_FIELD_GET(field_msk, reg_value); + + return mul_u64_u32_shr(reg_value, scale_factor, nshift); +} + +static void +hwm_field_scale_and_write(struct hwm_drvdata *ddat, i915_reg_t rgadr, + int nshift, unsigned int scale_factor, long lval) +{ + u32 nval; + + /* Computation in 64-bits to avoid overflow. Round to nearest. */ + nval = DIV_ROUND_CLOSEST_ULL((u64)lval << nshift, scale_factor); + + hwm_locked_with_pm_intel_uncore_rmw(ddat, rgadr, + PKG_PWR_LIM_1, + REG_FIELD_PREP(PKG_PWR_LIM_1, nval)); +} + +/* + * hwm_energy - Obtain energy value + * + * The underlying energy hardware register is 32-bits and is subject to + * overflow. How long before overflow? For example, with an example + * scaling bit shift of 14 bits (see register *PACKAGE_POWER_SKU_UNIT) and + * a power draw of 1000 watts, the 32-bit counter will overflow in + * approximately 4.36 minutes. + * + * Examples: + * 1 watt: (2^32 >> 14) / 1 W / (60 * 60 * 24) secs/day -> 3 days + * 1000 watts: (2^32 >> 14) / 1000 W / 60 secs/min -> 4.36 minutes + * + * The function significantly increases overflow duration (from 4.36 + * minutes) by accumulating the energy register into a 'long' as allowed by + * the hwmon API. Using x86_64 128 bit arithmetic (see mul_u64_u32_shr()), + * a 'long' of 63 bits, SF_ENERGY of 1e6 (~20 bits) and + * hwmon->scl_shift_energy of 14 bits we have 57 (63 - 20 + 14) bits before + * energy1_input overflows. This at 1000 W is an overflow duration of 278 years. + */ +static void +hwm_energy(struct hwm_drvdata *ddat, long *energy) +{ + struct intel_uncore *uncore = ddat->uncore; + struct i915_hwmon *hwmon = ddat->hwmon; + struct hwm_energy_info *ei = &ddat->ei; + intel_wakeref_t wakeref; + i915_reg_t rgaddr; + u32 reg_val; + + if (ddat->gt_n >= 0) + rgaddr = hwmon->rg.energy_status_tile; + else + rgaddr = hwmon->rg.energy_status_all; + + mutex_lock(&hwmon->hwmon_lock); + + with_intel_runtime_pm(uncore->rpm, wakeref) + reg_val = intel_uncore_read(uncore, rgaddr); + + if (reg_val >= ei->reg_val_prev) + ei->accum_energy += reg_val - ei->reg_val_prev; + else + ei->accum_energy += UINT_MAX - ei->reg_val_prev + reg_val; + ei->reg_val_prev = reg_val; + + *energy = mul_u64_u32_shr(ei->accum_energy, SF_ENERGY, + hwmon->scl_shift_energy); + mutex_unlock(&hwmon->hwmon_lock); +} + +static ssize_t +hwm_power1_max_interval_show(struct device *dev, struct device_attribute *attr, + char *buf) +{ + struct hwm_drvdata *ddat = dev_get_drvdata(dev); + struct i915_hwmon *hwmon = ddat->hwmon; + intel_wakeref_t wakeref; + u32 r, x, y, x_w = 2; /* 2 bits */ + u64 tau4, out; + + with_intel_runtime_pm(ddat->uncore->rpm, wakeref) + r = intel_uncore_read(ddat->uncore, hwmon->rg.pkg_rapl_limit); + + x = REG_FIELD_GET(PKG_PWR_LIM_1_TIME_X, r); + y = REG_FIELD_GET(PKG_PWR_LIM_1_TIME_Y, r); + /* + * tau = 1.x * power(2,y), x = bits(23:22), y = bits(21:17) + * = (4 | x) << (y - 2) + * where (y - 2) ensures a 1.x fixed point representation of 1.x + * However because y can be < 2, we compute + * tau4 = (4 | x) << y + * but add 2 when doing the final right shift to account for units + */ + tau4 = ((1 << x_w) | x) << y; + /* val in hwmon interface units (millisec) */ + out = mul_u64_u32_shr(tau4, SF_TIME, hwmon->scl_shift_time + x_w); + + return sysfs_emit(buf, "%llu\n", out); +} + +static ssize_t +hwm_power1_max_interval_store(struct device *dev, + struct device_attribute *attr, + const char *buf, size_t count) +{ + struct hwm_drvdata *ddat = dev_get_drvdata(dev); + struct i915_hwmon *hwmon = ddat->hwmon; + u32 x, y, rxy, x_w = 2; /* 2 bits */ + u64 tau4, r, max_win; + unsigned long val; + int ret; + + ret = kstrtoul(buf, 0, &val); + if (ret) + return ret; + + /* + * Max HW supported tau in '1.x * power(2,y)' format, x = 0, y = 0x12 + * The hwmon->scl_shift_time default of 0xa results in a max tau of 256 seconds + */ +#define PKG_MAX_WIN_DEFAULT 0x12ull + + /* + * val must be < max in hwmon interface units. The steps below are + * explained in i915_power1_max_interval_show() + */ + r = FIELD_PREP(PKG_MAX_WIN, PKG_MAX_WIN_DEFAULT); + x = REG_FIELD_GET(PKG_MAX_WIN_X, r); + y = REG_FIELD_GET(PKG_MAX_WIN_Y, r); + tau4 = ((1 << x_w) | x) << y; + max_win = mul_u64_u32_shr(tau4, SF_TIME, hwmon->scl_shift_time + x_w); + + if (val > max_win) + return -EINVAL; + + /* val in hw units */ + val = DIV_ROUND_CLOSEST_ULL((u64)val << hwmon->scl_shift_time, SF_TIME); + /* Convert to 1.x * power(2,y) */ + if (!val) + return -EINVAL; + y = ilog2(val); + /* x = (val - (1 << y)) >> (y - 2); */ + x = (val - (1ul << y)) << x_w >> y; + + rxy = REG_FIELD_PREP(PKG_PWR_LIM_1_TIME_X, x) | REG_FIELD_PREP(PKG_PWR_LIM_1_TIME_Y, y); + + hwm_locked_with_pm_intel_uncore_rmw(ddat, hwmon->rg.pkg_rapl_limit, + PKG_PWR_LIM_1_TIME, rxy); + return count; +} + +static SENSOR_DEVICE_ATTR(power1_max_interval, 0664, + hwm_power1_max_interval_show, + hwm_power1_max_interval_store, 0); + +static struct attribute *hwm_attributes[] = { + &sensor_dev_attr_power1_max_interval.dev_attr.attr, + NULL +}; + +static umode_t hwm_attributes_visible(struct kobject *kobj, + struct attribute *attr, int index) +{ + struct device *dev = kobj_to_dev(kobj); + struct hwm_drvdata *ddat = dev_get_drvdata(dev); + struct i915_hwmon *hwmon = ddat->hwmon; + + if (attr == &sensor_dev_attr_power1_max_interval.dev_attr.attr) + return i915_mmio_reg_valid(hwmon->rg.pkg_rapl_limit) ? attr->mode : 0; + + return 0; +} + +static const struct attribute_group hwm_attrgroup = { + .attrs = hwm_attributes, + .is_visible = hwm_attributes_visible, +}; + +static const struct attribute_group *hwm_groups[] = { + &hwm_attrgroup, + NULL +}; + +static const struct hwmon_channel_info *hwm_info[] = { + HWMON_CHANNEL_INFO(in, HWMON_I_INPUT), + HWMON_CHANNEL_INFO(power, HWMON_P_MAX | HWMON_P_RATED_MAX | HWMON_P_CRIT), + HWMON_CHANNEL_INFO(energy, HWMON_E_INPUT), + HWMON_CHANNEL_INFO(curr, HWMON_C_CRIT), + NULL +}; + +static const struct hwmon_channel_info *hwm_gt_info[] = { + HWMON_CHANNEL_INFO(energy, HWMON_E_INPUT), + NULL +}; + +/* I1 is exposed as power_crit or as curr_crit depending on bit 31 */ +static int hwm_pcode_read_i1(struct drm_i915_private *i915, u32 *uval) +{ + return snb_pcode_read_p(&i915->uncore, PCODE_POWER_SETUP, + POWER_SETUP_SUBCOMMAND_READ_I1, 0, uval); +} + +static int hwm_pcode_write_i1(struct drm_i915_private *i915, u32 uval) +{ + return snb_pcode_write_p(&i915->uncore, PCODE_POWER_SETUP, + POWER_SETUP_SUBCOMMAND_WRITE_I1, 0, uval); +} + +static umode_t +hwm_in_is_visible(const struct hwm_drvdata *ddat, u32 attr) +{ + struct drm_i915_private *i915 = ddat->uncore->i915; + + switch (attr) { + case hwmon_in_input: + return IS_DG1(i915) || IS_DG2(i915) ? 0444 : 0; + default: + return 0; + } +} + +static int +hwm_in_read(struct hwm_drvdata *ddat, u32 attr, long *val) +{ + struct i915_hwmon *hwmon = ddat->hwmon; + intel_wakeref_t wakeref; + u32 reg_value; + + switch (attr) { + case hwmon_in_input: + with_intel_runtime_pm(ddat->uncore->rpm, wakeref) + reg_value = intel_uncore_read(ddat->uncore, hwmon->rg.gt_perf_status); + /* HW register value in units of 2.5 millivolt */ + *val = DIV_ROUND_CLOSEST(REG_FIELD_GET(GEN12_VOLTAGE_MASK, reg_value) * 25, 10); + return 0; + default: + return -EOPNOTSUPP; + } +} + +static umode_t +hwm_power_is_visible(const struct hwm_drvdata *ddat, u32 attr, int chan) +{ + struct drm_i915_private *i915 = ddat->uncore->i915; + struct i915_hwmon *hwmon = ddat->hwmon; + u32 uval; + + switch (attr) { + case hwmon_power_max: + return i915_mmio_reg_valid(hwmon->rg.pkg_rapl_limit) ? 0664 : 0; + case hwmon_power_rated_max: + return i915_mmio_reg_valid(hwmon->rg.pkg_power_sku) ? 0444 : 0; + case hwmon_power_crit: + return (hwm_pcode_read_i1(i915, &uval) || + !(uval & POWER_SETUP_I1_WATTS)) ? 0 : 0644; + default: + return 0; + } +} + +static int +hwm_power_read(struct hwm_drvdata *ddat, u32 attr, int chan, long *val) +{ + struct i915_hwmon *hwmon = ddat->hwmon; + int ret; + u32 uval; + + switch (attr) { + case hwmon_power_max: + *val = hwm_field_read_and_scale(ddat, + hwmon->rg.pkg_rapl_limit, + PKG_PWR_LIM_1, + hwmon->scl_shift_power, + SF_POWER); + return 0; + case hwmon_power_rated_max: + *val = hwm_field_read_and_scale(ddat, + hwmon->rg.pkg_power_sku, + PKG_PKG_TDP, + hwmon->scl_shift_power, + SF_POWER); + return 0; + case hwmon_power_crit: + ret = hwm_pcode_read_i1(ddat->uncore->i915, &uval); + if (ret) + return ret; + if (!(uval & POWER_SETUP_I1_WATTS)) + return -ENODEV; + *val = mul_u64_u32_shr(REG_FIELD_GET(POWER_SETUP_I1_DATA_MASK, uval), + SF_POWER, POWER_SETUP_I1_SHIFT); + return 0; + default: + return -EOPNOTSUPP; + } +} + +static int +hwm_power_write(struct hwm_drvdata *ddat, u32 attr, int chan, long val) +{ + struct i915_hwmon *hwmon = ddat->hwmon; + u32 uval; + + switch (attr) { + case hwmon_power_max: + hwm_field_scale_and_write(ddat, + hwmon->rg.pkg_rapl_limit, + hwmon->scl_shift_power, + SF_POWER, val); + return 0; + case hwmon_power_crit: + uval = DIV_ROUND_CLOSEST_ULL(val << POWER_SETUP_I1_SHIFT, SF_POWER); + return hwm_pcode_write_i1(ddat->uncore->i915, uval); + default: + return -EOPNOTSUPP; + } +} + +static umode_t +hwm_energy_is_visible(const struct hwm_drvdata *ddat, u32 attr) +{ + struct i915_hwmon *hwmon = ddat->hwmon; + i915_reg_t rgaddr; + + switch (attr) { + case hwmon_energy_input: + if (ddat->gt_n >= 0) + rgaddr = hwmon->rg.energy_status_tile; + else + rgaddr = hwmon->rg.energy_status_all; + return i915_mmio_reg_valid(rgaddr) ? 0444 : 0; + default: + return 0; + } +} + +static int +hwm_energy_read(struct hwm_drvdata *ddat, u32 attr, long *val) +{ + switch (attr) { + case hwmon_energy_input: + hwm_energy(ddat, val); + return 0; + default: + return -EOPNOTSUPP; + } +} + +static umode_t +hwm_curr_is_visible(const struct hwm_drvdata *ddat, u32 attr) +{ + struct drm_i915_private *i915 = ddat->uncore->i915; + u32 uval; + + switch (attr) { + case hwmon_curr_crit: + return (hwm_pcode_read_i1(i915, &uval) || + (uval & POWER_SETUP_I1_WATTS)) ? 0 : 0644; + default: + return 0; + } +} + +static int +hwm_curr_read(struct hwm_drvdata *ddat, u32 attr, long *val) +{ + int ret; + u32 uval; + + switch (attr) { + case hwmon_curr_crit: + ret = hwm_pcode_read_i1(ddat->uncore->i915, &uval); + if (ret) + return ret; + if (uval & POWER_SETUP_I1_WATTS) + return -ENODEV; + *val = mul_u64_u32_shr(REG_FIELD_GET(POWER_SETUP_I1_DATA_MASK, uval), + SF_CURR, POWER_SETUP_I1_SHIFT); + return 0; + default: + return -EOPNOTSUPP; + } +} + +static int +hwm_curr_write(struct hwm_drvdata *ddat, u32 attr, long val) +{ + u32 uval; + + switch (attr) { + case hwmon_curr_crit: + uval = DIV_ROUND_CLOSEST_ULL(val << POWER_SETUP_I1_SHIFT, SF_CURR); + return hwm_pcode_write_i1(ddat->uncore->i915, uval); + default: + return -EOPNOTSUPP; + } +} + +static umode_t +hwm_is_visible(const void *drvdata, enum hwmon_sensor_types type, + u32 attr, int channel) +{ + struct hwm_drvdata *ddat = (struct hwm_drvdata *)drvdata; + + switch (type) { + case hwmon_in: + return hwm_in_is_visible(ddat, attr); + case hwmon_power: + return hwm_power_is_visible(ddat, attr, channel); + case hwmon_energy: + return hwm_energy_is_visible(ddat, attr); + case hwmon_curr: + return hwm_curr_is_visible(ddat, attr); + default: + return 0; + } +} + +static int +hwm_read(struct device *dev, enum hwmon_sensor_types type, u32 attr, + int channel, long *val) +{ + struct hwm_drvdata *ddat = dev_get_drvdata(dev); + + switch (type) { + case hwmon_in: + return hwm_in_read(ddat, attr, val); + case hwmon_power: + return hwm_power_read(ddat, attr, channel, val); + case hwmon_energy: + return hwm_energy_read(ddat, attr, val); + case hwmon_curr: + return hwm_curr_read(ddat, attr, val); + default: + return -EOPNOTSUPP; + } +} + +static int +hwm_write(struct device *dev, enum hwmon_sensor_types type, u32 attr, + int channel, long val) +{ + struct hwm_drvdata *ddat = dev_get_drvdata(dev); + + switch (type) { + case hwmon_power: + return hwm_power_write(ddat, attr, channel, val); + case hwmon_curr: + return hwm_curr_write(ddat, attr, val); + default: + return -EOPNOTSUPP; + } +} + +static const struct hwmon_ops hwm_ops = { + .is_visible = hwm_is_visible, + .read = hwm_read, + .write = hwm_write, +}; + +static const struct hwmon_chip_info hwm_chip_info = { + .ops = &hwm_ops, + .info = hwm_info, +}; + +static umode_t +hwm_gt_is_visible(const void *drvdata, enum hwmon_sensor_types type, + u32 attr, int channel) +{ + struct hwm_drvdata *ddat = (struct hwm_drvdata *)drvdata; + + switch (type) { + case hwmon_energy: + return hwm_energy_is_visible(ddat, attr); + default: + return 0; + } +} + +static int +hwm_gt_read(struct device *dev, enum hwmon_sensor_types type, u32 attr, + int channel, long *val) +{ + struct hwm_drvdata *ddat = dev_get_drvdata(dev); + + switch (type) { + case hwmon_energy: + return hwm_energy_read(ddat, attr, val); + default: + return -EOPNOTSUPP; + } +} + +static const struct hwmon_ops hwm_gt_ops = { + .is_visible = hwm_gt_is_visible, + .read = hwm_gt_read, +}; + +static const struct hwmon_chip_info hwm_gt_chip_info = { + .ops = &hwm_gt_ops, + .info = hwm_gt_info, +}; + +static void +hwm_get_preregistration_info(struct drm_i915_private *i915) +{ + struct i915_hwmon *hwmon = i915->hwmon; + struct intel_uncore *uncore = &i915->uncore; + struct hwm_drvdata *ddat = &hwmon->ddat; + intel_wakeref_t wakeref; + u32 val_sku_unit = 0; + struct intel_gt *gt; + long energy; + int i; + + /* Available for all Gen12+/dGfx */ + hwmon->rg.gt_perf_status = GEN12_RPSTAT1; + + if (IS_DG1(i915) || IS_DG2(i915)) { + hwmon->rg.pkg_power_sku_unit = PCU_PACKAGE_POWER_SKU_UNIT; + hwmon->rg.pkg_power_sku = PCU_PACKAGE_POWER_SKU; + hwmon->rg.pkg_rapl_limit = PCU_PACKAGE_RAPL_LIMIT; + hwmon->rg.energy_status_all = PCU_PACKAGE_ENERGY_STATUS; + hwmon->rg.energy_status_tile = INVALID_MMIO_REG; + } else if (IS_XEHPSDV(i915)) { + hwmon->rg.pkg_power_sku_unit = GT0_PACKAGE_POWER_SKU_UNIT; + hwmon->rg.pkg_power_sku = INVALID_MMIO_REG; + hwmon->rg.pkg_rapl_limit = GT0_PACKAGE_RAPL_LIMIT; + hwmon->rg.energy_status_all = GT0_PLATFORM_ENERGY_STATUS; + hwmon->rg.energy_status_tile = GT0_PACKAGE_ENERGY_STATUS; + } else { + hwmon->rg.pkg_power_sku_unit = INVALID_MMIO_REG; + hwmon->rg.pkg_power_sku = INVALID_MMIO_REG; + hwmon->rg.pkg_rapl_limit = INVALID_MMIO_REG; + hwmon->rg.energy_status_all = INVALID_MMIO_REG; + hwmon->rg.energy_status_tile = INVALID_MMIO_REG; + } + + with_intel_runtime_pm(uncore->rpm, wakeref) { + /* + * The contents of register hwmon->rg.pkg_power_sku_unit do not change, + * so read it once and store the shift values. + */ + if (i915_mmio_reg_valid(hwmon->rg.pkg_power_sku_unit)) + val_sku_unit = intel_uncore_read(uncore, + hwmon->rg.pkg_power_sku_unit); + } + + hwmon->scl_shift_power = REG_FIELD_GET(PKG_PWR_UNIT, val_sku_unit); + hwmon->scl_shift_energy = REG_FIELD_GET(PKG_ENERGY_UNIT, val_sku_unit); + hwmon->scl_shift_time = REG_FIELD_GET(PKG_TIME_UNIT, val_sku_unit); + + /* + * Initialize 'struct hwm_energy_info', i.e. set fields to the + * first value of the energy register read + */ + if (i915_mmio_reg_valid(hwmon->rg.energy_status_all)) + hwm_energy(ddat, &energy); + if (i915_mmio_reg_valid(hwmon->rg.energy_status_tile)) { + for_each_gt(gt, i915, i) + hwm_energy(&hwmon->ddat_gt[i], &energy); + } +} + +void i915_hwmon_register(struct drm_i915_private *i915) +{ + struct device *dev = i915->drm.dev; + struct i915_hwmon *hwmon; + struct device *hwmon_dev; + struct hwm_drvdata *ddat; + struct hwm_drvdata *ddat_gt; + struct intel_gt *gt; + int i; + + /* hwmon is available only for dGfx */ + if (!IS_DGFX(i915)) + return; + + hwmon = devm_kzalloc(dev, sizeof(*hwmon), GFP_KERNEL); + if (!hwmon) + return; + + i915->hwmon = hwmon; + mutex_init(&hwmon->hwmon_lock); + ddat = &hwmon->ddat; + + ddat->hwmon = hwmon; + ddat->uncore = &i915->uncore; + snprintf(ddat->name, sizeof(ddat->name), "i915"); + ddat->gt_n = -1; + + for_each_gt(gt, i915, i) { + ddat_gt = hwmon->ddat_gt + i; + + ddat_gt->hwmon = hwmon; + ddat_gt->uncore = gt->uncore; + snprintf(ddat_gt->name, sizeof(ddat_gt->name), "i915_gt%u", i); + ddat_gt->gt_n = i; + } + + hwm_get_preregistration_info(i915); + + /* hwmon_dev points to device hwmon<i> */ + hwmon_dev = devm_hwmon_device_register_with_info(dev, ddat->name, + ddat, + &hwm_chip_info, + hwm_groups); + if (IS_ERR(hwmon_dev)) { + i915->hwmon = NULL; + return; + } + + ddat->hwmon_dev = hwmon_dev; + + for_each_gt(gt, i915, i) { + ddat_gt = hwmon->ddat_gt + i; + /* + * Create per-gt directories only if a per-gt attribute is + * visible. Currently this is only energy + */ + if (!hwm_gt_is_visible(ddat_gt, hwmon_energy, hwmon_energy_input, 0)) + continue; + + hwmon_dev = devm_hwmon_device_register_with_info(dev, ddat_gt->name, + ddat_gt, + &hwm_gt_chip_info, + NULL); + if (!IS_ERR(hwmon_dev)) + ddat_gt->hwmon_dev = hwmon_dev; + } +} + +void i915_hwmon_unregister(struct drm_i915_private *i915) +{ + fetch_and_zero(&i915->hwmon); +} diff --git a/drivers/gpu/drm/i915/i915_hwmon.h b/drivers/gpu/drm/i915/i915_hwmon.h new file mode 100644 index 000000000000..7ca9cf2c34c9 --- /dev/null +++ b/drivers/gpu/drm/i915/i915_hwmon.h @@ -0,0 +1,20 @@ +/* SPDX-License-Identifier: MIT */ + +/* + * Copyright © 2022 Intel Corporation + */ + +#ifndef __I915_HWMON_H__ +#define __I915_HWMON_H__ + +struct drm_i915_private; + +#if IS_REACHABLE(CONFIG_HWMON) +void i915_hwmon_register(struct drm_i915_private *i915); +void i915_hwmon_unregister(struct drm_i915_private *i915); +#else +static inline void i915_hwmon_register(struct drm_i915_private *i915) { }; +static inline void i915_hwmon_unregister(struct drm_i915_private *i915) { }; +#endif + +#endif /* __I915_HWMON_H__ */ diff --git a/drivers/gpu/drm/i915/i915_irq.c b/drivers/gpu/drm/i915/i915_irq.c index 86a42d9e8041..edfe363af838 100644 --- a/drivers/gpu/drm/i915/i915_irq.c +++ b/drivers/gpu/drm/i915/i915_irq.c @@ -28,7 +28,6 @@ #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt -#include <linux/circ_buf.h> #include <linux/slab.h> #include <linux/sysrq.h> @@ -248,7 +247,7 @@ void gen3_irq_reset(struct intel_uncore *uncore, i915_reg_t imr, intel_uncore_posting_read(uncore, iir); } -void gen2_irq_reset(struct intel_uncore *uncore) +static void gen2_irq_reset(struct intel_uncore *uncore) { intel_uncore_write16(uncore, GEN2_IMR, 0xffff); intel_uncore_posting_read16(uncore, GEN2_IMR); @@ -309,8 +308,8 @@ void gen3_irq_init(struct intel_uncore *uncore, intel_uncore_posting_read(uncore, imr); } -void gen2_irq_init(struct intel_uncore *uncore, - u32 imr_val, u32 ier_val) +static void gen2_irq_init(struct intel_uncore *uncore, + u32 imr_val, u32 ier_val) { gen2_assert_iir_is_zero(uncore); @@ -325,15 +324,10 @@ i915_hotplug_interrupt_update_locked(struct drm_i915_private *dev_priv, u32 mask, u32 bits) { - u32 val; - lockdep_assert_held(&dev_priv->irq_lock); drm_WARN_ON(&dev_priv->drm, bits & ~mask); - val = intel_uncore_read(&dev_priv->uncore, PORT_HOTPLUG_EN); - val &= ~mask; - val |= bits; - intel_uncore_write(&dev_priv->uncore, PORT_HOTPLUG_EN, val); + intel_uncore_rmw(&dev_priv->uncore, PORT_HOTPLUG_EN, mask, bits); } /** @@ -1057,8 +1051,8 @@ static void ivb_parity_work(struct work_struct *work) if (drm_WARN_ON(&dev_priv->drm, !dev_priv->l3_parity.which_slice)) goto out; - misccpctl = intel_uncore_read(&dev_priv->uncore, GEN7_MISCCPCTL); - intel_uncore_write(&dev_priv->uncore, GEN7_MISCCPCTL, misccpctl & ~GEN7_DOP_CLOCK_GATE_ENABLE); + misccpctl = intel_uncore_rmw(&dev_priv->uncore, GEN7_MISCCPCTL, + GEN7_DOP_CLOCK_GATE_ENABLE, 0); intel_uncore_posting_read(&dev_priv->uncore, GEN7_MISCCPCTL); while ((slice = ffs(dev_priv->l3_parity.which_slice)) != 0) { @@ -1091,8 +1085,9 @@ static void ivb_parity_work(struct work_struct *work) kobject_uevent_env(&dev_priv->drm.primary->kdev->kobj, KOBJ_CHANGE, parity_event); - DRM_DEBUG("Parity error: Slice = %d, Row = %d, Bank = %d, Sub bank = %d.\n", - slice, row, bank, subbank); + drm_dbg(&dev_priv->drm, + "Parity error: Slice = %d, Row = %d, Bank = %d, Sub bank = %d.\n", + slice, row, bank, subbank); kfree(parity_event[4]); kfree(parity_event[3]); @@ -1689,8 +1684,7 @@ static irqreturn_t valleyview_irq_handler(int irq, void *arg) * bits this time around. */ intel_uncore_write(&dev_priv->uncore, VLV_MASTER_IER, 0); - ier = intel_uncore_read(&dev_priv->uncore, VLV_IER); - intel_uncore_write(&dev_priv->uncore, VLV_IER, 0); + ier = intel_uncore_rmw(&dev_priv->uncore, VLV_IER, ~0, 0); if (gt_iir) intel_uncore_write(&dev_priv->uncore, GTIIR, gt_iir); @@ -1775,8 +1769,7 @@ static irqreturn_t cherryview_irq_handler(int irq, void *arg) * bits this time around. */ intel_uncore_write(&dev_priv->uncore, GEN8_MASTER_IRQ, 0); - ier = intel_uncore_read(&dev_priv->uncore, VLV_IER); - intel_uncore_write(&dev_priv->uncore, VLV_IER, 0); + ier = intel_uncore_rmw(&dev_priv->uncore, VLV_IER, ~0, 0); gen8_gt_irq_handler(to_gt(dev_priv), master_ctl); @@ -1981,8 +1974,7 @@ static void icp_irq_handler(struct drm_i915_private *dev_priv, u32 pch_iir) if (ddi_hotplug_trigger) { u32 dig_hotplug_reg; - dig_hotplug_reg = intel_uncore_read(&dev_priv->uncore, SHOTPLUG_CTL_DDI); - intel_uncore_write(&dev_priv->uncore, SHOTPLUG_CTL_DDI, dig_hotplug_reg); + dig_hotplug_reg = intel_uncore_rmw(&dev_priv->uncore, SHOTPLUG_CTL_DDI, 0, 0); intel_get_hpd_pins(dev_priv, &pin_mask, &long_mask, ddi_hotplug_trigger, dig_hotplug_reg, @@ -1993,8 +1985,7 @@ static void icp_irq_handler(struct drm_i915_private *dev_priv, u32 pch_iir) if (tc_hotplug_trigger) { u32 dig_hotplug_reg; - dig_hotplug_reg = intel_uncore_read(&dev_priv->uncore, SHOTPLUG_CTL_TC); - intel_uncore_write(&dev_priv->uncore, SHOTPLUG_CTL_TC, dig_hotplug_reg); + dig_hotplug_reg = intel_uncore_rmw(&dev_priv->uncore, SHOTPLUG_CTL_TC, 0, 0); intel_get_hpd_pins(dev_priv, &pin_mask, &long_mask, tc_hotplug_trigger, dig_hotplug_reg, @@ -2019,8 +2010,7 @@ static void spt_irq_handler(struct drm_i915_private *dev_priv, u32 pch_iir) if (hotplug_trigger) { u32 dig_hotplug_reg; - dig_hotplug_reg = intel_uncore_read(&dev_priv->uncore, PCH_PORT_HOTPLUG); - intel_uncore_write(&dev_priv->uncore, PCH_PORT_HOTPLUG, dig_hotplug_reg); + dig_hotplug_reg = intel_uncore_rmw(&dev_priv->uncore, PCH_PORT_HOTPLUG, 0, 0); intel_get_hpd_pins(dev_priv, &pin_mask, &long_mask, hotplug_trigger, dig_hotplug_reg, @@ -2031,8 +2021,7 @@ static void spt_irq_handler(struct drm_i915_private *dev_priv, u32 pch_iir) if (hotplug2_trigger) { u32 dig_hotplug_reg; - dig_hotplug_reg = intel_uncore_read(&dev_priv->uncore, PCH_PORT_HOTPLUG2); - intel_uncore_write(&dev_priv->uncore, PCH_PORT_HOTPLUG2, dig_hotplug_reg); + dig_hotplug_reg = intel_uncore_rmw(&dev_priv->uncore, PCH_PORT_HOTPLUG2, 0, 0); intel_get_hpd_pins(dev_priv, &pin_mask, &long_mask, hotplug2_trigger, dig_hotplug_reg, @@ -2052,8 +2041,7 @@ static void ilk_hpd_irq_handler(struct drm_i915_private *dev_priv, { u32 dig_hotplug_reg, pin_mask = 0, long_mask = 0; - dig_hotplug_reg = intel_uncore_read(&dev_priv->uncore, DIGITAL_PORT_HOTPLUG_CNTRL); - intel_uncore_write(&dev_priv->uncore, DIGITAL_PORT_HOTPLUG_CNTRL, dig_hotplug_reg); + dig_hotplug_reg = intel_uncore_rmw(&dev_priv->uncore, DIGITAL_PORT_HOTPLUG_CNTRL, 0, 0); intel_get_hpd_pins(dev_priv, &pin_mask, &long_mask, hotplug_trigger, dig_hotplug_reg, @@ -2232,8 +2220,7 @@ static void bxt_hpd_irq_handler(struct drm_i915_private *dev_priv, { u32 dig_hotplug_reg, pin_mask = 0, long_mask = 0; - dig_hotplug_reg = intel_uncore_read(&dev_priv->uncore, PCH_PORT_HOTPLUG); - intel_uncore_write(&dev_priv->uncore, PCH_PORT_HOTPLUG, dig_hotplug_reg); + dig_hotplug_reg = intel_uncore_rmw(&dev_priv->uncore, PCH_PORT_HOTPLUG, 0, 0); intel_get_hpd_pins(dev_priv, &pin_mask, &long_mask, hotplug_trigger, dig_hotplug_reg, @@ -2252,8 +2239,7 @@ static void gen11_hpd_irq_handler(struct drm_i915_private *dev_priv, u32 iir) if (trigger_tc) { u32 dig_hotplug_reg; - dig_hotplug_reg = intel_uncore_read(&dev_priv->uncore, GEN11_TC_HOTPLUG_CTL); - intel_uncore_write(&dev_priv->uncore, GEN11_TC_HOTPLUG_CTL, dig_hotplug_reg); + dig_hotplug_reg = intel_uncore_rmw(&dev_priv->uncore, GEN11_TC_HOTPLUG_CTL, 0, 0); intel_get_hpd_pins(dev_priv, &pin_mask, &long_mask, trigger_tc, dig_hotplug_reg, @@ -2264,8 +2250,7 @@ static void gen11_hpd_irq_handler(struct drm_i915_private *dev_priv, u32 iir) if (trigger_tbt) { u32 dig_hotplug_reg; - dig_hotplug_reg = intel_uncore_read(&dev_priv->uncore, GEN11_TBT_HOTPLUG_CTL); - intel_uncore_write(&dev_priv->uncore, GEN11_TBT_HOTPLUG_CTL, dig_hotplug_reg); + dig_hotplug_reg = intel_uncore_rmw(&dev_priv->uncore, GEN11_TBT_HOTPLUG_CTL, 0, 0); intel_get_hpd_pins(dev_priv, &pin_mask, &long_mask, trigger_tbt, dig_hotplug_reg, @@ -2355,8 +2340,7 @@ gen8_de_misc_irq_handler(struct drm_i915_private *dev_priv, u32 iir) else iir_reg = EDP_PSR_IIR; - psr_iir = intel_uncore_read(&dev_priv->uncore, iir_reg); - intel_uncore_write(&dev_priv->uncore, iir_reg, psr_iir); + psr_iir = intel_uncore_rmw(&dev_priv->uncore, iir_reg, 0, 0); if (psr_iir) found = true; @@ -2426,8 +2410,7 @@ static void gen11_dsi_te_interrupt_handler(struct drm_i915_private *dev_priv, /* clear TE in dsi IIR */ port = (te_trigger & DSI1_TE) ? PORT_B : PORT_A; - tmp = intel_uncore_read(&dev_priv->uncore, DSI_INTR_IDENT_REG(port)); - intel_uncore_write(&dev_priv->uncore, DSI_INTR_IDENT_REG(port), tmp); + tmp = intel_uncore_rmw(&dev_priv->uncore, DSI_INTR_IDENT_REG(port), 0, 0); } static u32 gen8_de_pipe_flip_done_mask(struct drm_i915_private *i915) @@ -2791,7 +2774,8 @@ static irqreturn_t dg1_irq_handler(int irq, void *arg) master_ctl = raw_reg_read(regs, GEN11_GFX_MSTR_IRQ); raw_reg_write(regs, GEN11_GFX_MSTR_IRQ, master_ctl); } else { - DRM_ERROR("Tile not supported: 0x%08x\n", master_tile_ctl); + drm_err(&i915->drm, "Tile not supported: 0x%08x\n", + master_tile_ctl); dg1_master_intr_enable(regs); return IRQ_NONE; } @@ -2884,7 +2868,6 @@ static bool gen11_dsi_configure_te(struct intel_crtc *intel_crtc, { struct drm_i915_private *dev_priv = to_i915(intel_crtc->base.dev); enum port port; - u32 tmp; if (!(intel_crtc->mode_flags & (I915_MODE_FLAG_DSI_USE_TE1 | I915_MODE_FLAG_DSI_USE_TE0))) @@ -2896,16 +2879,10 @@ static bool gen11_dsi_configure_te(struct intel_crtc *intel_crtc, else port = PORT_A; - tmp = intel_uncore_read(&dev_priv->uncore, DSI_INTR_MASK_REG(port)); - if (enable) - tmp &= ~DSI_TE_EVENT; - else - tmp |= DSI_TE_EVENT; - - intel_uncore_write(&dev_priv->uncore, DSI_INTR_MASK_REG(port), tmp); + intel_uncore_rmw(&dev_priv->uncore, DSI_INTR_MASK_REG(port), DSI_TE_EVENT, + enable ? 0 : DSI_TE_EVENT); - tmp = intel_uncore_read(&dev_priv->uncore, DSI_INTR_IDENT_REG(port)); - intel_uncore_write(&dev_priv->uncore, DSI_INTR_IDENT_REG(port), tmp); + intel_uncore_rmw(&dev_priv->uncore, DSI_INTR_IDENT_REG(port), 0, 0); return true; } @@ -3020,7 +2997,7 @@ static void vlv_display_irq_reset(struct drm_i915_private *dev_priv) intel_uncore_write(uncore, DPINVGTT, DPINVGTT_STATUS_MASK_VLV); i915_hotplug_interrupt_update_locked(dev_priv, 0xffffffff, 0); - intel_uncore_write(uncore, PORT_HOTPLUG_STAT, intel_uncore_read(&dev_priv->uncore, PORT_HOTPLUG_STAT)); + intel_uncore_rmw(uncore, PORT_HOTPLUG_STAT, 0, 0); i9xx_pipestat_irq_reset(dev_priv); @@ -3118,7 +3095,7 @@ static void gen8_irq_reset(struct drm_i915_private *dev_priv) { struct intel_uncore *uncore = &dev_priv->uncore; - gen8_master_intr_disable(dev_priv->uncore.regs); + gen8_master_intr_disable(uncore->regs); gen8_gt_irq_reset(to_gt(dev_priv)); gen8_display_irq_reset(dev_priv); @@ -3250,7 +3227,7 @@ static void cherryview_irq_reset(struct drm_i915_private *dev_priv) { struct intel_uncore *uncore = &dev_priv->uncore; - intel_uncore_write(&dev_priv->uncore, GEN8_MASTER_IRQ, 0); + intel_uncore_write(uncore, GEN8_MASTER_IRQ, 0); intel_uncore_posting_read(&dev_priv->uncore, GEN8_MASTER_IRQ); gen8_gt_irq_reset(to_gt(dev_priv)); @@ -3290,23 +3267,20 @@ static u32 ibx_hotplug_enables(struct drm_i915_private *i915, static void ibx_hpd_detection_setup(struct drm_i915_private *dev_priv) { - u32 hotplug; - /* * Enable digital hotplug on the PCH, and configure the DP short pulse * duration to 2ms (which is the minimum in the Display Port spec). * The pulse duration bits are reserved on LPT+. */ - hotplug = intel_uncore_read(&dev_priv->uncore, PCH_PORT_HOTPLUG); - hotplug &= ~(PORTA_HOTPLUG_ENABLE | - PORTB_HOTPLUG_ENABLE | - PORTC_HOTPLUG_ENABLE | - PORTD_HOTPLUG_ENABLE | - PORTB_PULSE_DURATION_MASK | - PORTC_PULSE_DURATION_MASK | - PORTD_PULSE_DURATION_MASK); - hotplug |= intel_hpd_hotplug_enables(dev_priv, ibx_hotplug_enables); - intel_uncore_write(&dev_priv->uncore, PCH_PORT_HOTPLUG, hotplug); + intel_uncore_rmw(&dev_priv->uncore, PCH_PORT_HOTPLUG, + PORTA_HOTPLUG_ENABLE | + PORTB_HOTPLUG_ENABLE | + PORTC_HOTPLUG_ENABLE | + PORTD_HOTPLUG_ENABLE | + PORTB_PULSE_DURATION_MASK | + PORTC_PULSE_DURATION_MASK | + PORTD_PULSE_DURATION_MASK, + intel_hpd_hotplug_enables(dev_priv, ibx_hotplug_enables)); } static void ibx_hpd_irq_setup(struct drm_i915_private *dev_priv) @@ -3353,30 +3327,24 @@ static u32 icp_tc_hotplug_enables(struct drm_i915_private *i915, static void icp_ddi_hpd_detection_setup(struct drm_i915_private *dev_priv) { - u32 hotplug; - - hotplug = intel_uncore_read(&dev_priv->uncore, SHOTPLUG_CTL_DDI); - hotplug &= ~(SHOTPLUG_CTL_DDI_HPD_ENABLE(HPD_PORT_A) | - SHOTPLUG_CTL_DDI_HPD_ENABLE(HPD_PORT_B) | - SHOTPLUG_CTL_DDI_HPD_ENABLE(HPD_PORT_C) | - SHOTPLUG_CTL_DDI_HPD_ENABLE(HPD_PORT_D)); - hotplug |= intel_hpd_hotplug_enables(dev_priv, icp_ddi_hotplug_enables); - intel_uncore_write(&dev_priv->uncore, SHOTPLUG_CTL_DDI, hotplug); + intel_uncore_rmw(&dev_priv->uncore, SHOTPLUG_CTL_DDI, + SHOTPLUG_CTL_DDI_HPD_ENABLE(HPD_PORT_A) | + SHOTPLUG_CTL_DDI_HPD_ENABLE(HPD_PORT_B) | + SHOTPLUG_CTL_DDI_HPD_ENABLE(HPD_PORT_C) | + SHOTPLUG_CTL_DDI_HPD_ENABLE(HPD_PORT_D), + intel_hpd_hotplug_enables(dev_priv, icp_ddi_hotplug_enables)); } static void icp_tc_hpd_detection_setup(struct drm_i915_private *dev_priv) { - u32 hotplug; - - hotplug = intel_uncore_read(&dev_priv->uncore, SHOTPLUG_CTL_TC); - hotplug &= ~(ICP_TC_HPD_ENABLE(HPD_PORT_TC1) | - ICP_TC_HPD_ENABLE(HPD_PORT_TC2) | - ICP_TC_HPD_ENABLE(HPD_PORT_TC3) | - ICP_TC_HPD_ENABLE(HPD_PORT_TC4) | - ICP_TC_HPD_ENABLE(HPD_PORT_TC5) | - ICP_TC_HPD_ENABLE(HPD_PORT_TC6)); - hotplug |= intel_hpd_hotplug_enables(dev_priv, icp_tc_hotplug_enables); - intel_uncore_write(&dev_priv->uncore, SHOTPLUG_CTL_TC, hotplug); + intel_uncore_rmw(&dev_priv->uncore, SHOTPLUG_CTL_TC, + ICP_TC_HPD_ENABLE(HPD_PORT_TC1) | + ICP_TC_HPD_ENABLE(HPD_PORT_TC2) | + ICP_TC_HPD_ENABLE(HPD_PORT_TC3) | + ICP_TC_HPD_ENABLE(HPD_PORT_TC4) | + ICP_TC_HPD_ENABLE(HPD_PORT_TC5) | + ICP_TC_HPD_ENABLE(HPD_PORT_TC6), + intel_hpd_hotplug_enables(dev_priv, icp_tc_hotplug_enables)); } static void icp_hpd_irq_setup(struct drm_i915_private *dev_priv) @@ -3411,62 +3379,54 @@ static u32 gen11_hotplug_enables(struct drm_i915_private *i915, } } -static void dg1_hpd_irq_setup(struct drm_i915_private *dev_priv) +static void dg1_hpd_invert(struct drm_i915_private *i915) { - u32 val; - - val = intel_uncore_read(&dev_priv->uncore, SOUTH_CHICKEN1); - val |= (INVERT_DDIA_HPD | - INVERT_DDIB_HPD | - INVERT_DDIC_HPD | - INVERT_DDID_HPD); - intel_uncore_write(&dev_priv->uncore, SOUTH_CHICKEN1, val); + u32 val = (INVERT_DDIA_HPD | + INVERT_DDIB_HPD | + INVERT_DDIC_HPD | + INVERT_DDID_HPD); + intel_uncore_rmw(&i915->uncore, SOUTH_CHICKEN1, 0, val); +} +static void dg1_hpd_irq_setup(struct drm_i915_private *dev_priv) +{ + dg1_hpd_invert(dev_priv); icp_hpd_irq_setup(dev_priv); } static void gen11_tc_hpd_detection_setup(struct drm_i915_private *dev_priv) { - u32 hotplug; - - hotplug = intel_uncore_read(&dev_priv->uncore, GEN11_TC_HOTPLUG_CTL); - hotplug &= ~(GEN11_HOTPLUG_CTL_ENABLE(HPD_PORT_TC1) | - GEN11_HOTPLUG_CTL_ENABLE(HPD_PORT_TC2) | - GEN11_HOTPLUG_CTL_ENABLE(HPD_PORT_TC3) | - GEN11_HOTPLUG_CTL_ENABLE(HPD_PORT_TC4) | - GEN11_HOTPLUG_CTL_ENABLE(HPD_PORT_TC5) | - GEN11_HOTPLUG_CTL_ENABLE(HPD_PORT_TC6)); - hotplug |= intel_hpd_hotplug_enables(dev_priv, gen11_hotplug_enables); - intel_uncore_write(&dev_priv->uncore, GEN11_TC_HOTPLUG_CTL, hotplug); + intel_uncore_rmw(&dev_priv->uncore, GEN11_TC_HOTPLUG_CTL, + GEN11_HOTPLUG_CTL_ENABLE(HPD_PORT_TC1) | + GEN11_HOTPLUG_CTL_ENABLE(HPD_PORT_TC2) | + GEN11_HOTPLUG_CTL_ENABLE(HPD_PORT_TC3) | + GEN11_HOTPLUG_CTL_ENABLE(HPD_PORT_TC4) | + GEN11_HOTPLUG_CTL_ENABLE(HPD_PORT_TC5) | + GEN11_HOTPLUG_CTL_ENABLE(HPD_PORT_TC6), + intel_hpd_hotplug_enables(dev_priv, gen11_hotplug_enables)); } static void gen11_tbt_hpd_detection_setup(struct drm_i915_private *dev_priv) { - u32 hotplug; - - hotplug = intel_uncore_read(&dev_priv->uncore, GEN11_TBT_HOTPLUG_CTL); - hotplug &= ~(GEN11_HOTPLUG_CTL_ENABLE(HPD_PORT_TC1) | - GEN11_HOTPLUG_CTL_ENABLE(HPD_PORT_TC2) | - GEN11_HOTPLUG_CTL_ENABLE(HPD_PORT_TC3) | - GEN11_HOTPLUG_CTL_ENABLE(HPD_PORT_TC4) | - GEN11_HOTPLUG_CTL_ENABLE(HPD_PORT_TC5) | - GEN11_HOTPLUG_CTL_ENABLE(HPD_PORT_TC6)); - hotplug |= intel_hpd_hotplug_enables(dev_priv, gen11_hotplug_enables); - intel_uncore_write(&dev_priv->uncore, GEN11_TBT_HOTPLUG_CTL, hotplug); + intel_uncore_rmw(&dev_priv->uncore, GEN11_TBT_HOTPLUG_CTL, + GEN11_HOTPLUG_CTL_ENABLE(HPD_PORT_TC1) | + GEN11_HOTPLUG_CTL_ENABLE(HPD_PORT_TC2) | + GEN11_HOTPLUG_CTL_ENABLE(HPD_PORT_TC3) | + GEN11_HOTPLUG_CTL_ENABLE(HPD_PORT_TC4) | + GEN11_HOTPLUG_CTL_ENABLE(HPD_PORT_TC5) | + GEN11_HOTPLUG_CTL_ENABLE(HPD_PORT_TC6), + intel_hpd_hotplug_enables(dev_priv, gen11_hotplug_enables)); } static void gen11_hpd_irq_setup(struct drm_i915_private *dev_priv) { u32 hotplug_irqs, enabled_irqs; - u32 val; enabled_irqs = intel_hpd_enabled_irqs(dev_priv, dev_priv->display.hotplug.hpd); hotplug_irqs = intel_hpd_hotplug_irqs(dev_priv, dev_priv->display.hotplug.hpd); - val = intel_uncore_read(&dev_priv->uncore, GEN11_DE_HPD_IMR); - val &= ~hotplug_irqs; - val |= ~enabled_irqs & hotplug_irqs; - intel_uncore_write(&dev_priv->uncore, GEN11_DE_HPD_IMR, val); + intel_uncore_rmw(&dev_priv->uncore, GEN11_DE_HPD_IMR, hotplug_irqs, + ~enabled_irqs & hotplug_irqs); intel_uncore_posting_read(&dev_priv->uncore, GEN11_DE_HPD_IMR); gen11_tc_hpd_detection_setup(dev_priv); @@ -3506,29 +3466,22 @@ static u32 spt_hotplug2_enables(struct drm_i915_private *i915, static void spt_hpd_detection_setup(struct drm_i915_private *dev_priv) { - u32 val, hotplug; - /* Display WA #1179 WaHardHangonHotPlug: cnp */ if (HAS_PCH_CNP(dev_priv)) { - val = intel_uncore_read(&dev_priv->uncore, SOUTH_CHICKEN1); - val &= ~CHASSIS_CLK_REQ_DURATION_MASK; - val |= CHASSIS_CLK_REQ_DURATION(0xf); - intel_uncore_write(&dev_priv->uncore, SOUTH_CHICKEN1, val); + intel_uncore_rmw(&dev_priv->uncore, SOUTH_CHICKEN1, CHASSIS_CLK_REQ_DURATION_MASK, + CHASSIS_CLK_REQ_DURATION(0xf)); } /* Enable digital hotplug on the PCH */ - hotplug = intel_uncore_read(&dev_priv->uncore, PCH_PORT_HOTPLUG); - hotplug &= ~(PORTA_HOTPLUG_ENABLE | - PORTB_HOTPLUG_ENABLE | - PORTC_HOTPLUG_ENABLE | - PORTD_HOTPLUG_ENABLE); - hotplug |= intel_hpd_hotplug_enables(dev_priv, spt_hotplug_enables); - intel_uncore_write(&dev_priv->uncore, PCH_PORT_HOTPLUG, hotplug); + intel_uncore_rmw(&dev_priv->uncore, PCH_PORT_HOTPLUG, + PORTA_HOTPLUG_ENABLE | + PORTB_HOTPLUG_ENABLE | + PORTC_HOTPLUG_ENABLE | + PORTD_HOTPLUG_ENABLE, + intel_hpd_hotplug_enables(dev_priv, spt_hotplug_enables)); - hotplug = intel_uncore_read(&dev_priv->uncore, PCH_PORT_HOTPLUG2); - hotplug &= ~PORTE_HOTPLUG_ENABLE; - hotplug |= intel_hpd_hotplug_enables(dev_priv, spt_hotplug2_enables); - intel_uncore_write(&dev_priv->uncore, PCH_PORT_HOTPLUG2, hotplug); + intel_uncore_rmw(&dev_priv->uncore, PCH_PORT_HOTPLUG2, PORTE_HOTPLUG_ENABLE, + intel_hpd_hotplug_enables(dev_priv, spt_hotplug2_enables)); } static void spt_hpd_irq_setup(struct drm_i915_private *dev_priv) @@ -3560,18 +3513,14 @@ static u32 ilk_hotplug_enables(struct drm_i915_private *i915, static void ilk_hpd_detection_setup(struct drm_i915_private *dev_priv) { - u32 hotplug; - /* * Enable digital hotplug on the CPU, and configure the DP short pulse * duration to 2ms (which is the minimum in the Display Port spec) * The pulse duration bits are reserved on HSW+. */ - hotplug = intel_uncore_read(&dev_priv->uncore, DIGITAL_PORT_HOTPLUG_CNTRL); - hotplug &= ~(DIGITAL_PORTA_HOTPLUG_ENABLE | - DIGITAL_PORTA_PULSE_DURATION_MASK); - hotplug |= intel_hpd_hotplug_enables(dev_priv, ilk_hotplug_enables); - intel_uncore_write(&dev_priv->uncore, DIGITAL_PORT_HOTPLUG_CNTRL, hotplug); + intel_uncore_rmw(&dev_priv->uncore, DIGITAL_PORT_HOTPLUG_CNTRL, + DIGITAL_PORTA_HOTPLUG_ENABLE | DIGITAL_PORTA_PULSE_DURATION_MASK, + intel_hpd_hotplug_enables(dev_priv, ilk_hotplug_enables)); } static void ilk_hpd_irq_setup(struct drm_i915_private *dev_priv) @@ -3619,17 +3568,12 @@ static u32 bxt_hotplug_enables(struct drm_i915_private *i915, static void bxt_hpd_detection_setup(struct drm_i915_private *dev_priv) { - u32 hotplug; - - hotplug = intel_uncore_read(&dev_priv->uncore, PCH_PORT_HOTPLUG); - hotplug &= ~(PORTA_HOTPLUG_ENABLE | - PORTB_HOTPLUG_ENABLE | - PORTC_HOTPLUG_ENABLE | - BXT_DDIA_HPD_INVERT | - BXT_DDIB_HPD_INVERT | - BXT_DDIC_HPD_INVERT); - hotplug |= intel_hpd_hotplug_enables(dev_priv, bxt_hotplug_enables); - intel_uncore_write(&dev_priv->uncore, PCH_PORT_HOTPLUG, hotplug); + intel_uncore_rmw(&dev_priv->uncore, PCH_PORT_HOTPLUG, + PORTA_HOTPLUG_ENABLE | + PORTB_HOTPLUG_ENABLE | + PORTC_HOTPLUG_ENABLE | + BXT_DDI_HPD_INVERT_MASK, + intel_hpd_hotplug_enables(dev_priv, bxt_hotplug_enables)); } static void bxt_hpd_irq_setup(struct drm_i915_private *dev_priv) @@ -3928,7 +3872,7 @@ static void i8xx_irq_reset(struct drm_i915_private *dev_priv) i9xx_pipestat_irq_reset(dev_priv); - GEN2_IRQ_RESET(uncore); + gen2_irq_reset(uncore); dev_priv->irq_mask = ~0u; } @@ -3954,7 +3898,7 @@ static void i8xx_irq_postinstall(struct drm_i915_private *dev_priv) I915_MASTER_ERROR_INTERRUPT | I915_USER_INTERRUPT; - GEN2_IRQ_INIT(uncore, dev_priv->irq_mask, enable_mask); + gen2_irq_init(uncore, dev_priv->irq_mask, enable_mask); /* Interrupt setup is already guaranteed to be single-threaded, this is * just to make the assert_spin_locked check happy. */ @@ -3997,7 +3941,7 @@ static void i8xx_error_irq_ack(struct drm_i915_private *i915, static void i8xx_error_irq_handler(struct drm_i915_private *dev_priv, u16 eir, u16 eir_stuck) { - DRM_DEBUG("Master Error: EIR 0x%04x\n", eir); + drm_dbg(&dev_priv->drm, "Master Error: EIR 0x%04x\n", eir); if (eir_stuck) drm_dbg(&dev_priv->drm, "EIR stuck: 0x%04x, masked\n", @@ -4009,9 +3953,7 @@ static void i9xx_error_irq_ack(struct drm_i915_private *dev_priv, { u32 emr; - *eir = intel_uncore_read(&dev_priv->uncore, EIR); - - intel_uncore_write(&dev_priv->uncore, EIR, *eir); + *eir = intel_uncore_rmw(&dev_priv->uncore, EIR, 0, 0); *eir_stuck = intel_uncore_read(&dev_priv->uncore, EIR); if (*eir_stuck == 0) @@ -4027,15 +3969,14 @@ static void i9xx_error_irq_ack(struct drm_i915_private *dev_priv, * (or by a GPU reset) so we mask any bit that * remains set. */ - emr = intel_uncore_read(&dev_priv->uncore, EMR); - intel_uncore_write(&dev_priv->uncore, EMR, 0xffffffff); + emr = intel_uncore_rmw(&dev_priv->uncore, EMR, ~0, 0xffffffff); intel_uncore_write(&dev_priv->uncore, EMR, emr | *eir_stuck); } static void i9xx_error_irq_handler(struct drm_i915_private *dev_priv, u32 eir, u32 eir_stuck) { - DRM_DEBUG("Master Error, EIR 0x%08x\n", eir); + drm_dbg(&dev_priv->drm, "Master Error, EIR 0x%08x\n", eir); if (eir_stuck) drm_dbg(&dev_priv->drm, "EIR stuck: 0x%08x, masked\n", @@ -4095,7 +4036,7 @@ static void i915_irq_reset(struct drm_i915_private *dev_priv) if (I915_HAS_HOTPLUG(dev_priv)) { i915_hotplug_interrupt_update(dev_priv, 0xffffffff, 0); - intel_uncore_write(&dev_priv->uncore, PORT_HOTPLUG_STAT, intel_uncore_read(&dev_priv->uncore, PORT_HOTPLUG_STAT)); + intel_uncore_rmw(&dev_priv->uncore, PORT_HOTPLUG_STAT, 0, 0); } i9xx_pipestat_irq_reset(dev_priv); @@ -4109,8 +4050,8 @@ static void i915_irq_postinstall(struct drm_i915_private *dev_priv) struct intel_uncore *uncore = &dev_priv->uncore; u32 enable_mask; - intel_uncore_write(&dev_priv->uncore, EMR, ~(I915_ERROR_PAGE_TABLE | - I915_ERROR_MEMORY_REFRESH)); + intel_uncore_write(uncore, EMR, ~(I915_ERROR_PAGE_TABLE | + I915_ERROR_MEMORY_REFRESH)); /* Unmask the interrupts that we always want on. */ dev_priv->irq_mask = @@ -4205,7 +4146,7 @@ static void i965_irq_reset(struct drm_i915_private *dev_priv) struct intel_uncore *uncore = &dev_priv->uncore; i915_hotplug_interrupt_update(dev_priv, 0xffffffff, 0); - intel_uncore_write(&dev_priv->uncore, PORT_HOTPLUG_STAT, intel_uncore_read(&dev_priv->uncore, PORT_HOTPLUG_STAT)); + intel_uncore_rmw(uncore, PORT_HOTPLUG_STAT, 0, 0); i9xx_pipestat_irq_reset(dev_priv); @@ -4232,7 +4173,7 @@ static void i965_irq_postinstall(struct drm_i915_private *dev_priv) error_mask = ~(I915_ERROR_PAGE_TABLE | I915_ERROR_MEMORY_REFRESH); } - intel_uncore_write(&dev_priv->uncore, EMR, error_mask); + intel_uncore_write(uncore, EMR, error_mask); /* Unmask the interrupts that we always want on. */ dev_priv->irq_mask = @@ -4383,7 +4324,6 @@ void intel_hpd_irq_setup(struct drm_i915_private *i915) */ void intel_irq_init(struct drm_i915_private *dev_priv) { - struct drm_device *dev = &dev_priv->drm; int i; INIT_WORK(&dev_priv->l3_parity.error_work, ivb_parity_work); @@ -4399,9 +4339,9 @@ void intel_irq_init(struct drm_i915_private *dev_priv) intel_hpd_init_pins(dev_priv); - intel_hpd_init_work(dev_priv); + intel_hpd_init_early(dev_priv); - dev->vblank_disable_immediate = true; + dev_priv->drm.vblank_disable_immediate = true; /* Most platforms treat the display irq block as an always-on * power domain. vlv/chv can disable it at runtime and need @@ -4413,15 +4353,6 @@ void intel_irq_init(struct drm_i915_private *dev_priv) if (IS_VALLEYVIEW(dev_priv) || IS_CHERRYVIEW(dev_priv)) dev_priv->display_irqs_enabled = false; - dev_priv->display.hotplug.hpd_storm_threshold = HPD_STORM_DEFAULT_THRESHOLD; - /* If we have MST support, we want to avoid doing short HPD IRQ storm - * detection, as short HPD storms will occur as a natural part of - * sideband messaging with MST. - * On older platforms however, IRQ storms can occur with both long and - * short pulses, as seen on some G4x systems. - */ - dev_priv->display.hotplug.hpd_short_storm_enabled = !HAS_DP_MST(dev_priv); - if (HAS_GMCH(dev_priv)) { if (I915_HAS_HOTPLUG(dev_priv)) dev_priv->display.funcs.hotplug = &i915_hpd_funcs; diff --git a/drivers/gpu/drm/i915/i915_irq.h b/drivers/gpu/drm/i915/i915_irq.h index 82639d9d7e82..9b004fc3444e 100644 --- a/drivers/gpu/drm/i915/i915_irq.h +++ b/drivers/gpu/drm/i915/i915_irq.h @@ -90,12 +90,9 @@ void i965_disable_vblank(struct drm_crtc *crtc); void ilk_disable_vblank(struct drm_crtc *crtc); void bdw_disable_vblank(struct drm_crtc *crtc); -void gen2_irq_reset(struct intel_uncore *uncore); void gen3_irq_reset(struct intel_uncore *uncore, i915_reg_t imr, i915_reg_t iir, i915_reg_t ier); -void gen2_irq_init(struct intel_uncore *uncore, - u32 imr_val, u32 ier_val); void gen3_irq_init(struct intel_uncore *uncore, i915_reg_t imr, u32 imr_val, i915_reg_t ier, u32 ier_val, @@ -111,9 +108,6 @@ void gen3_irq_init(struct intel_uncore *uncore, #define GEN3_IRQ_RESET(uncore, type) \ gen3_irq_reset((uncore), type##IMR, type##IIR, type##IER) -#define GEN2_IRQ_RESET(uncore) \ - gen2_irq_reset(uncore) - #define GEN8_IRQ_INIT_NDX(uncore, type, which, imr_val, ier_val) \ ({ \ unsigned int which_ = which; \ @@ -129,7 +123,4 @@ void gen3_irq_init(struct intel_uncore *uncore, type##IER, ier_val, \ type##IIR) -#define GEN2_IRQ_INIT(uncore, imr_val, ier_val) \ - gen2_irq_init((uncore), imr_val, ier_val) - #endif /* __I915_IRQ_H__ */ diff --git a/drivers/gpu/drm/i915/i915_pci.c b/drivers/gpu/drm/i915/i915_pci.c index cd4487a1d3be..6da9784fe4a2 100644 --- a/drivers/gpu/drm/i915/i915_pci.c +++ b/drivers/gpu/drm/i915/i915_pci.c @@ -1023,6 +1023,8 @@ static const struct intel_device_info adl_p_info = { .has_logical_ring_contexts = 1, \ .has_logical_ring_elsq = 1, \ .has_mslice_steering = 1, \ + .has_oa_bpc_reporting = 1, \ + .has_oa_slice_contrib_limits = 1, \ .has_rc6 = 1, \ .has_reset_engine = 1, \ .has_rps = 1, \ @@ -1042,7 +1044,6 @@ static const struct intel_device_info xehpsdv_info = { PLATFORM(INTEL_XEHPSDV), NO_DISPLAY, .has_64k_pages = 1, - .needs_compact_pt = 1, .has_media_ratio_mode = 1, .__runtime.platform_engine_mask = BIT(RCS0) | BIT(BCS0) | @@ -1064,8 +1065,8 @@ static const struct intel_device_info xehpsdv_info = { .has_64k_pages = 1, \ .has_guc_deprivilege = 1, \ .has_heci_pxp = 1, \ - .needs_compact_pt = 1, \ .has_media_ratio_mode = 1, \ + .display.has_cdclk_squash = 1, \ .__runtime.platform_engine_mask = \ BIT(RCS0) | BIT(BCS0) | \ BIT(VECS0) | BIT(VECS1) | \ @@ -1077,7 +1078,6 @@ static const struct intel_device_info dg2_info = { XE_LPD_FEATURES, .__runtime.cpu_transcoder_mask = BIT(TRANSCODER_A) | BIT(TRANSCODER_B) | BIT(TRANSCODER_C) | BIT(TRANSCODER_D), - .require_force_probe = 1, }; static const struct intel_device_info ats_m_info = { @@ -1144,6 +1144,9 @@ static const struct intel_device_info mtl_info = { .display.has_modular_fia = 1, .extra_gt_list = xelpmp_extra_gt, .has_flat_ccs = 0, + .has_gmd_id = 1, + .has_guc_deprivilege = 1, + .has_mslice_steering = 0, .has_snoop = 1, .__runtime.memory_regions = REGION_SMEM | REGION_STOLEN_LMEM, .__runtime.platform_engine_mask = BIT(RCS0) | BIT(BCS0) | BIT(CCS0), @@ -1296,9 +1299,7 @@ bool i915_pci_resource_valid(struct pci_dev *pdev, int bar) static bool intel_mmio_bar_valid(struct pci_dev *pdev, struct intel_device_info *intel_info) { - int gttmmaddr_bar = intel_info->__runtime.graphics.ip.ver == 2 ? GEN2_GTTMMADR_BAR : GTTMMADR_BAR; - - return i915_pci_resource_valid(pdev, gttmmaddr_bar); + return i915_pci_resource_valid(pdev, intel_mmio_bar(intel_info->__runtime.graphics.ip.ver)); } static int i915_pci_probe(struct pci_dev *pdev, const struct pci_device_id *ent) diff --git a/drivers/gpu/drm/i915/i915_perf.c b/drivers/gpu/drm/i915/i915_perf.c index 0defbb43ceea..00e09bb18b13 100644 --- a/drivers/gpu/drm/i915/i915_perf.c +++ b/drivers/gpu/drm/i915/i915_perf.c @@ -204,15 +204,18 @@ #include "gt/intel_gpu_commands.h" #include "gt/intel_gt.h" #include "gt/intel_gt_clock_utils.h" +#include "gt/intel_gt_mcr.h" #include "gt/intel_gt_regs.h" #include "gt/intel_lrc.h" #include "gt/intel_lrc_reg.h" #include "gt/intel_ring.h" +#include "gt/uc/intel_guc_slpc.h" #include "i915_drv.h" #include "i915_file_private.h" #include "i915_perf.h" #include "i915_perf_oa_regs.h" +#include "i915_reg.h" /* HW requires this to be a power of two, between 128k and 16M, though driver * is currently generally designed assuming the largest 16M size is used such @@ -286,6 +289,7 @@ static u32 i915_perf_stream_paranoid = true; #define OAREPORT_REASON_CTX_SWITCH (1<<3) #define OAREPORT_REASON_CLK_RATIO (1<<5) +#define HAS_MI_SET_PREDICATE(i915) (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) /* For sysctl proc_dointvec_minmax of i915_oa_max_sample_rate * @@ -320,6 +324,8 @@ static const struct i915_oa_format oa_formats[I915_OA_FORMAT_MAX] = { [I915_OA_FORMAT_A12] = { 0, 64 }, [I915_OA_FORMAT_A12_B8_C8] = { 2, 128 }, [I915_OA_FORMAT_A32u40_A4u32_B8_C8] = { 5, 256 }, + [I915_OAR_FORMAT_A32u40_A4u32_B8_C8] = { 5, 256 }, + [I915_OA_FORMAT_A24u40_A14u32_B8_C8] = { 5, 256 }, }; #define SAMPLE_OA_REPORT (1<<0) @@ -462,7 +468,7 @@ static u32 gen7_oa_hw_tail_read(struct i915_perf_stream *stream) static bool oa_buffer_check_unlocked(struct i915_perf_stream *stream) { u32 gtt_offset = i915_ggtt_offset(stream->oa_buffer.vma); - int report_size = stream->oa_buffer.format_size; + int report_size = stream->oa_buffer.format->size; unsigned long flags; bool pollin; u32 hw_tail; @@ -525,9 +531,9 @@ static bool oa_buffer_check_unlocked(struct i915_perf_stream *stream) if (OA_TAKEN(hw_tail, tail) > report_size && __ratelimit(&stream->perf->tail_pointer_race)) - DRM_NOTE("unlanded report(s) head=0x%x " - "tail=0x%x hw_tail=0x%x\n", - head, tail, hw_tail); + drm_notice(&stream->uncore->i915->drm, + "unlanded report(s) head=0x%x tail=0x%x hw_tail=0x%x\n", + head, tail, hw_tail); stream->oa_buffer.tail = gtt_offset + tail; stream->oa_buffer.aging_tail = gtt_offset + hw_tail; @@ -599,7 +605,7 @@ static int append_oa_sample(struct i915_perf_stream *stream, size_t *offset, const u8 *report) { - int report_size = stream->oa_buffer.format_size; + int report_size = stream->oa_buffer.format->size; struct drm_i915_perf_record_header header; header.type = DRM_I915_PERF_RECORD_SAMPLE; @@ -649,14 +655,13 @@ static int gen8_append_oa_reports(struct i915_perf_stream *stream, size_t *offset) { struct intel_uncore *uncore = stream->uncore; - int report_size = stream->oa_buffer.format_size; + int report_size = stream->oa_buffer.format->size; u8 *oa_buf_base = stream->oa_buffer.vaddr; u32 gtt_offset = i915_ggtt_offset(stream->oa_buffer.vma); u32 mask = (OA_BUFFER_SIZE - 1); size_t start_offset = *offset; unsigned long flags; u32 head, tail; - u32 taken; int ret = 0; if (drm_WARN_ON(&uncore->i915->drm, !stream->enabled)) @@ -692,7 +697,7 @@ static int gen8_append_oa_reports(struct i915_perf_stream *stream, for (/* none */; - (taken = OA_TAKEN(tail, head)); + OA_TAKEN(tail, head); head = (head + report_size) & mask) { u8 *report = oa_buf_base + head; u32 *report32 = (void *)report; @@ -774,7 +779,7 @@ static int gen8_append_oa_reports(struct i915_perf_stream *stream, * switches since it's not-uncommon for periodic samples to * identify a switch before any 'context switch' report. */ - if (!stream->perf->exclusive_stream->ctx || + if (!stream->ctx || stream->specific_ctx_id == ctx_id || stream->oa_buffer.last_ctx_id == stream->specific_ctx_id || reason & OAREPORT_REASON_CTX_SWITCH) { @@ -783,7 +788,7 @@ static int gen8_append_oa_reports(struct i915_perf_stream *stream, * While filtering for a single context we avoid * leaking the IDs of other contexts. */ - if (stream->perf->exclusive_stream->ctx && + if (stream->ctx && stream->specific_ctx_id != ctx_id) { report32[2] = INVALID_CTX_ID; } @@ -943,14 +948,13 @@ static int gen7_append_oa_reports(struct i915_perf_stream *stream, size_t *offset) { struct intel_uncore *uncore = stream->uncore; - int report_size = stream->oa_buffer.format_size; + int report_size = stream->oa_buffer.format->size; u8 *oa_buf_base = stream->oa_buffer.vaddr; u32 gtt_offset = i915_ggtt_offset(stream->oa_buffer.vma); u32 mask = (OA_BUFFER_SIZE - 1); size_t start_offset = *offset; unsigned long flags; u32 head, tail; - u32 taken; int ret = 0; if (drm_WARN_ON(&uncore->i915->drm, !stream->enabled)) @@ -984,7 +988,7 @@ static int gen7_append_oa_reports(struct i915_perf_stream *stream, for (/* none */; - (taken = OA_TAKEN(tail, head)); + OA_TAKEN(tail, head); head = (head + report_size) & mask) { u8 *report = oa_buf_base + head; u32 *report32 = (void *)report; @@ -1012,7 +1016,8 @@ static int gen7_append_oa_reports(struct i915_perf_stream *stream, */ if (report32[0] == 0) { if (__ratelimit(&stream->perf->spurious_report_rs)) - DRM_NOTE("Skipping spurious, invalid OA report\n"); + drm_notice(&uncore->i915->drm, + "Skipping spurious, invalid OA report\n"); continue; } @@ -1233,6 +1238,196 @@ retry: return stream->pinned_ctx; } +static int +__store_reg_to_mem(struct i915_request *rq, i915_reg_t reg, u32 ggtt_offset) +{ + u32 *cs, cmd; + + cmd = MI_STORE_REGISTER_MEM | MI_SRM_LRM_GLOBAL_GTT; + if (GRAPHICS_VER(rq->engine->i915) >= 8) + cmd++; + + cs = intel_ring_begin(rq, 4); + if (IS_ERR(cs)) + return PTR_ERR(cs); + + *cs++ = cmd; + *cs++ = i915_mmio_reg_offset(reg); + *cs++ = ggtt_offset; + *cs++ = 0; + + intel_ring_advance(rq, cs); + + return 0; +} + +static int +__read_reg(struct intel_context *ce, i915_reg_t reg, u32 ggtt_offset) +{ + struct i915_request *rq; + int err; + + rq = i915_request_create(ce); + if (IS_ERR(rq)) + return PTR_ERR(rq); + + i915_request_get(rq); + + err = __store_reg_to_mem(rq, reg, ggtt_offset); + + i915_request_add(rq); + if (!err && i915_request_wait(rq, 0, HZ / 2) < 0) + err = -ETIME; + + i915_request_put(rq); + + return err; +} + +static int +gen12_guc_sw_ctx_id(struct intel_context *ce, u32 *ctx_id) +{ + struct i915_vma *scratch; + u32 *val; + int err; + + scratch = __vm_create_scratch_for_read_pinned(&ce->engine->gt->ggtt->vm, 4); + if (IS_ERR(scratch)) + return PTR_ERR(scratch); + + err = i915_vma_sync(scratch); + if (err) + goto err_scratch; + + err = __read_reg(ce, RING_EXECLIST_STATUS_HI(ce->engine->mmio_base), + i915_ggtt_offset(scratch)); + if (err) + goto err_scratch; + + val = i915_gem_object_pin_map_unlocked(scratch->obj, I915_MAP_WB); + if (IS_ERR(val)) { + err = PTR_ERR(val); + goto err_scratch; + } + + *ctx_id = *val; + i915_gem_object_unpin_map(scratch->obj); + +err_scratch: + i915_vma_unpin_and_release(&scratch, 0); + return err; +} + +/* + * For execlist mode of submission, pick an unused context id + * 0 - (NUM_CONTEXT_TAG -1) are used by other contexts + * XXX_MAX_CONTEXT_HW_ID is used by idle context + * + * For GuC mode of submission read context id from the upper dword of the + * EXECLIST_STATUS register. Note that we read this value only once and expect + * that the value stays fixed for the entire OA use case. There are cases where + * GuC KMD implementation may deregister a context to reuse it's context id, but + * we prevent that from happening to the OA context by pinning it. + */ +static int gen12_get_render_context_id(struct i915_perf_stream *stream) +{ + u32 ctx_id, mask; + int ret; + + if (intel_engine_uses_guc(stream->engine)) { + ret = gen12_guc_sw_ctx_id(stream->pinned_ctx, &ctx_id); + if (ret) + return ret; + + mask = ((1U << GEN12_GUC_SW_CTX_ID_WIDTH) - 1) << + (GEN12_GUC_SW_CTX_ID_SHIFT - 32); + } else if (GRAPHICS_VER_FULL(stream->engine->i915) >= IP_VER(12, 50)) { + ctx_id = (XEHP_MAX_CONTEXT_HW_ID - 1) << + (XEHP_SW_CTX_ID_SHIFT - 32); + + mask = ((1U << XEHP_SW_CTX_ID_WIDTH) - 1) << + (XEHP_SW_CTX_ID_SHIFT - 32); + } else { + ctx_id = (GEN12_MAX_CONTEXT_HW_ID - 1) << + (GEN11_SW_CTX_ID_SHIFT - 32); + + mask = ((1U << GEN11_SW_CTX_ID_WIDTH) - 1) << + (GEN11_SW_CTX_ID_SHIFT - 32); + } + stream->specific_ctx_id = ctx_id & mask; + stream->specific_ctx_id_mask = mask; + + return 0; +} + +static bool oa_find_reg_in_lri(u32 *state, u32 reg, u32 *offset, u32 end) +{ + u32 idx = *offset; + u32 len = min(MI_LRI_LEN(state[idx]) + idx, end); + bool found = false; + + idx++; + for (; idx < len; idx += 2) { + if (state[idx] == reg) { + found = true; + break; + } + } + + *offset = idx; + return found; +} + +static u32 oa_context_image_offset(struct intel_context *ce, u32 reg) +{ + u32 offset, len = (ce->engine->context_size - PAGE_SIZE) / 4; + u32 *state = ce->lrc_reg_state; + + for (offset = 0; offset < len; ) { + if (IS_MI_LRI_CMD(state[offset])) { + /* + * We expect reg-value pairs in MI_LRI command, so + * MI_LRI_LEN() should be even, if not, issue a warning. + */ + drm_WARN_ON(&ce->engine->i915->drm, + MI_LRI_LEN(state[offset]) & 0x1); + + if (oa_find_reg_in_lri(state, reg, &offset, len)) + break; + } else { + offset++; + } + } + + return offset < len ? offset : U32_MAX; +} + +static int set_oa_ctx_ctrl_offset(struct intel_context *ce) +{ + i915_reg_t reg = GEN12_OACTXCONTROL(ce->engine->mmio_base); + struct i915_perf *perf = &ce->engine->i915->perf; + u32 offset = perf->ctx_oactxctrl_offset; + + /* Do this only once. Failure is stored as offset of U32_MAX */ + if (offset) + goto exit; + + offset = oa_context_image_offset(ce, i915_mmio_reg_offset(reg)); + perf->ctx_oactxctrl_offset = offset; + + drm_dbg(&ce->engine->i915->drm, + "%s oa ctx control at 0x%08x dword offset\n", + ce->engine->name, offset); + +exit: + return offset && offset != U32_MAX ? 0 : -ENODEV; +} + +static bool engine_supports_mi_query(struct intel_engine_cs *engine) +{ + return engine->class == RENDER_CLASS; +} + /** * oa_get_render_ctx_id - determine and hold ctx hw id * @stream: An i915-perf stream opened for OA metrics @@ -1246,11 +1441,27 @@ retry: static int oa_get_render_ctx_id(struct i915_perf_stream *stream) { struct intel_context *ce; + int ret = 0; ce = oa_pin_context(stream); if (IS_ERR(ce)) return PTR_ERR(ce); + if (engine_supports_mi_query(stream->engine)) { + /* + * We are enabling perf query here. If we don't find the context + * offset here, just return an error. + */ + ret = set_oa_ctx_ctrl_offset(ce); + if (ret) { + intel_context_unpin(ce); + drm_err(&stream->perf->i915->drm, + "Enabling perf query failed for %s\n", + stream->engine->name); + return ret; + } + } + switch (GRAPHICS_VER(ce->engine->i915)) { case 7: { /* @@ -1292,24 +1503,7 @@ static int oa_get_render_ctx_id(struct i915_perf_stream *stream) case 11: case 12: - if (GRAPHICS_VER_FULL(ce->engine->i915) >= IP_VER(12, 50)) { - stream->specific_ctx_id_mask = - ((1U << XEHP_SW_CTX_ID_WIDTH) - 1) << - (XEHP_SW_CTX_ID_SHIFT - 32); - stream->specific_ctx_id = - (XEHP_MAX_CONTEXT_HW_ID - 1) << - (XEHP_SW_CTX_ID_SHIFT - 32); - } else { - stream->specific_ctx_id_mask = - ((1U << GEN11_SW_CTX_ID_WIDTH) - 1) << (GEN11_SW_CTX_ID_SHIFT - 32); - /* - * Pick an unused context id - * 0 - BITS_PER_LONG are used by other contexts - * GEN12_MAX_CONTEXT_HW_ID (0x7ff) is used by idle context - */ - stream->specific_ctx_id = - (GEN12_MAX_CONTEXT_HW_ID - 1) << (GEN11_SW_CTX_ID_SHIFT - 32); - } + ret = gen12_get_render_context_id(stream); break; default: @@ -1323,7 +1517,7 @@ static int oa_get_render_ctx_id(struct i915_perf_stream *stream) stream->specific_ctx_id, stream->specific_ctx_id_mask); - return 0; + return ret; } /** @@ -1375,8 +1569,9 @@ free_noa_wait(struct i915_perf_stream *stream) static void i915_oa_stream_destroy(struct i915_perf_stream *stream) { struct i915_perf *perf = stream->perf; + struct intel_gt *gt = stream->engine->gt; - if (WARN_ON(stream != perf->exclusive_stream)) + if (WARN_ON(stream != gt->perf.exclusive_stream)) return; /* @@ -1385,11 +1580,20 @@ static void i915_oa_stream_destroy(struct i915_perf_stream *stream) * * See i915_oa_init_reg_state() and lrc_configure_all_contexts() */ - WRITE_ONCE(perf->exclusive_stream, NULL); + WRITE_ONCE(gt->perf.exclusive_stream, NULL); perf->ops.disable_metric_set(stream); free_oa_buffer(stream); + /* + * Wa_16011777198:dg2: Unset the override of GUCRC mode to enable rc6. + */ + if (intel_uc_uses_guc_rc(>->uc) && + (IS_DG2_GRAPHICS_STEP(gt->i915, G10, STEP_A0, STEP_C0) || + IS_DG2_GRAPHICS_STEP(gt->i915, G11, STEP_A0, STEP_B0))) + drm_WARN_ON(>->i915->drm, + intel_guc_slpc_unset_gucrc_mode(>->uc.guc.slpc)); + intel_uncore_forcewake_put(stream->uncore, FORCEWAKE_ALL); intel_engine_pm_put(stream->engine); @@ -1400,8 +1604,9 @@ static void i915_oa_stream_destroy(struct i915_perf_stream *stream) free_noa_wait(stream); if (perf->spurious_report_rs.missed) { - DRM_NOTE("%d spurious OA report notices suppressed due to ratelimiting\n", - perf->spurious_report_rs.missed); + drm_notice(>->i915->drm, + "%d spurious OA report notices suppressed due to ratelimiting\n", + perf->spurious_report_rs.missed); } } @@ -1563,6 +1768,7 @@ static void gen12_init_oa_buffer(struct i915_perf_stream *stream) static int alloc_oa_buffer(struct i915_perf_stream *stream) { struct drm_i915_private *i915 = stream->perf->i915; + struct intel_gt *gt = stream->engine->gt; struct drm_i915_gem_object *bo; struct i915_vma *vma; int ret; @@ -1582,11 +1788,22 @@ static int alloc_oa_buffer(struct i915_perf_stream *stream) i915_gem_object_set_cache_coherency(bo, I915_CACHE_LLC); /* PreHSW required 512K alignment, HSW requires 16M */ - vma = i915_gem_object_ggtt_pin(bo, NULL, 0, SZ_16M, 0); + vma = i915_vma_instance(bo, >->ggtt->vm, NULL); if (IS_ERR(vma)) { ret = PTR_ERR(vma); goto err_unref; } + + /* + * PreHSW required 512K alignment. + * HSW and onwards, align to requested size of OA buffer. + */ + ret = i915_vma_pin(vma, 0, SZ_16M, PIN_GLOBAL | PIN_HIGH); + if (ret) { + drm_err(>->i915->drm, "Failed to pin OA buffer %d\n", ret); + goto err_unref; + } + stream->oa_buffer.vma = vma; stream->oa_buffer.vaddr = @@ -1636,6 +1853,7 @@ static u32 *save_restore_register(struct i915_perf_stream *stream, u32 *cs, static int alloc_noa_wait(struct i915_perf_stream *stream) { struct drm_i915_private *i915 = stream->perf->i915; + struct intel_gt *gt = stream->engine->gt; struct drm_i915_gem_object *bo; struct i915_vma *vma; const u64 delay_ticks = 0xffffffffffffffff - @@ -1654,6 +1872,9 @@ static int alloc_noa_wait(struct i915_perf_stream *stream) DELTA_TARGET, N_CS_GPR }; + i915_reg_t mi_predicate_result = HAS_MI_SET_PREDICATE(i915) ? + MI_PREDICATE_RESULT_2_ENGINE(base) : + MI_PREDICATE_RESULT_1(RENDER_RING_BASE); bo = i915_gem_object_create_internal(i915, 4096); if (IS_ERR(bo)) { @@ -1673,12 +1894,16 @@ retry: * multiple OA config BOs will have a jump to this address and it * needs to be fixed during the lifetime of the i915/perf stream. */ - vma = i915_gem_object_ggtt_pin_ww(bo, &ww, NULL, 0, 0, PIN_HIGH); + vma = i915_vma_instance(bo, >->ggtt->vm, NULL); if (IS_ERR(vma)) { ret = PTR_ERR(vma); goto out_ww; } + ret = i915_vma_pin_ww(vma, &ww, 0, 0, PIN_GLOBAL | PIN_HIGH); + if (ret) + goto out_ww; + batch = cs = i915_gem_object_pin_map(bo, I915_MAP_WB); if (IS_ERR(batch)) { ret = PTR_ERR(batch); @@ -1691,7 +1916,7 @@ retry: stream, cs, true /* save */, CS_GPR(i), INTEL_GT_SCRATCH_FIELD_PERF_CS_GPR + 8 * i, 2); cs = save_restore_register( - stream, cs, true /* save */, MI_PREDICATE_RESULT_1(RENDER_RING_BASE), + stream, cs, true /* save */, mi_predicate_result, INTEL_GT_SCRATCH_FIELD_PERF_PREDICATE_RESULT_1, 1); /* First timestamp snapshot location. */ @@ -1745,7 +1970,10 @@ retry: */ *cs++ = MI_LOAD_REGISTER_REG | (3 - 2); *cs++ = i915_mmio_reg_offset(CS_GPR(JUMP_PREDICATE)); - *cs++ = i915_mmio_reg_offset(MI_PREDICATE_RESULT_1(RENDER_RING_BASE)); + *cs++ = i915_mmio_reg_offset(mi_predicate_result); + + if (HAS_MI_SET_PREDICATE(i915)) + *cs++ = MI_SET_PREDICATE | 1; /* Restart from the beginning if we had timestamps roll over. */ *cs++ = (GRAPHICS_VER(i915) < 8 ? @@ -1755,6 +1983,9 @@ retry: *cs++ = i915_ggtt_offset(vma) + (ts0 - batch) * 4; *cs++ = 0; + if (HAS_MI_SET_PREDICATE(i915)) + *cs++ = MI_SET_PREDICATE; + /* * Now add the diff between to previous timestamps and add it to : * (((1 * << 64) - 1) - delay_ns) @@ -1782,7 +2013,10 @@ retry: */ *cs++ = MI_LOAD_REGISTER_REG | (3 - 2); *cs++ = i915_mmio_reg_offset(CS_GPR(JUMP_PREDICATE)); - *cs++ = i915_mmio_reg_offset(MI_PREDICATE_RESULT_1(RENDER_RING_BASE)); + *cs++ = i915_mmio_reg_offset(mi_predicate_result); + + if (HAS_MI_SET_PREDICATE(i915)) + *cs++ = MI_SET_PREDICATE | 1; /* Predicate the jump. */ *cs++ = (GRAPHICS_VER(i915) < 8 ? @@ -1792,13 +2026,16 @@ retry: *cs++ = i915_ggtt_offset(vma) + (jump - batch) * 4; *cs++ = 0; + if (HAS_MI_SET_PREDICATE(i915)) + *cs++ = MI_SET_PREDICATE; + /* Restore registers. */ for (i = 0; i < N_CS_GPR; i++) cs = save_restore_register( stream, cs, false /* restore */, CS_GPR(i), INTEL_GT_SCRATCH_FIELD_PERF_CS_GPR + 8 * i, 2); cs = save_restore_register( - stream, cs, false /* restore */, MI_PREDICATE_RESULT_1(RENDER_RING_BASE), + stream, cs, false /* restore */, mi_predicate_result, INTEL_GT_SCRATCH_FIELD_PERF_PREDICATE_RESULT_1, 1); /* And return to the ring. */ @@ -2017,9 +2254,7 @@ retry: goto err_add_request; } - err = i915_request_await_object(rq, vma->obj, 0); - if (!err) - err = i915_vma_move_to_active(vma, rq, 0); + err = i915_vma_move_to_active(vma, rq, 0); if (err) goto err_add_request; @@ -2283,11 +2518,12 @@ static int gen12_configure_oar_context(struct i915_perf_stream *stream, { int err; struct intel_context *ce = stream->pinned_ctx; - u32 format = stream->oa_buffer.format; + u32 format = stream->oa_buffer.format->format; + u32 offset = stream->perf->ctx_oactxctrl_offset; struct flex regs_context[] = { { GEN8_OACTXCONTROL, - stream->perf->ctx_oactxctrl_offset + 1, + offset + 1, active ? GEN8_OA_COUNTER_RESUME : 0, }, }; @@ -2312,12 +2548,13 @@ static int gen12_configure_oar_context(struct i915_perf_stream *stream, }, }; - /* Modify the context image of pinned context with regs_context*/ + /* Modify the context image of pinned context with regs_context */ err = intel_context_lock_pinned(ce); if (err) return err; - err = gen8_modify_context(ce, regs_context, ARRAY_SIZE(regs_context)); + err = gen8_modify_context(ce, regs_context, + ARRAY_SIZE(regs_context)); intel_context_unlock_pinned(ce); if (err) return err; @@ -2359,10 +2596,11 @@ oa_configure_all_contexts(struct i915_perf_stream *stream, { struct drm_i915_private *i915 = stream->perf->i915; struct intel_engine_cs *engine; + struct intel_gt *gt = stream->engine->gt; struct i915_gem_context *ctx, *cn; int err; - lockdep_assert_held(&stream->perf->lock); + lockdep_assert_held(>->perf.lock); /* * The OA register config is setup through the context image. This image @@ -2442,6 +2680,7 @@ lrc_configure_all_contexts(struct i915_perf_stream *stream, const struct i915_oa_config *oa_config, struct i915_active *active) { + u32 ctx_oactxctrl = stream->perf->ctx_oactxctrl_offset; /* The MMIO offsets for Flex EU registers aren't contiguous */ const u32 ctx_flexeu0 = stream->perf->ctx_flexeu0_offset; #define ctx_flexeuN(N) (ctx_flexeu0 + 2 * (N) + 1) @@ -2452,7 +2691,7 @@ lrc_configure_all_contexts(struct i915_perf_stream *stream, }, { GEN8_OACTXCONTROL, - stream->perf->ctx_oactxctrl_offset + 1, + ctx_oactxctrl + 1, }, { EU_PERF_CNTL0, ctx_flexeuN(0) }, { EU_PERF_CNTL1, ctx_flexeuN(1) }, @@ -2540,12 +2779,26 @@ static int gen12_enable_metric_set(struct i915_perf_stream *stream, struct i915_active *active) { + struct drm_i915_private *i915 = stream->perf->i915; struct intel_uncore *uncore = stream->uncore; struct i915_oa_config *oa_config = stream->oa_config; bool periodic = stream->periodic; u32 period_exponent = stream->period_exponent; + u32 sqcnt1; int ret; + /* + * Wa_1508761755:xehpsdv, dg2 + * EU NOA signals behave incorrectly if EU clock gating is enabled. + * Disable thread stall DOP gating and EU DOP gating. + */ + if (IS_XEHPSDV(i915) || IS_DG2(i915)) { + intel_gt_mcr_multicast_write(uncore->gt, GEN8_ROW_CHICKEN, + _MASKED_BIT_ENABLE(STALL_DOP_GATING_DISABLE)); + intel_uncore_write(uncore, GEN7_ROW_CHICKEN2, + _MASKED_BIT_ENABLE(GEN12_DISABLE_DOP_GATING)); + } + intel_uncore_write(uncore, GEN12_OAG_OA_DEBUG, /* Disable clk ratio reports, like previous Gens. */ _MASKED_BIT_ENABLE(GEN12_OAG_OA_DEBUG_DISABLE_CLK_RATIO_REPORTS | @@ -2563,6 +2816,16 @@ gen12_enable_metric_set(struct i915_perf_stream *stream, : 0); /* + * Initialize Super Queue Internal Cnt Register + * Set PMON Enable in order to collect valid metrics. + * Enable byets per clock reporting in OA for XEHPSDV onward. + */ + sqcnt1 = GEN12_SQCNT1_PMON_ENABLE | + (HAS_OA_BPC_REPORTING(i915) ? GEN12_SQCNT1_OABPC : 0); + + intel_uncore_rmw(uncore, GEN12_SQCNT1, 0, sqcnt1); + + /* * Update all contexts prior writing the mux configurations as we need * to make sure all slices/subslices are ON before writing to NOA * registers. @@ -2611,6 +2874,19 @@ static void gen11_disable_metric_set(struct i915_perf_stream *stream) static void gen12_disable_metric_set(struct i915_perf_stream *stream) { struct intel_uncore *uncore = stream->uncore; + struct drm_i915_private *i915 = stream->perf->i915; + u32 sqcnt1; + + /* + * Wa_1508761755:xehpsdv, dg2 + * Enable thread stall DOP gating and EU DOP gating. + */ + if (IS_XEHPSDV(i915) || IS_DG2(i915)) { + intel_gt_mcr_multicast_write(uncore->gt, GEN8_ROW_CHICKEN, + _MASKED_BIT_DISABLE(STALL_DOP_GATING_DISABLE)); + intel_uncore_write(uncore, GEN7_ROW_CHICKEN2, + _MASKED_BIT_DISABLE(GEN12_DISABLE_DOP_GATING)); + } /* Reset all contexts' slices/subslices configurations. */ gen12_configure_all_contexts(stream, NULL, NULL); @@ -2621,6 +2897,12 @@ static void gen12_disable_metric_set(struct i915_perf_stream *stream) /* Make sure we disable noa to save power. */ intel_uncore_rmw(uncore, RPM_CONFIG1, GEN10_GT_NOA_ENABLE, 0); + + sqcnt1 = GEN12_SQCNT1_PMON_ENABLE | + (HAS_OA_BPC_REPORTING(i915) ? GEN12_SQCNT1_OABPC : 0); + + /* Reset PMON Enable to save power. */ + intel_uncore_rmw(uncore, GEN12_SQCNT1, sqcnt1, 0); } static void gen7_oa_enable(struct i915_perf_stream *stream) @@ -2630,7 +2912,7 @@ static void gen7_oa_enable(struct i915_perf_stream *stream) u32 ctx_id = stream->specific_ctx_id; bool periodic = stream->periodic; u32 period_exponent = stream->period_exponent; - u32 report_format = stream->oa_buffer.format; + u32 report_format = stream->oa_buffer.format->format; /* * Reset buf pointers so we don't forward reports from before now. @@ -2656,7 +2938,7 @@ static void gen7_oa_enable(struct i915_perf_stream *stream) static void gen8_oa_enable(struct i915_perf_stream *stream) { struct intel_uncore *uncore = stream->uncore; - u32 report_format = stream->oa_buffer.format; + u32 report_format = stream->oa_buffer.format->format; /* * Reset buf pointers so we don't forward reports from before now. @@ -2682,7 +2964,7 @@ static void gen8_oa_enable(struct i915_perf_stream *stream) static void gen12_oa_enable(struct i915_perf_stream *stream) { struct intel_uncore *uncore = stream->uncore; - u32 report_format = stream->oa_buffer.format; + u32 report_format = stream->oa_buffer.format->format; /* * If we don't want OA reports from the OA buffer, then we don't even @@ -2838,6 +3120,30 @@ get_sseu_config(struct intel_sseu *out_sseu, return i915_gem_user_to_context_sseu(engine->gt, drm_sseu, out_sseu); } +/* + * OA timestamp frequency = CS timestamp frequency in most platforms. On some + * platforms OA unit ignores the CTC_SHIFT and the 2 timestamps differ. In such + * cases, return the adjusted CS timestamp frequency to the user. + */ +u32 i915_perf_oa_timestamp_frequency(struct drm_i915_private *i915) +{ + /* Wa_18013179988:dg2 */ + if (IS_DG2(i915)) { + intel_wakeref_t wakeref; + u32 reg, shift; + + with_intel_runtime_pm(to_gt(i915)->uncore->rpm, wakeref) + reg = intel_uncore_read(to_gt(i915)->uncore, RPM_CONFIG0); + + shift = REG_FIELD_GET(GEN10_RPM_CONFIG0_CTC_SHIFT_PARAMETER_MASK, + reg); + + return to_gt(i915)->clock_frequency << (3 - shift); + } + + return to_gt(i915)->clock_frequency; +} + /** * i915_oa_stream_init - validate combined props for OA stream and init * @stream: An i915 perf stream @@ -2862,7 +3168,7 @@ static int i915_oa_stream_init(struct i915_perf_stream *stream, { struct drm_i915_private *i915 = stream->perf->i915; struct i915_perf *perf = stream->perf; - int format_size; + struct intel_gt *gt; int ret; if (!props->engine) { @@ -2870,6 +3176,7 @@ static int i915_oa_stream_init(struct i915_perf_stream *stream, "OA engine not specified\n"); return -EINVAL; } + gt = props->engine->gt; /* * If the sysfs metrics/ directory wasn't registered for some @@ -2900,7 +3207,7 @@ static int i915_oa_stream_init(struct i915_perf_stream *stream, * counter reports and marshal to the appropriate client * we currently only allow exclusive access */ - if (perf->exclusive_stream) { + if (gt->perf.exclusive_stream) { drm_dbg(&stream->perf->i915->drm, "OA unit already in use\n"); return -EBUSY; @@ -2917,20 +3224,15 @@ static int i915_oa_stream_init(struct i915_perf_stream *stream, stream->sample_size = sizeof(struct drm_i915_perf_record_header); - format_size = perf->oa_formats[props->oa_format].size; + stream->oa_buffer.format = &perf->oa_formats[props->oa_format]; + if (drm_WARN_ON(&i915->drm, stream->oa_buffer.format->size == 0)) + return -EINVAL; stream->sample_flags = props->sample_flags; - stream->sample_size += format_size; - - stream->oa_buffer.format_size = format_size; - if (drm_WARN_ON(&i915->drm, stream->oa_buffer.format_size == 0)) - return -EINVAL; + stream->sample_size += stream->oa_buffer.format->size; stream->hold_preemption = props->hold_preemption; - stream->oa_buffer.format = - perf->oa_formats[props->oa_format].format; - stream->periodic = props->oa_periodic; if (stream->periodic) stream->period_exponent = props->oa_period_exponent; @@ -2974,14 +3276,31 @@ static int i915_oa_stream_init(struct i915_perf_stream *stream, intel_engine_pm_get(stream->engine); intel_uncore_forcewake_get(stream->uncore, FORCEWAKE_ALL); + /* + * Wa_16011777198:dg2: GuC resets render as part of the Wa. This causes + * OA to lose the configuration state. Prevent this by overriding GUCRC + * mode. + */ + if (intel_uc_uses_guc_rc(>->uc) && + (IS_DG2_GRAPHICS_STEP(gt->i915, G10, STEP_A0, STEP_C0) || + IS_DG2_GRAPHICS_STEP(gt->i915, G11, STEP_A0, STEP_B0))) { + ret = intel_guc_slpc_override_gucrc_mode(>->uc.guc.slpc, + SLPC_GUCRC_MODE_GUCRC_NO_RC6); + if (ret) { + drm_dbg(&stream->perf->i915->drm, + "Unable to override gucrc mode\n"); + goto err_config; + } + } + ret = alloc_oa_buffer(stream); if (ret) goto err_oa_buf_alloc; stream->ops = &i915_oa_stream_ops; - perf->sseu = props->sseu; - WRITE_ONCE(perf->exclusive_stream, stream); + stream->engine->gt->perf.sseu = props->sseu; + WRITE_ONCE(gt->perf.exclusive_stream, stream); ret = i915_perf_stream_enable_sync(stream); if (ret) { @@ -2999,11 +3318,12 @@ static int i915_oa_stream_init(struct i915_perf_stream *stream, stream->poll_check_timer.function = oa_poll_check_timer_cb; init_waitqueue_head(&stream->poll_wq); spin_lock_init(&stream->oa_buffer.ptr_lock); + mutex_init(&stream->lock); return 0; err_enable: - WRITE_ONCE(perf->exclusive_stream, NULL); + WRITE_ONCE(gt->perf.exclusive_stream, NULL); perf->ops.disable_metric_set(stream); free_oa_buffer(stream); @@ -3033,7 +3353,7 @@ void i915_oa_init_reg_state(const struct intel_context *ce, return; /* perf.exclusive_stream serialised by lrc_configure_all_contexts() */ - stream = READ_ONCE(engine->i915->perf.exclusive_stream); + stream = READ_ONCE(engine->gt->perf.exclusive_stream); if (stream && GRAPHICS_VER(stream->perf->i915) < 12) gen8_update_reg_state_unlocked(ce, stream); } @@ -3062,7 +3382,6 @@ static ssize_t i915_perf_read(struct file *file, loff_t *ppos) { struct i915_perf_stream *stream = file->private_data; - struct i915_perf *perf = stream->perf; size_t offset = 0; int ret; @@ -3086,14 +3405,14 @@ static ssize_t i915_perf_read(struct file *file, if (ret) return ret; - mutex_lock(&perf->lock); + mutex_lock(&stream->lock); ret = stream->ops->read(stream, buf, count, &offset); - mutex_unlock(&perf->lock); + mutex_unlock(&stream->lock); } while (!offset && !ret); } else { - mutex_lock(&perf->lock); + mutex_lock(&stream->lock); ret = stream->ops->read(stream, buf, count, &offset); - mutex_unlock(&perf->lock); + mutex_unlock(&stream->lock); } /* We allow the poll checking to sometimes report false positive EPOLLIN @@ -3140,9 +3459,6 @@ static enum hrtimer_restart oa_poll_check_timer_cb(struct hrtimer *hrtimer) * &i915_perf_stream_ops->poll_wait to call poll_wait() with a wait queue that * will be woken for new stream data. * - * Note: The &perf->lock mutex has been taken to serialize - * with any non-file-operation driver hooks. - * * Returns: any poll events that are ready without sleeping */ static __poll_t i915_perf_poll_locked(struct i915_perf_stream *stream, @@ -3181,12 +3497,11 @@ static __poll_t i915_perf_poll_locked(struct i915_perf_stream *stream, static __poll_t i915_perf_poll(struct file *file, poll_table *wait) { struct i915_perf_stream *stream = file->private_data; - struct i915_perf *perf = stream->perf; __poll_t ret; - mutex_lock(&perf->lock); + mutex_lock(&stream->lock); ret = i915_perf_poll_locked(stream, file, wait); - mutex_unlock(&perf->lock); + mutex_unlock(&stream->lock); return ret; } @@ -3285,9 +3600,6 @@ static long i915_perf_config_locked(struct i915_perf_stream *stream, * @cmd: the ioctl request * @arg: the ioctl data * - * Note: The &perf->lock mutex has been taken to serialize - * with any non-file-operation driver hooks. - * * Returns: zero on success or a negative error code. Returns -EINVAL for * an unknown ioctl request. */ @@ -3325,12 +3637,11 @@ static long i915_perf_ioctl(struct file *file, unsigned long arg) { struct i915_perf_stream *stream = file->private_data; - struct i915_perf *perf = stream->perf; long ret; - mutex_lock(&perf->lock); + mutex_lock(&stream->lock); ret = i915_perf_ioctl_locked(stream, cmd, arg); - mutex_unlock(&perf->lock); + mutex_unlock(&stream->lock); return ret; } @@ -3342,7 +3653,7 @@ static long i915_perf_ioctl(struct file *file, * Frees all resources associated with the given i915 perf @stream, disabling * any associated data capture in the process. * - * Note: The &perf->lock mutex has been taken to serialize + * Note: The >->perf.lock mutex has been taken to serialize * with any non-file-operation driver hooks. */ static void i915_perf_destroy_locked(struct i915_perf_stream *stream) @@ -3374,10 +3685,16 @@ static int i915_perf_release(struct inode *inode, struct file *file) { struct i915_perf_stream *stream = file->private_data; struct i915_perf *perf = stream->perf; + struct intel_gt *gt = stream->engine->gt; - mutex_lock(&perf->lock); + /* + * Within this call, we know that the fd is being closed and we have no + * other user of stream->lock. Use the perf lock to destroy the stream + * here. + */ + mutex_lock(>->perf.lock); i915_perf_destroy_locked(stream); - mutex_unlock(&perf->lock); + mutex_unlock(>->perf.lock); /* Release the reference the perf stream kept on the driver. */ drm_dev_put(&perf->i915->drm); @@ -3410,7 +3727,7 @@ static const struct file_operations fops = { * See i915_perf_ioctl_open() for interface details. * * Implements further stream config validation and stream initialization on - * behalf of i915_perf_open_ioctl() with the &perf->lock mutex + * behalf of i915_perf_open_ioctl() with the >->perf.lock mutex * taken to serialize with any non-file-operation driver hooks. * * Note: at this point the @props have only been validated in isolation and @@ -3565,8 +3882,10 @@ err: static u64 oa_exponent_to_ns(struct i915_perf *perf, int exponent) { - return intel_gt_clock_interval_to_ns(to_gt(perf->i915), - 2ULL << exponent); + u64 nom = (2ULL << exponent) * NSEC_PER_SEC; + u32 den = i915_perf_oa_timestamp_frequency(perf->i915); + + return div_u64(nom + den - 1, den); } static __always_inline bool @@ -3794,7 +4113,7 @@ static int read_properties_unlocked(struct i915_perf *perf, * mutex to avoid an awkward lockdep with mmap_lock. * * Most of the implementation details are handled by - * i915_perf_open_ioctl_locked() after taking the &perf->lock + * i915_perf_open_ioctl_locked() after taking the >->perf.lock * mutex for serializing with any non-file-operation driver hooks. * * Return: A newly opened i915 Perf stream file descriptor or negative @@ -3805,6 +4124,7 @@ int i915_perf_open_ioctl(struct drm_device *dev, void *data, { struct i915_perf *perf = &to_i915(dev)->perf; struct drm_i915_perf_open_param *param = data; + struct intel_gt *gt; struct perf_open_properties props; u32 known_open_flags; int ret; @@ -3831,9 +4151,11 @@ int i915_perf_open_ioctl(struct drm_device *dev, void *data, if (ret) return ret; - mutex_lock(&perf->lock); + gt = props.engine->gt; + + mutex_lock(>->perf.lock); ret = i915_perf_open_ioctl_locked(perf, param, &props, file); - mutex_unlock(&perf->lock); + mutex_unlock(>->perf.lock); return ret; } @@ -3849,6 +4171,7 @@ int i915_perf_open_ioctl(struct drm_device *dev, void *data, void i915_perf_register(struct drm_i915_private *i915) { struct i915_perf *perf = &i915->perf; + struct intel_gt *gt = to_gt(i915); if (!perf->i915) return; @@ -3857,13 +4180,13 @@ void i915_perf_register(struct drm_i915_private *i915) * i915_perf_open_ioctl(); considering that we register after * being exposed to userspace. */ - mutex_lock(&perf->lock); + mutex_lock(>->perf.lock); perf->metrics_kobj = kobject_create_and_add("metrics", &i915->drm.primary->kdev->kobj); - mutex_unlock(&perf->lock); + mutex_unlock(>->perf.lock); } /** @@ -3939,6 +4262,11 @@ static const struct i915_range gen12_oa_b_counters[] = { {} }; +static const struct i915_range xehp_oa_b_counters[] = { + { .start = 0xdc48, .end = 0xdc48 }, /* OAA_ENABLE_REG */ + { .start = 0xdd00, .end = 0xdd48 }, /* OAG_LCE0_0 - OAA_LENABLE_REG */ +}; + static const struct i915_range gen7_oa_mux_regs[] = { { .start = 0x91b8, .end = 0x91cc }, /* OA_PERFCNT[1-2], OA_PERFMATRIX */ { .start = 0x9800, .end = 0x9888 }, /* MICRO_BP0_0 - NOA_WRITE */ @@ -4013,6 +4341,12 @@ static bool gen12_is_valid_b_counter_addr(struct i915_perf *perf, u32 addr) return reg_in_range_table(addr, gen12_oa_b_counters); } +static bool xehp_is_valid_b_counter_addr(struct i915_perf *perf, u32 addr) +{ + return reg_in_range_table(addr, xehp_oa_b_counters) || + reg_in_range_table(addr, gen12_oa_b_counters); +} + static bool gen12_is_valid_mux_addr(struct i915_perf *perf, u32 addr) { return reg_in_range_table(addr, gen12_oa_mux_regs); @@ -4411,11 +4745,47 @@ static void oa_init_supported_formats(struct i915_perf *perf) oa_format_add(perf, I915_OA_FORMAT_C4_B8); break; + case INTEL_DG2: + oa_format_add(perf, I915_OAR_FORMAT_A32u40_A4u32_B8_C8); + oa_format_add(perf, I915_OA_FORMAT_A24u40_A14u32_B8_C8); + break; + default: MISSING_CASE(platform); } } +static void i915_perf_init_info(struct drm_i915_private *i915) +{ + struct i915_perf *perf = &i915->perf; + + switch (GRAPHICS_VER(i915)) { + case 8: + perf->ctx_oactxctrl_offset = 0x120; + perf->ctx_flexeu0_offset = 0x2ce; + perf->gen8_valid_ctx_bit = BIT(25); + break; + case 9: + perf->ctx_oactxctrl_offset = 0x128; + perf->ctx_flexeu0_offset = 0x3de; + perf->gen8_valid_ctx_bit = BIT(16); + break; + case 11: + perf->ctx_oactxctrl_offset = 0x124; + perf->ctx_flexeu0_offset = 0x78e; + perf->gen8_valid_ctx_bit = BIT(16); + break; + case 12: + /* + * Calculate offset at runtime in oa_pin_context for gen12 and + * cache the value in perf->ctx_oactxctrl_offset. + */ + break; + default: + MISSING_CASE(GRAPHICS_VER(i915)); + } +} + /** * i915_perf_init - initialize i915-perf state on module bind * @i915: i915 device instance @@ -4429,12 +4799,6 @@ void i915_perf_init(struct drm_i915_private *i915) { struct i915_perf *perf = &i915->perf; - /* XXX const struct i915_perf_ops! */ - - /* i915_perf is not enabled for DG2 yet */ - if (IS_DG2(i915)) - return; - perf->oa_formats = oa_formats; if (IS_HASWELL(i915)) { perf->ops.is_valid_b_counter_reg = gen7_is_valid_b_counter_addr; @@ -4454,6 +4818,7 @@ void i915_perf_init(struct drm_i915_private *i915) * execlist mode by default. */ perf->ops.read = gen8_oa_read; + i915_perf_init_info(i915); if (IS_GRAPHICS_VER(i915, 8, 9)) { perf->ops.is_valid_b_counter_reg = @@ -4473,18 +4838,6 @@ void i915_perf_init(struct drm_i915_private *i915) perf->ops.enable_metric_set = gen8_enable_metric_set; perf->ops.disable_metric_set = gen8_disable_metric_set; perf->ops.oa_hw_tail_read = gen8_oa_hw_tail_read; - - if (GRAPHICS_VER(i915) == 8) { - perf->ctx_oactxctrl_offset = 0x120; - perf->ctx_flexeu0_offset = 0x2ce; - - perf->gen8_valid_ctx_bit = BIT(25); - } else { - perf->ctx_oactxctrl_offset = 0x128; - perf->ctx_flexeu0_offset = 0x3de; - - perf->gen8_valid_ctx_bit = BIT(16); - } } else if (GRAPHICS_VER(i915) == 11) { perf->ops.is_valid_b_counter_reg = gen7_is_valid_b_counter_addr; @@ -4498,13 +4851,10 @@ void i915_perf_init(struct drm_i915_private *i915) perf->ops.enable_metric_set = gen8_enable_metric_set; perf->ops.disable_metric_set = gen11_disable_metric_set; perf->ops.oa_hw_tail_read = gen8_oa_hw_tail_read; - - perf->ctx_oactxctrl_offset = 0x124; - perf->ctx_flexeu0_offset = 0x78e; - - perf->gen8_valid_ctx_bit = BIT(16); } else if (GRAPHICS_VER(i915) == 12) { perf->ops.is_valid_b_counter_reg = + HAS_OA_SLICE_CONTRIB_LIMITS(i915) ? + xehp_is_valid_b_counter_addr : gen12_is_valid_b_counter_addr; perf->ops.is_valid_mux_reg = gen12_is_valid_mux_addr; @@ -4516,14 +4866,15 @@ void i915_perf_init(struct drm_i915_private *i915) perf->ops.enable_metric_set = gen12_enable_metric_set; perf->ops.disable_metric_set = gen12_disable_metric_set; perf->ops.oa_hw_tail_read = gen12_oa_hw_tail_read; - - perf->ctx_flexeu0_offset = 0; - perf->ctx_oactxctrl_offset = 0x144; } } if (perf->ops.enable_metric_set) { - mutex_init(&perf->lock); + struct intel_gt *gt; + int i; + + for_each_gt(gt, i915, i) + mutex_init(>->perf.lock); /* Choose a representative limit */ oa_sample_rate_hard_limit = to_gt(i915)->clock_frequency / 2; diff --git a/drivers/gpu/drm/i915/i915_perf.h b/drivers/gpu/drm/i915/i915_perf.h index 1d1329e5af3a..f96e09a4af04 100644 --- a/drivers/gpu/drm/i915/i915_perf.h +++ b/drivers/gpu/drm/i915/i915_perf.h @@ -57,4 +57,6 @@ static inline void i915_oa_config_put(struct i915_oa_config *oa_config) kref_put(&oa_config->ref, i915_oa_config_release); } +u32 i915_perf_oa_timestamp_frequency(struct drm_i915_private *i915); + #endif /* __I915_PERF_H__ */ diff --git a/drivers/gpu/drm/i915/i915_perf_oa_regs.h b/drivers/gpu/drm/i915/i915_perf_oa_regs.h index f31c9f13a9fc..381d94101610 100644 --- a/drivers/gpu/drm/i915/i915_perf_oa_regs.h +++ b/drivers/gpu/drm/i915/i915_perf_oa_regs.h @@ -97,7 +97,7 @@ #define GEN12_OAR_OACONTROL_COUNTER_FORMAT_SHIFT 1 #define GEN12_OAR_OACONTROL_COUNTER_ENABLE (1 << 0) -#define GEN12_OACTXCONTROL _MMIO(0x2360) +#define GEN12_OACTXCONTROL(base) _MMIO((base) + 0x360) #define GEN12_OAR_OASTATUS _MMIO(0x2968) /* Gen12 OAG unit */ @@ -134,4 +134,8 @@ #define GDT_CHICKEN_BITS _MMIO(0x9840) #define GT_NOA_ENABLE 0x00000080 +#define GEN12_SQCNT1 _MMIO(0x8718) +#define GEN12_SQCNT1_PMON_ENABLE REG_BIT(30) +#define GEN12_SQCNT1_OABPC REG_BIT(29) + #endif /* __INTEL_PERF_OA_REGS__ */ diff --git a/drivers/gpu/drm/i915/i915_perf_types.h b/drivers/gpu/drm/i915/i915_perf_types.h index 05cb9a335a97..ca150b7af3f2 100644 --- a/drivers/gpu/drm/i915/i915_perf_types.h +++ b/drivers/gpu/drm/i915/i915_perf_types.h @@ -147,6 +147,11 @@ struct i915_perf_stream { struct intel_engine_cs *engine; /** + * @lock: Lock associated with operations on stream + */ + struct mutex lock; + + /** * @sample_flags: Flags representing the `DRM_I915_PERF_PROP_SAMPLE_*` * properties given when opening a stream, representing the contents * of a single sample as read() by userspace. @@ -245,11 +250,10 @@ struct i915_perf_stream { * @oa_buffer: State of the OA buffer. */ struct { + const struct i915_oa_format *format; struct i915_vma *vma; u8 *vaddr; u32 last_ctx_id; - int format; - int format_size; int size_exponent; /** @@ -380,6 +384,26 @@ struct i915_oa_ops { u32 (*oa_hw_tail_read)(struct i915_perf_stream *stream); }; +struct i915_perf_gt { + /* + * Lock associated with anything below within this structure. + */ + struct mutex lock; + + /** + * @sseu: sseu configuration selected to run while perf is active, + * applies to all contexts. + */ + struct intel_sseu sseu; + + /* + * @exclusive_stream: The stream currently using the OA unit. This is + * sometimes accessed outside a syscall associated to its file + * descriptor. + */ + struct i915_perf_stream *exclusive_stream; +}; + struct i915_perf { struct drm_i915_private *i915; @@ -397,25 +421,6 @@ struct i915_perf { */ struct idr metrics_idr; - /* - * Lock associated with anything below within this structure - * except exclusive_stream. - */ - struct mutex lock; - - /* - * The stream currently using the OA unit. If accessed - * outside a syscall associated to its file - * descriptor. - */ - struct i915_perf_stream *exclusive_stream; - - /** - * @sseu: sseu configuration selected to run while perf is active, - * applies to all contexts. - */ - struct intel_sseu sseu; - /** * For rate limiting any notifications of spurious * invalid OA reports diff --git a/drivers/gpu/drm/i915/i915_pmu.c b/drivers/gpu/drm/i915/i915_pmu.c index 958b37123bf1..52531ab28c5f 100644 --- a/drivers/gpu/drm/i915/i915_pmu.c +++ b/drivers/gpu/drm/i915/i915_pmu.c @@ -148,13 +148,13 @@ static u64 __get_rc6(struct intel_gt *gt) struct drm_i915_private *i915 = gt->i915; u64 val; - val = intel_rc6_residency_ns(>->rc6, GEN6_GT_GFX_RC6); + val = intel_rc6_residency_ns(>->rc6, INTEL_RC6_RES_RC6); if (HAS_RC6p(i915)) - val += intel_rc6_residency_ns(>->rc6, GEN6_GT_GFX_RC6p); + val += intel_rc6_residency_ns(>->rc6, INTEL_RC6_RES_RC6p); if (HAS_RC6pp(i915)) - val += intel_rc6_residency_ns(>->rc6, GEN6_GT_GFX_RC6pp); + val += intel_rc6_residency_ns(>->rc6, INTEL_RC6_RES_RC6pp); return val; } @@ -371,7 +371,6 @@ static void frequency_sample(struct intel_gt *gt, unsigned int period_ns) { struct drm_i915_private *i915 = gt->i915; - struct intel_uncore *uncore = gt->uncore; struct i915_pmu *pmu = &i915->pmu; struct intel_rps *rps = >->rps; @@ -394,7 +393,7 @@ frequency_sample(struct intel_gt *gt, unsigned int period_ns) * case we assume the system is running at the intended * frequency. Fortunately, the read should rarely fail! */ - val = intel_uncore_read_fw(uncore, GEN6_RPSTAT1); + val = intel_rps_read_rpstat_fw(rps); if (val) val = intel_rps_get_cagf(rps, val); else diff --git a/drivers/gpu/drm/i915/i915_query.c b/drivers/gpu/drm/i915/i915_query.c index 6ec9c9fb7b0d..00871ef99792 100644 --- a/drivers/gpu/drm/i915/i915_query.c +++ b/drivers/gpu/drm/i915/i915_query.c @@ -250,8 +250,9 @@ static int query_perf_config_data(struct drm_i915_private *i915, return total_size; if (query_item->length < total_size) { - DRM_DEBUG("Invalid query config data item size=%u expected=%u\n", - query_item->length, total_size); + drm_dbg(&i915->drm, + "Invalid query config data item size=%u expected=%u\n", + query_item->length, total_size); return -EINVAL; } @@ -418,9 +419,10 @@ static int query_perf_config_list(struct drm_i915_private *i915, } while (n_configs > alloc); if (query_item->length < sizeof_perf_config_list(n_configs)) { - DRM_DEBUG("Invalid query config list item size=%u expected=%zu\n", - query_item->length, - sizeof_perf_config_list(n_configs)); + drm_dbg(&i915->drm, + "Invalid query config list item size=%u expected=%zu\n", + query_item->length, + sizeof_perf_config_list(n_configs)); kfree(oa_config_ids); return -EINVAL; } diff --git a/drivers/gpu/drm/i915/i915_reg.h b/drivers/gpu/drm/i915/i915_reg.h index da35bb2db26b..8e1892d14774 100644 --- a/drivers/gpu/drm/i915/i915_reg.h +++ b/drivers/gpu/drm/i915/i915_reg.h @@ -26,6 +26,7 @@ #define _I915_REG_H_ #include "i915_reg_defs.h" +#include "display/intel_display_reg_defs.h" /** * DOC: The i915 register macro definition style guide @@ -115,75 +116,6 @@ * #define GEN8_BAR _MMIO(0xb888) */ -#define DISPLAY_MMIO_BASE(dev_priv) (INTEL_INFO(dev_priv)->display.mmio_offset) - -/* - * Given the first two numbers __a and __b of arbitrarily many evenly spaced - * numbers, pick the 0-based __index'th value. - * - * Always prefer this over _PICK() if the numbers are evenly spaced. - */ -#define _PICK_EVEN(__index, __a, __b) ((__a) + (__index) * ((__b) - (__a))) - -/* - * Given the arbitrary numbers in varargs, pick the 0-based __index'th number. - * - * Always prefer _PICK_EVEN() over this if the numbers are evenly spaced. - */ -#define _PICK(__index, ...) (((const u32 []){ __VA_ARGS__ })[__index]) - -/* - * Named helper wrappers around _PICK_EVEN() and _PICK(). - */ -#define _PIPE(pipe, a, b) _PICK_EVEN(pipe, a, b) -#define _PLANE(plane, a, b) _PICK_EVEN(plane, a, b) -#define _TRANS(tran, a, b) _PICK_EVEN(tran, a, b) -#define _PORT(port, a, b) _PICK_EVEN(port, a, b) -#define _PLL(pll, a, b) _PICK_EVEN(pll, a, b) -#define _PHY(phy, a, b) _PICK_EVEN(phy, a, b) - -#define _MMIO_PIPE(pipe, a, b) _MMIO(_PIPE(pipe, a, b)) -#define _MMIO_PLANE(plane, a, b) _MMIO(_PLANE(plane, a, b)) -#define _MMIO_TRANS(tran, a, b) _MMIO(_TRANS(tran, a, b)) -#define _MMIO_PORT(port, a, b) _MMIO(_PORT(port, a, b)) -#define _MMIO_PLL(pll, a, b) _MMIO(_PLL(pll, a, b)) -#define _MMIO_PHY(phy, a, b) _MMIO(_PHY(phy, a, b)) - -#define _PHY3(phy, ...) _PICK(phy, __VA_ARGS__) - -#define _MMIO_PIPE3(pipe, a, b, c) _MMIO(_PICK(pipe, a, b, c)) -#define _MMIO_PORT3(pipe, a, b, c) _MMIO(_PICK(pipe, a, b, c)) -#define _MMIO_PHY3(phy, a, b, c) _MMIO(_PHY3(phy, a, b, c)) -#define _MMIO_PLL3(pll, ...) _MMIO(_PICK(pll, __VA_ARGS__)) - - -/* - * Device info offset array based helpers for groups of registers with unevenly - * spaced base offsets. - */ -#define _MMIO_PIPE2(pipe, reg) _MMIO(INTEL_INFO(dev_priv)->display.pipe_offsets[(pipe)] - \ - INTEL_INFO(dev_priv)->display.pipe_offsets[PIPE_A] + \ - DISPLAY_MMIO_BASE(dev_priv) + (reg)) -#define _MMIO_TRANS2(tran, reg) _MMIO(INTEL_INFO(dev_priv)->display.trans_offsets[(tran)] - \ - INTEL_INFO(dev_priv)->display.trans_offsets[TRANSCODER_A] + \ - DISPLAY_MMIO_BASE(dev_priv) + (reg)) -#define _MMIO_CURSOR2(pipe, reg) _MMIO(INTEL_INFO(dev_priv)->display.cursor_offsets[(pipe)] - \ - INTEL_INFO(dev_priv)->display.cursor_offsets[PIPE_A] + \ - DISPLAY_MMIO_BASE(dev_priv) + (reg)) - -#define __MASKED_FIELD(mask, value) ((mask) << 16 | (value)) -#define _MASKED_FIELD(mask, value) ({ \ - if (__builtin_constant_p(mask)) \ - BUILD_BUG_ON_MSG(((mask) & 0xffff0000), "Incorrect mask"); \ - if (__builtin_constant_p(value)) \ - BUILD_BUG_ON_MSG((value) & 0xffff0000, "Incorrect value"); \ - if (__builtin_constant_p(mask) && __builtin_constant_p(value)) \ - BUILD_BUG_ON_MSG((value) & ~(mask), \ - "Incorrect value for mask"); \ - __MASKED_FIELD(mask, value); }) -#define _MASKED_BIT_ENABLE(a) ({ typeof(a) _a = (a); _MASKED_FIELD(_a, _a); }) -#define _MASKED_BIT_DISABLE(a) (_MASKED_FIELD((a), 0)) - #define GU_CNTL _MMIO(0x101010) #define LMEM_INIT REG_BIT(7) @@ -970,6 +902,7 @@ #define GEN11_VEBOX2_RING_BASE 0x1d8000 #define XEHP_VEBOX3_RING_BASE 0x1e8000 #define XEHP_VEBOX4_RING_BASE 0x1f8000 +#define MTL_GSC_RING_BASE 0x11a000 #define GEN12_COMPUTE0_RING_BASE 0x1a000 #define GEN12_COMPUTE1_RING_BASE 0x1c000 #define GEN12_COMPUTE2_RING_BASE 0x1e000 @@ -1147,11 +1080,6 @@ #define MBUS_JOIN_PIPE_SELECT(pipe) REG_FIELD_PREP(MBUS_JOIN_PIPE_SELECT_MASK, pipe) #define MBUS_JOIN_PIPE_SELECT_NONE MBUS_JOIN_PIPE_SELECT(7) -#define HDPORT_STATE _MMIO(0x45050) -#define HDPORT_DPLL_USED_MASK REG_GENMASK(15, 12) -#define HDPORT_DDI_USED(phy) REG_BIT(2 * (phy) + 1) -#define HDPORT_ENABLED REG_BIT(0) - /* Make render/texture TLB fetches lower priorty than associated data * fetches. This is not turned on by default */ @@ -1781,9 +1709,10 @@ #define _PALETTE_A 0xa000 #define _PALETTE_B 0xa800 #define _CHV_PALETTE_C 0xc000 -#define PALETTE_RED_MASK REG_GENMASK(23, 16) -#define PALETTE_GREEN_MASK REG_GENMASK(15, 8) -#define PALETTE_BLUE_MASK REG_GENMASK(7, 0) +/* 8bit mode / i965+ 10.6 interpolated mode ldw/udw */ +#define PALETTE_RED_MASK REG_GENMASK(23, 16) +#define PALETTE_GREEN_MASK REG_GENMASK(15, 8) +#define PALETTE_BLUE_MASK REG_GENMASK(7, 0) #define PALETTE(pipe, i) _MMIO(DISPLAY_MMIO_BASE(dev_priv) + \ _PICK((pipe), _PALETTE_A, \ _PALETTE_B, _CHV_PALETTE_C) + \ @@ -1796,6 +1725,15 @@ #define XEHPSDV_RP_STATE_CAP _MMIO(0x250014) #define PVC_RP_STATE_CAP _MMIO(0x281014) +#define MTL_RP_STATE_CAP _MMIO(0x138000) +#define MTL_MEDIAP_STATE_CAP _MMIO(0x138020) +#define MTL_RP0_CAP_MASK REG_GENMASK(8, 0) +#define MTL_RPN_CAP_MASK REG_GENMASK(24, 16) + +#define MTL_GT_RPE_FREQUENCY _MMIO(0x13800c) +#define MTL_MPE_FREQUENCY _MMIO(0x13802c) +#define MTL_RPE_MASK REG_GENMASK(8, 0) + #define GT0_PERF_LIMIT_REASONS _MMIO(0x1381a8) #define GT0_PERF_LIMIT_REASONS_MASK 0xde3 #define PROCHOT_MASK REG_BIT(0) @@ -1806,6 +1744,8 @@ #define POWER_LIMIT_4_MASK REG_BIT(8) #define POWER_LIMIT_1_MASK REG_BIT(10) #define POWER_LIMIT_2_MASK REG_BIT(11) +#define GT0_PERF_LIMIT_REASONS_LOG_MASK REG_GENMASK(31, 16) +#define MTL_MEDIA_PERF_LIMIT_REASONS _MMIO(0x138030) #define CHV_CLK_CTL1 _MMIO(0x101100) #define VLV_CLK_CTL2 _MMIO(0x101104) @@ -3747,9 +3687,10 @@ /* Skylake+ pipe bottom (background) color */ #define _SKL_BOTTOM_COLOR_A 0x70034 +#define _SKL_BOTTOM_COLOR_B 0x71034 #define SKL_BOTTOM_COLOR_GAMMA_ENABLE REG_BIT(31) #define SKL_BOTTOM_COLOR_CSC_ENABLE REG_BIT(30) -#define SKL_BOTTOM_COLOR(pipe) _MMIO_PIPE2(pipe, _SKL_BOTTOM_COLOR_A) +#define SKL_BOTTOM_COLOR(pipe) _MMIO_PIPE(pipe, _SKL_BOTTOM_COLOR_A, _SKL_BOTTOM_COLOR_B) #define _ICL_PIPE_A_STATUS 0x70058 #define ICL_PIPESTATUS(pipe) _MMIO_PIPE2(pipe, _ICL_PIPE_A_STATUS) @@ -5367,17 +5308,24 @@ /* legacy palette */ #define _LGC_PALETTE_A 0x4a000 #define _LGC_PALETTE_B 0x4a800 -#define LGC_PALETTE_RED_MASK REG_GENMASK(23, 16) -#define LGC_PALETTE_GREEN_MASK REG_GENMASK(15, 8) -#define LGC_PALETTE_BLUE_MASK REG_GENMASK(7, 0) +/* see PALETTE_* for the bits */ #define LGC_PALETTE(pipe, i) _MMIO(_PIPE(pipe, _LGC_PALETTE_A, _LGC_PALETTE_B) + (i) * 4) /* ilk/snb precision palette */ #define _PREC_PALETTE_A 0x4b000 #define _PREC_PALETTE_B 0x4c000 -#define PREC_PALETTE_RED_MASK REG_GENMASK(29, 20) -#define PREC_PALETTE_GREEN_MASK REG_GENMASK(19, 10) -#define PREC_PALETTE_BLUE_MASK REG_GENMASK(9, 0) +/* 10bit mode */ +#define PREC_PALETTE_10_RED_MASK REG_GENMASK(29, 20) +#define PREC_PALETTE_10_GREEN_MASK REG_GENMASK(19, 10) +#define PREC_PALETTE_10_BLUE_MASK REG_GENMASK(9, 0) +/* 12.4 interpolated mode ldw */ +#define PREC_PALETTE_12P4_RED_LDW_MASK REG_GENMASK(29, 24) +#define PREC_PALETTE_12P4_GREEN_LDW_MASK REG_GENMASK(19, 14) +#define PREC_PALETTE_12P4_BLUE_LDW_MASK REG_GENMASK(9, 4) +/* 12.4 interpolated mode udw */ +#define PREC_PALETTE_12P4_RED_UDW_MASK REG_GENMASK(29, 20) +#define PREC_PALETTE_12P4_GREEN_UDW_MASK REG_GENMASK(19, 10) +#define PREC_PALETTE_12P4_BLUE_UDW_MASK REG_GENMASK(9, 0) #define PREC_PALETTE(pipe, i) _MMIO(_PIPE(pipe, _PREC_PALETTE_A, _PREC_PALETTE_B) + (i) * 4) #define _PREC_PIPEAGCMAX 0x4d000 @@ -5847,6 +5795,11 @@ #define ICL_DSSM_CDCLK_PLL_REFCLK_19_2MHz (1 << 29) #define ICL_DSSM_CDCLK_PLL_REFCLK_38_4MHz (2 << 29) +#define GMD_ID_DISPLAY _MMIO(0x510a0) +#define GMD_ID_ARCH_MASK REG_GENMASK(31, 22) +#define GMD_ID_RELEASE_MASK REG_GENMASK(21, 14) +#define GMD_ID_STEP REG_GENMASK(5, 0) + /*GEN11 chicken */ #define _PIPEA_CHICKEN 0x70038 #define _PIPEB_CHICKEN 0x71038 @@ -6647,6 +6600,12 @@ #define DG1_PCODE_STATUS 0x7E #define DG1_UNCORE_GET_INIT_STATUS 0x0 #define DG1_UNCORE_INIT_STATUS_COMPLETE 0x1 +#define PCODE_POWER_SETUP 0x7C +#define POWER_SETUP_SUBCOMMAND_READ_I1 0x4 +#define POWER_SETUP_SUBCOMMAND_WRITE_I1 0x5 +#define POWER_SETUP_I1_WATTS REG_BIT(31) +#define POWER_SETUP_I1_SHIFT 6 /* 10.6 fixed point format */ +#define POWER_SETUP_I1_DATA_MASK REG_GENMASK(15, 0) #define GEN12_PCODE_READ_SAGV_BLOCK_TIME_US 0x23 #define XEHP_PCODE_FREQUENCY_CONFIG 0x6e /* xehpsdv, pvc */ /* XEHP_PCODE_FREQUENCY_CONFIG sub-commands (param1) */ @@ -6656,6 +6615,15 @@ /* XEHP_PCODE_FREQUENCY_CONFIG param2 */ #define PCODE_MBOX_DOMAIN_NONE 0x0 #define PCODE_MBOX_DOMAIN_MEDIAFF 0x3 + +/* Wa_14017210380: mtl */ +#define PCODE_MBOX_GT_STATE 0x50 +/* sub-commands (param1) */ +#define PCODE_MBOX_GT_STATE_MEDIA_BUSY 0x1 +#define PCODE_MBOX_GT_STATE_MEDIA_NOT_BUSY 0x2 +/* param2 */ +#define PCODE_MBOX_GT_STATE_DOMAIN_MEDIA 0x1 + #define GEN6_PCODE_DATA _MMIO(0x138128) #define GEN6_PCODE_FREQ_IA_RATIO_SHIFT 8 #define GEN6_PCODE_FREQ_RING_RATIO_SHIFT 16 @@ -7413,185 +7381,6 @@ enum skl_power_gate { _ADLS_DPLL4_CFGCR1, \ _ADLS_DPLL3_CFGCR1) -#define _DKL_PHY1_BASE 0x168000 -#define _DKL_PHY2_BASE 0x169000 -#define _DKL_PHY3_BASE 0x16A000 -#define _DKL_PHY4_BASE 0x16B000 -#define _DKL_PHY5_BASE 0x16C000 -#define _DKL_PHY6_BASE 0x16D000 - -#define DKL_REG_TC_PORT(__reg) \ - (TC_PORT_1 + ((__reg).reg - _DKL_PHY1_BASE) / (_DKL_PHY2_BASE - _DKL_PHY1_BASE)) - -/* DEKEL PHY MMIO Address = Phy base + (internal address & ~index_mask) */ -#define _DKL_PCS_DW5 0x14 -#define DKL_PCS_DW5(tc_port) _MMIO(_PORT(tc_port, _DKL_PHY1_BASE, \ - _DKL_PHY2_BASE) + \ - _DKL_PCS_DW5) -#define DKL_PCS_DW5_CORE_SOFTRESET REG_BIT(11) - -#define _DKL_PLL_DIV0 0x200 -#define DKL_PLL_DIV0_AFC_STARTUP_MASK REG_GENMASK(27, 25) -#define DKL_PLL_DIV0_AFC_STARTUP(val) REG_FIELD_PREP(DKL_PLL_DIV0_AFC_STARTUP_MASK, (val)) -#define DKL_PLL_DIV0_INTEG_COEFF(x) ((x) << 16) -#define DKL_PLL_DIV0_INTEG_COEFF_MASK (0x1F << 16) -#define DKL_PLL_DIV0_PROP_COEFF(x) ((x) << 12) -#define DKL_PLL_DIV0_PROP_COEFF_MASK (0xF << 12) -#define DKL_PLL_DIV0_FBPREDIV_SHIFT (8) -#define DKL_PLL_DIV0_FBPREDIV(x) ((x) << DKL_PLL_DIV0_FBPREDIV_SHIFT) -#define DKL_PLL_DIV0_FBPREDIV_MASK (0xF << DKL_PLL_DIV0_FBPREDIV_SHIFT) -#define DKL_PLL_DIV0_FBDIV_INT(x) ((x) << 0) -#define DKL_PLL_DIV0_FBDIV_INT_MASK (0xFF << 0) -#define DKL_PLL_DIV0_MASK (DKL_PLL_DIV0_INTEG_COEFF_MASK | \ - DKL_PLL_DIV0_PROP_COEFF_MASK | \ - DKL_PLL_DIV0_FBPREDIV_MASK | \ - DKL_PLL_DIV0_FBDIV_INT_MASK) -#define DKL_PLL_DIV0(tc_port) _MMIO(_PORT(tc_port, _DKL_PHY1_BASE, \ - _DKL_PHY2_BASE) + \ - _DKL_PLL_DIV0) - -#define _DKL_PLL_DIV1 0x204 -#define DKL_PLL_DIV1_IREF_TRIM(x) ((x) << 16) -#define DKL_PLL_DIV1_IREF_TRIM_MASK (0x1F << 16) -#define DKL_PLL_DIV1_TDC_TARGET_CNT(x) ((x) << 0) -#define DKL_PLL_DIV1_TDC_TARGET_CNT_MASK (0xFF << 0) -#define DKL_PLL_DIV1(tc_port) _MMIO(_PORT(tc_port, _DKL_PHY1_BASE, \ - _DKL_PHY2_BASE) + \ - _DKL_PLL_DIV1) - -#define _DKL_PLL_SSC 0x210 -#define DKL_PLL_SSC_IREF_NDIV_RATIO(x) ((x) << 29) -#define DKL_PLL_SSC_IREF_NDIV_RATIO_MASK (0x7 << 29) -#define DKL_PLL_SSC_STEP_LEN(x) ((x) << 16) -#define DKL_PLL_SSC_STEP_LEN_MASK (0xFF << 16) -#define DKL_PLL_SSC_STEP_NUM(x) ((x) << 11) -#define DKL_PLL_SSC_STEP_NUM_MASK (0x7 << 11) -#define DKL_PLL_SSC_EN (1 << 9) -#define DKL_PLL_SSC(tc_port) _MMIO(_PORT(tc_port, _DKL_PHY1_BASE, \ - _DKL_PHY2_BASE) + \ - _DKL_PLL_SSC) - -#define _DKL_PLL_BIAS 0x214 -#define DKL_PLL_BIAS_FRAC_EN_H (1 << 30) -#define DKL_PLL_BIAS_FBDIV_SHIFT (8) -#define DKL_PLL_BIAS_FBDIV_FRAC(x) ((x) << DKL_PLL_BIAS_FBDIV_SHIFT) -#define DKL_PLL_BIAS_FBDIV_FRAC_MASK (0x3FFFFF << DKL_PLL_BIAS_FBDIV_SHIFT) -#define DKL_PLL_BIAS(tc_port) _MMIO(_PORT(tc_port, _DKL_PHY1_BASE, \ - _DKL_PHY2_BASE) + \ - _DKL_PLL_BIAS) - -#define _DKL_PLL_TDC_COLDST_BIAS 0x218 -#define DKL_PLL_TDC_SSC_STEP_SIZE(x) ((x) << 8) -#define DKL_PLL_TDC_SSC_STEP_SIZE_MASK (0xFF << 8) -#define DKL_PLL_TDC_FEED_FWD_GAIN(x) ((x) << 0) -#define DKL_PLL_TDC_FEED_FWD_GAIN_MASK (0xFF << 0) -#define DKL_PLL_TDC_COLDST_BIAS(tc_port) _MMIO(_PORT(tc_port, \ - _DKL_PHY1_BASE, \ - _DKL_PHY2_BASE) + \ - _DKL_PLL_TDC_COLDST_BIAS) - -#define _DKL_REFCLKIN_CTL 0x12C -/* Bits are the same as MG_REFCLKIN_CTL */ -#define DKL_REFCLKIN_CTL(tc_port) _MMIO(_PORT(tc_port, \ - _DKL_PHY1_BASE, \ - _DKL_PHY2_BASE) + \ - _DKL_REFCLKIN_CTL) - -#define _DKL_CLKTOP2_HSCLKCTL 0xD4 -/* Bits are the same as MG_CLKTOP2_HSCLKCTL */ -#define DKL_CLKTOP2_HSCLKCTL(tc_port) _MMIO(_PORT(tc_port, \ - _DKL_PHY1_BASE, \ - _DKL_PHY2_BASE) + \ - _DKL_CLKTOP2_HSCLKCTL) - -#define _DKL_CLKTOP2_CORECLKCTL1 0xD8 -/* Bits are the same as MG_CLKTOP2_CORECLKCTL1 */ -#define DKL_CLKTOP2_CORECLKCTL1(tc_port) _MMIO(_PORT(tc_port, \ - _DKL_PHY1_BASE, \ - _DKL_PHY2_BASE) + \ - _DKL_CLKTOP2_CORECLKCTL1) - -#define _DKL_TX_DPCNTL0 0x2C0 -#define DKL_TX_PRESHOOT_COEFF(x) ((x) << 13) -#define DKL_TX_PRESHOOT_COEFF_MASK (0x1f << 13) -#define DKL_TX_DE_EMPHASIS_COEFF(x) ((x) << 8) -#define DKL_TX_DE_EMPAHSIS_COEFF_MASK (0x1f << 8) -#define DKL_TX_VSWING_CONTROL(x) ((x) << 0) -#define DKL_TX_VSWING_CONTROL_MASK (0x7 << 0) -#define DKL_TX_DPCNTL0(tc_port) _MMIO(_PORT(tc_port, \ - _DKL_PHY1_BASE, \ - _DKL_PHY2_BASE) + \ - _DKL_TX_DPCNTL0) - -#define _DKL_TX_DPCNTL1 0x2C4 -/* Bits are the same as DKL_TX_DPCNTRL0 */ -#define DKL_TX_DPCNTL1(tc_port) _MMIO(_PORT(tc_port, \ - _DKL_PHY1_BASE, \ - _DKL_PHY2_BASE) + \ - _DKL_TX_DPCNTL1) - -#define _DKL_TX_DPCNTL2 0x2C8 -#define DKL_TX_DP20BITMODE REG_BIT(2) -#define DKL_TX_DPCNTL2_CFG_LOADGENSELECT_TX1_MASK REG_GENMASK(4, 3) -#define DKL_TX_DPCNTL2_CFG_LOADGENSELECT_TX1(val) REG_FIELD_PREP(DKL_TX_DPCNTL2_CFG_LOADGENSELECT_TX1_MASK, (val)) -#define DKL_TX_DPCNTL2_CFG_LOADGENSELECT_TX2_MASK REG_GENMASK(6, 5) -#define DKL_TX_DPCNTL2_CFG_LOADGENSELECT_TX2(val) REG_FIELD_PREP(DKL_TX_DPCNTL2_CFG_LOADGENSELECT_TX2_MASK, (val)) -#define DKL_TX_DPCNTL2(tc_port) _MMIO(_PORT(tc_port, \ - _DKL_PHY1_BASE, \ - _DKL_PHY2_BASE) + \ - _DKL_TX_DPCNTL2) - -#define _DKL_TX_FW_CALIB 0x2F8 -#define DKL_TX_CFG_DISABLE_WAIT_INIT (1 << 7) -#define DKL_TX_FW_CALIB(tc_port) _MMIO(_PORT(tc_port, \ - _DKL_PHY1_BASE, \ - _DKL_PHY2_BASE) + \ - _DKL_TX_FW_CALIB) - -#define _DKL_TX_PMD_LANE_SUS 0xD00 -#define DKL_TX_PMD_LANE_SUS(tc_port) _MMIO(_PORT(tc_port, \ - _DKL_PHY1_BASE, \ - _DKL_PHY2_BASE) + \ - _DKL_TX_PMD_LANE_SUS) - -#define _DKL_TX_DW17 0xDC4 -#define DKL_TX_DW17(tc_port) _MMIO(_PORT(tc_port, \ - _DKL_PHY1_BASE, \ - _DKL_PHY2_BASE) + \ - _DKL_TX_DW17) - -#define _DKL_TX_DW18 0xDC8 -#define DKL_TX_DW18(tc_port) _MMIO(_PORT(tc_port, \ - _DKL_PHY1_BASE, \ - _DKL_PHY2_BASE) + \ - _DKL_TX_DW18) - -#define _DKL_DP_MODE 0xA0 -#define DKL_DP_MODE(tc_port) _MMIO(_PORT(tc_port, \ - _DKL_PHY1_BASE, \ - _DKL_PHY2_BASE) + \ - _DKL_DP_MODE) - -#define _DKL_CMN_UC_DW27 0x36C -#define DKL_CMN_UC_DW27_UC_HEALTH (0x1 << 15) -#define DKL_CMN_UC_DW_27(tc_port) _MMIO(_PORT(tc_port, \ - _DKL_PHY1_BASE, \ - _DKL_PHY2_BASE) + \ - _DKL_CMN_UC_DW27) - -/* - * Each Dekel PHY is addressed through a 4KB aperture. Each PHY has more than - * 4KB of register space, so a separate index is programmed in HIP_INDEX_REG0 - * or HIP_INDEX_REG1, based on the port number, to set the upper 2 address - * bits that point the 4KB window into the full PHY register space. - */ -#define _HIP_INDEX_REG0 0x1010A0 -#define _HIP_INDEX_REG1 0x1010A4 -#define HIP_INDEX_REG(tc_port) _MMIO((tc_port) < 4 ? _HIP_INDEX_REG0 \ - : _HIP_INDEX_REG1) -#define _HIP_INDEX_SHIFT(tc_port) (8 * ((tc_port) % 4)) -#define HIP_INDEX_VAL(tc_port, val) ((val) << _HIP_INDEX_SHIFT(tc_port)) - /* BXT display engine PLL */ #define BXT_DE_PLL_CTL _MMIO(0x6d000) #define BXT_DE_PLL_RATIO(x) (x) /* {60,65,100} * 19.2MHz */ @@ -7775,12 +7564,10 @@ enum skl_power_gate { #define _PAL_PREC_DATA_A 0x4A404 #define _PAL_PREC_DATA_B 0x4AC04 #define _PAL_PREC_DATA_C 0x4B404 +/* see PREC_PALETTE_* for the bits */ #define _PAL_PREC_GC_MAX_A 0x4A410 #define _PAL_PREC_GC_MAX_B 0x4AC10 #define _PAL_PREC_GC_MAX_C 0x4B410 -#define PREC_PAL_DATA_RED_MASK REG_GENMASK(29, 20) -#define PREC_PAL_DATA_GREEN_MASK REG_GENMASK(19, 10) -#define PREC_PAL_DATA_BLUE_MASK REG_GENMASK(9, 0) #define _PAL_PREC_EXT_GC_MAX_A 0x4A420 #define _PAL_PREC_EXT_GC_MAX_B 0x4AC20 #define _PAL_PREC_EXT_GC_MAX_C 0x4B420 @@ -7813,12 +7600,7 @@ enum skl_power_gate { #define _PAL_PREC_MULTI_SEG_DATA_A 0x4A40C #define _PAL_PREC_MULTI_SEG_DATA_B 0x4AC0C -#define PAL_PREC_MULTI_SEG_RED_LDW_MASK REG_GENMASK(29, 24) -#define PAL_PREC_MULTI_SEG_RED_UDW_MASK REG_GENMASK(29, 20) -#define PAL_PREC_MULTI_SEG_GREEN_LDW_MASK REG_GENMASK(19, 14) -#define PAL_PREC_MULTI_SEG_GREEN_UDW_MASK REG_GENMASK(19, 10) -#define PAL_PREC_MULTI_SEG_BLUE_LDW_MASK REG_GENMASK(9, 4) -#define PAL_PREC_MULTI_SEG_BLUE_UDW_MASK REG_GENMASK(9, 0) +/* see PREC_PALETTE_12P4_* for the bits */ #define PREC_PAL_MULTI_SEG_INDEX(pipe) _MMIO_PIPE(pipe, \ _PAL_PREC_MULTI_SEG_INDEX_A, \ @@ -7879,13 +7661,17 @@ enum skl_power_gate { #define _CGM_PIPE_A_CSC_COEFF67 (VLV_DISPLAY_BASE + 0x6790C) #define _CGM_PIPE_A_CSC_COEFF8 (VLV_DISPLAY_BASE + 0x67910) #define _CGM_PIPE_A_DEGAMMA (VLV_DISPLAY_BASE + 0x66000) -#define CGM_PIPE_DEGAMMA_RED_MASK REG_GENMASK(13, 0) -#define CGM_PIPE_DEGAMMA_GREEN_MASK REG_GENMASK(29, 16) -#define CGM_PIPE_DEGAMMA_BLUE_MASK REG_GENMASK(13, 0) +/* cgm degamma ldw */ +#define CGM_PIPE_DEGAMMA_GREEN_LDW_MASK REG_GENMASK(29, 16) +#define CGM_PIPE_DEGAMMA_BLUE_LDW_MASK REG_GENMASK(13, 0) +/* cgm degamma udw */ +#define CGM_PIPE_DEGAMMA_RED_UDW_MASK REG_GENMASK(13, 0) #define _CGM_PIPE_A_GAMMA (VLV_DISPLAY_BASE + 0x67000) -#define CGM_PIPE_GAMMA_RED_MASK REG_GENMASK(9, 0) -#define CGM_PIPE_GAMMA_GREEN_MASK REG_GENMASK(25, 16) -#define CGM_PIPE_GAMMA_BLUE_MASK REG_GENMASK(9, 0) +/* cgm gamma ldw */ +#define CGM_PIPE_GAMMA_GREEN_LDW_MASK REG_GENMASK(25, 16) +#define CGM_PIPE_GAMMA_BLUE_LDW_MASK REG_GENMASK(9, 0) +/* cgm gamma udw */ +#define CGM_PIPE_GAMMA_RED_UDW_MASK REG_GENMASK(9, 0) #define _CGM_PIPE_A_MODE (VLV_DISPLAY_BASE + 0x67A00) #define CGM_PIPE_MODE_GAMMA (1 << 2) #define CGM_PIPE_MODE_CSC (1 << 1) @@ -7962,8 +7748,13 @@ enum skl_power_gate { _ICL_PIPE_DSS_CTL2_PB, \ _ICL_PIPE_DSS_CTL2_PC) +#define GGC _MMIO(0x108040) +#define GMS_MASK REG_GENMASK(15, 8) +#define GGMS_MASK REG_GENMASK(7, 6) + #define GEN12_GSMBASE _MMIO(0x108100) #define GEN12_DSMBASE _MMIO(0x1080C0) +#define GEN12_BDSM_MASK REG_GENMASK64(63, 20) #define XEHP_CLOCK_GATE_DIS _MMIO(0x101014) #define SGSI_SIDECLK_DIS REG_BIT(17) @@ -8344,6 +8135,11 @@ enum skl_power_gate { #define GEN12_CULLBIT2 _MMIO(0x7030) #define GEN12_STATE_ACK_DEBUG _MMIO(0x20BC) +#define _MTL_CLKGATE_DIS_TRANS_A 0x604E8 +#define _MTL_CLKGATE_DIS_TRANS_B 0x614E8 +#define MTL_CLKGATE_DIS_TRANS(trans) _MMIO_TRANS2(trans, _MTL_CLKGATE_DIS_TRANS_A) +#define MTL_CLKGATE_DIS_TRANS_DMASC_GATING_DIS REG_BIT(7) + #define MTL_LATENCY_LP0_LP1 _MMIO(0x45780) #define MTL_LATENCY_LP2_LP3 _MMIO(0x45784) #define MTL_LATENCY_LP4_LP5 _MMIO(0x45788) @@ -8367,4 +8163,6 @@ enum skl_power_gate { #define MTL_TRAS_MASK REG_GENMASK(16, 8) #define MTL_TRDPRE_MASK REG_GENMASK(7, 0) +#define MTL_MEDIA_GSI_BASE 0x380000 + #endif /* _I915_REG_H_ */ diff --git a/drivers/gpu/drm/i915/i915_reg_defs.h b/drivers/gpu/drm/i915/i915_reg_defs.h index 8f486f77609f..be43580a6979 100644 --- a/drivers/gpu/drm/i915/i915_reg_defs.h +++ b/drivers/gpu/drm/i915/i915_reg_defs.h @@ -98,29 +98,54 @@ */ #define REG_FIELD_GET64(__mask, __val) ((u64)FIELD_GET(__mask, __val)) +#define __MASKED_FIELD(mask, value) ((mask) << 16 | (value)) +#define _MASKED_FIELD(mask, value) ({ \ + if (__builtin_constant_p(mask)) \ + BUILD_BUG_ON_MSG(((mask) & 0xffff0000), "Incorrect mask"); \ + if (__builtin_constant_p(value)) \ + BUILD_BUG_ON_MSG((value) & 0xffff0000, "Incorrect value"); \ + if (__builtin_constant_p(mask) && __builtin_constant_p(value)) \ + BUILD_BUG_ON_MSG((value) & ~(mask), \ + "Incorrect value for mask"); \ + __MASKED_FIELD(mask, value); }) +#define _MASKED_BIT_ENABLE(a) ({ typeof(a) _a = (a); _MASKED_FIELD(_a, _a); }) +#define _MASKED_BIT_DISABLE(a) (_MASKED_FIELD((a), 0)) + +/* + * Given the first two numbers __a and __b of arbitrarily many evenly spaced + * numbers, pick the 0-based __index'th value. + * + * Always prefer this over _PICK() if the numbers are evenly spaced. + */ +#define _PICK_EVEN(__index, __a, __b) ((__a) + (__index) * ((__b) - (__a))) + +/* + * Given the arbitrary numbers in varargs, pick the 0-based __index'th number. + * + * Always prefer _PICK_EVEN() over this if the numbers are evenly spaced. + */ +#define _PICK(__index, ...) (((const u32 []){ __VA_ARGS__ })[__index]) + typedef struct { u32 reg; } i915_reg_t; #define _MMIO(r) ((const i915_reg_t){ .reg = (r) }) -#define INVALID_MMIO_REG _MMIO(0) - -static __always_inline u32 i915_mmio_reg_offset(i915_reg_t reg) -{ - return reg.reg; -} - -static inline bool i915_mmio_reg_equal(i915_reg_t a, i915_reg_t b) -{ - return i915_mmio_reg_offset(a) == i915_mmio_reg_offset(b); -} +typedef struct { + u32 reg; +} i915_mcr_reg_t; -static inline bool i915_mmio_reg_valid(i915_reg_t reg) -{ - return !i915_mmio_reg_equal(reg, INVALID_MMIO_REG); -} +#define INVALID_MMIO_REG _MMIO(0) -#define VLV_DISPLAY_BASE 0x180000 +/* + * These macros can be used on either i915_reg_t or i915_mcr_reg_t since they're + * simply operations on the register's offset and don't care about the MCR vs + * non-MCR nature of the register. + */ +#define i915_mmio_reg_offset(r) \ + _Generic((r), i915_reg_t: (r).reg, i915_mcr_reg_t: (r).reg) +#define i915_mmio_reg_equal(a, b) (i915_mmio_reg_offset(a) == i915_mmio_reg_offset(b)) +#define i915_mmio_reg_valid(r) (!i915_mmio_reg_equal(r, INVALID_MMIO_REG)) #endif /* __I915_REG_DEFS__ */ diff --git a/drivers/gpu/drm/i915/i915_request.c b/drivers/gpu/drm/i915/i915_request.c index 62fad16a55e8..f949a9495758 100644 --- a/drivers/gpu/drm/i915/i915_request.c +++ b/drivers/gpu/drm/i915/i915_request.c @@ -1621,6 +1621,20 @@ i915_request_await_object(struct i915_request *to, return ret; } +static void i915_request_await_huc(struct i915_request *rq) +{ + struct intel_huc *huc = &rq->context->engine->gt->uc.huc; + + /* don't stall kernel submissions! */ + if (!rcu_access_pointer(rq->context->gem_context)) + return; + + if (intel_huc_wait_required(huc)) + i915_sw_fence_await_sw_fence(&rq->submit, + &huc->delayed_load.fence, + &rq->hucq); +} + static struct i915_request * __i915_request_ensure_parallel_ordering(struct i915_request *rq, struct intel_timeline *timeline) @@ -1703,6 +1717,16 @@ __i915_request_add_to_timeline(struct i915_request *rq) struct i915_request *prev; /* + * Media workloads may require HuC, so stall them until HuC loading is + * complete. Note that HuC not being loaded when a user submission + * arrives can only happen when HuC is loaded via GSC and in that case + * we still expect the window between us starting to accept submissions + * and HuC loading completion to be small (a few hundred ms). + */ + if (rq->engine->class == VIDEO_DECODE_CLASS) + i915_request_await_huc(rq); + + /* * Dependency tracking and request ordering along the timeline * is special cased so that we can eliminate redundant ordering * operations while building the request (we know that the timeline diff --git a/drivers/gpu/drm/i915/i915_request.h b/drivers/gpu/drm/i915/i915_request.h index 47041ec68df8..f5e1bb5e857a 100644 --- a/drivers/gpu/drm/i915/i915_request.h +++ b/drivers/gpu/drm/i915/i915_request.h @@ -348,6 +348,11 @@ struct i915_request { #define GUC_PRIO_FINI 0xfe u8 guc_prio; + /** + * @hucq: wait queue entry used to wait on the HuC load to complete + */ + wait_queue_entry_t hucq; + I915_SELFTEST_DECLARE(struct { struct list_head link; unsigned long delay; diff --git a/drivers/gpu/drm/i915/i915_scatterlist.c b/drivers/gpu/drm/i915/i915_scatterlist.c index dcc081874ec8..114e5e39aa72 100644 --- a/drivers/gpu/drm/i915/i915_scatterlist.c +++ b/drivers/gpu/drm/i915/i915_scatterlist.c @@ -158,7 +158,7 @@ struct i915_refct_sgt *i915_rsgt_from_buddy_resource(struct ttm_resource *res, u32 page_alignment) { struct i915_ttm_buddy_resource *bman_res = to_ttm_buddy_resource(res); - const u64 size = res->num_pages << PAGE_SHIFT; + const u64 size = res->size; const u32 max_segment = round_down(UINT_MAX, page_alignment); struct drm_buddy *mm = bman_res->mm; struct list_head *blocks = &bman_res->blocks; @@ -177,7 +177,7 @@ struct i915_refct_sgt *i915_rsgt_from_buddy_resource(struct ttm_resource *res, i915_refct_sgt_init(rsgt, size); st = &rsgt->table; - if (sg_alloc_table(st, res->num_pages, GFP_KERNEL)) { + if (sg_alloc_table(st, PFN_UP(res->size), GFP_KERNEL)) { i915_refct_sgt_put(rsgt); return ERR_PTR(-ENOMEM); } diff --git a/drivers/gpu/drm/i915/i915_selftest.h b/drivers/gpu/drm/i915/i915_selftest.h index f54de0499be7..bdf3e22c0a34 100644 --- a/drivers/gpu/drm/i915/i915_selftest.h +++ b/drivers/gpu/drm/i915/i915_selftest.h @@ -92,12 +92,14 @@ int __i915_subtests(const char *caller, T, ARRAY_SIZE(T), data) #define i915_live_subtests(T, data) ({ \ typecheck(struct drm_i915_private *, data); \ + (data)->gt[0]->uc.guc.submission_state.sched_disable_delay_ms = 0; \ __i915_subtests(__func__, \ __i915_live_setup, __i915_live_teardown, \ T, ARRAY_SIZE(T), data); \ }) #define intel_gt_live_subtests(T, data) ({ \ typecheck(struct intel_gt *, data); \ + (data)->uc.guc.submission_state.sched_disable_delay_ms = 0; \ __i915_subtests(__func__, \ __intel_gt_live_setup, __intel_gt_live_teardown, \ T, ARRAY_SIZE(T), data); \ diff --git a/drivers/gpu/drm/i915/i915_sw_fence.c b/drivers/gpu/drm/i915/i915_sw_fence.c index 6fc0d1b89690..cc2a8821d22a 100644 --- a/drivers/gpu/drm/i915/i915_sw_fence.c +++ b/drivers/gpu/drm/i915/i915_sw_fence.c @@ -571,7 +571,6 @@ int __i915_sw_fence_await_dma_fence(struct i915_sw_fence *fence, int i915_sw_fence_await_reservation(struct i915_sw_fence *fence, struct dma_resv *resv, - const struct dma_fence_ops *exclude, bool write, unsigned long timeout, gfp_t gfp) diff --git a/drivers/gpu/drm/i915/i915_sw_fence.h b/drivers/gpu/drm/i915/i915_sw_fence.h index 619fc5a22f0c..f752bfc7c6e1 100644 --- a/drivers/gpu/drm/i915/i915_sw_fence.h +++ b/drivers/gpu/drm/i915/i915_sw_fence.h @@ -91,7 +91,6 @@ int i915_sw_fence_await_dma_fence(struct i915_sw_fence *fence, int i915_sw_fence_await_reservation(struct i915_sw_fence *fence, struct dma_resv *resv, - const struct dma_fence_ops *exclude, bool write, unsigned long timeout, gfp_t gfp); diff --git a/drivers/gpu/drm/i915/i915_sysfs.c b/drivers/gpu/drm/i915/i915_sysfs.c index 1e2750210831..595e8b574990 100644 --- a/drivers/gpu/drm/i915/i915_sysfs.c +++ b/drivers/gpu/drm/i915/i915_sysfs.c @@ -218,7 +218,8 @@ static const struct bin_attribute error_state_attr = { static void i915_setup_error_capture(struct device *kdev) { if (sysfs_create_bin_file(&kdev->kobj, &error_state_attr)) - DRM_ERROR("error_state sysfs setup failed\n"); + drm_err(&kdev_minor_to_i915(kdev)->drm, + "error_state sysfs setup failed\n"); } static void i915_teardown_error_capture(struct device *kdev) diff --git a/drivers/gpu/drm/i915/i915_trace.h b/drivers/gpu/drm/i915/i915_trace.h index 37b5c9e9d260..f6f9228a1351 100644 --- a/drivers/gpu/drm/i915/i915_trace.h +++ b/drivers/gpu/drm/i915/i915_trace.h @@ -15,7 +15,6 @@ #include "gt/intel_engine.h" #include "i915_drv.h" -#include "i915_irq.h" /* object tracking */ @@ -671,21 +670,6 @@ TRACE_EVENT_CONDITION(i915_reg_rw, (u32)(__entry->val >> 32)) ); -TRACE_EVENT(intel_gpu_freq_change, - TP_PROTO(u32 freq), - TP_ARGS(freq), - - TP_STRUCT__entry( - __field(u32, freq) - ), - - TP_fast_assign( - __entry->freq = freq; - ), - - TP_printk("new_freq=%u", __entry->freq) -); - /** * DOC: i915_ppgtt_create and i915_ppgtt_release tracepoints * diff --git a/drivers/gpu/drm/i915/i915_ttm_buddy_manager.c b/drivers/gpu/drm/i915/i915_ttm_buddy_manager.c index e19452f0e100..7e611476c7a4 100644 --- a/drivers/gpu/drm/i915/i915_ttm_buddy_manager.c +++ b/drivers/gpu/drm/i915/i915_ttm_buddy_manager.c @@ -62,8 +62,8 @@ static int i915_ttm_buddy_man_alloc(struct ttm_resource_manager *man, if (place->fpfn || lpfn != man->size) bman_res->flags |= DRM_BUDDY_RANGE_ALLOCATION; - GEM_BUG_ON(!bman_res->base.num_pages); - size = bman_res->base.num_pages << PAGE_SHIFT; + GEM_BUG_ON(!bman_res->base.size); + size = bman_res->base.size; min_page_size = bman->default_page_size; if (bo->page_alignment) @@ -72,7 +72,7 @@ static int i915_ttm_buddy_man_alloc(struct ttm_resource_manager *man, GEM_BUG_ON(min_page_size < mm->chunk_size); GEM_BUG_ON(!IS_ALIGNED(size, min_page_size)); - if (place->fpfn + bman_res->base.num_pages != place->lpfn && + if (place->fpfn + PFN_UP(bman_res->base.size) != place->lpfn && place->flags & TTM_PL_FLAG_CONTIGUOUS) { unsigned long pages; @@ -108,7 +108,7 @@ static int i915_ttm_buddy_man_alloc(struct ttm_resource_manager *man, goto err_free_blocks; if (place->flags & TTM_PL_FLAG_CONTIGUOUS) { - u64 original_size = (u64)bman_res->base.num_pages << PAGE_SHIFT; + u64 original_size = (u64)bman_res->base.size; drm_buddy_block_trim(mm, original_size, @@ -116,7 +116,7 @@ static int i915_ttm_buddy_man_alloc(struct ttm_resource_manager *man, } if (lpfn <= bman->visible_size) { - bman_res->used_visible_size = bman_res->base.num_pages; + bman_res->used_visible_size = PFN_UP(bman_res->base.size); } else { struct drm_buddy_block *block; @@ -228,7 +228,7 @@ static bool i915_ttm_buddy_man_compatible(struct ttm_resource_manager *man, if (!place->fpfn && place->lpfn == i915_ttm_buddy_man_visible_size(man)) - return bman_res->used_visible_size == res->num_pages; + return bman_res->used_visible_size == PFN_UP(res->size); /* Check each drm buddy block individually */ list_for_each_entry(block, &bman_res->blocks, link) { diff --git a/drivers/gpu/drm/i915/i915_vma.c b/drivers/gpu/drm/i915/i915_vma.c index f17c09ead7d7..703fee6b5f75 100644 --- a/drivers/gpu/drm/i915/i915_vma.c +++ b/drivers/gpu/drm/i915/i915_vma.c @@ -73,14 +73,16 @@ static void vma_print_allocator(struct i915_vma *vma, const char *reason) char buf[512]; if (!vma->node.stack) { - DRM_DEBUG_DRIVER("vma.node [%08llx + %08llx] %s: unknown owner\n", - vma->node.start, vma->node.size, reason); + drm_dbg(&to_i915(vma->obj->base.dev)->drm, + "vma.node [%08llx + %08llx] %s: unknown owner\n", + vma->node.start, vma->node.size, reason); return; } stack_depot_snprint(vma->node.stack, buf, sizeof(buf), 0); - DRM_DEBUG_DRIVER("vma.node [%08llx + %08llx] %s: inserted at %s\n", - vma->node.start, vma->node.size, reason, buf); + drm_dbg(&to_i915(vma->obj->base.dev)->drm, + "vma.node [%08llx + %08llx] %s: inserted at %s\n", + vma->node.start, vma->node.size, reason, buf); } #else @@ -776,21 +778,15 @@ i915_vma_insert(struct i915_vma *vma, struct i915_gem_ww_ctx *ww, GEM_BUG_ON(!IS_ALIGNED(end, I915_GTT_PAGE_SIZE)); alignment = max(alignment, i915_vm_obj_min_alignment(vma->vm, vma->obj)); - /* - * for compact-pt we round up the reservation to prevent - * any smaller pages being used within the same PDE - */ - if (NEEDS_COMPACT_PT(vma->vm->i915)) - size = round_up(size, alignment); /* If binding the object/GGTT view requires more space than the entire * aperture has, reject it early before evicting everything in a vain * attempt to find space. */ if (size > end) { - DRM_DEBUG("Attempting to bind an object larger than the aperture: request=%llu > %s aperture=%llu\n", - size, flags & PIN_MAPPABLE ? "mappable" : "total", - end); + drm_dbg(&to_i915(vma->obj->base.dev)->drm, + "Attempting to bind an object larger than the aperture: request=%llu > %s aperture=%llu\n", + size, flags & PIN_MAPPABLE ? "mappable" : "total", end); return -ENOSPC; } @@ -820,7 +816,8 @@ i915_vma_insert(struct i915_vma *vma, struct i915_gem_ww_ctx *ww, * forseeable future. See also i915_ggtt_offset(). */ if (upper_32_bits(end - 1) && - vma->page_sizes.sg > I915_GTT_PAGE_SIZE) { + vma->page_sizes.sg > I915_GTT_PAGE_SIZE && + !HAS_64K_PAGES(vma->vm->i915)) { /* * We can't mix 64K and 4K PTEs in the same page-table * (2M block), and so to avoid the ugliness and @@ -1847,6 +1844,11 @@ int _i915_vma_move_to_active(struct i915_vma *vma, GEM_BUG_ON(!vma->pages); + if (!(flags & __EXEC_OBJECT_NO_REQUEST_AWAIT)) { + err = i915_request_await_object(rq, vma->obj, flags & EXEC_OBJECT_WRITE); + if (unlikely(err)) + return err; + } err = __i915_vma_move_to_active(vma, rq); if (unlikely(err)) return err; diff --git a/drivers/gpu/drm/i915/i915_vma.h b/drivers/gpu/drm/i915/i915_vma.h index aecd9c64486b..0757977a489b 100644 --- a/drivers/gpu/drm/i915/i915_vma.h +++ b/drivers/gpu/drm/i915/i915_vma.h @@ -55,6 +55,7 @@ static inline bool i915_vma_is_active(const struct i915_vma *vma) /* do not reserve memory to prevent deadlocks */ #define __EXEC_OBJECT_NO_RESERVE BIT(31) +#define __EXEC_OBJECT_NO_REQUEST_AWAIT BIT(30) int __must_check _i915_vma_move_to_active(struct i915_vma *vma, struct i915_request *rq, diff --git a/drivers/gpu/drm/i915/intel_device_info.c b/drivers/gpu/drm/i915/intel_device_info.c index 20575eb77ea7..849baf6c3b3c 100644 --- a/drivers/gpu/drm/i915/intel_device_info.c +++ b/drivers/gpu/drm/i915/intel_device_info.c @@ -29,9 +29,11 @@ #include "display/intel_cdclk.h" #include "display/intel_de.h" -#include "intel_device_info.h" +#include "gt/intel_gt_regs.h" #include "i915_drv.h" +#include "i915_reg.h" #include "i915_utils.h" +#include "intel_device_info.h" #define PLATFORM_NAME(x) [INTEL_##x] = #x static const char * const platform_names[] = { @@ -231,7 +233,7 @@ static bool find_devid(u16 id, const u16 *p, unsigned int num) return false; } -void intel_device_info_subplatform_init(struct drm_i915_private *i915) +static void intel_device_info_subplatform_init(struct drm_i915_private *i915) { const struct intel_device_info *info = INTEL_INFO(i915); const struct intel_runtime_info *rinfo = RUNTIME_INFO(i915); @@ -288,6 +290,78 @@ void intel_device_info_subplatform_init(struct drm_i915_private *i915) RUNTIME_INFO(i915)->platform_mask[pi] |= mask; } +static void ip_ver_read(struct drm_i915_private *i915, u32 offset, struct intel_ip_version *ip) +{ + struct pci_dev *pdev = to_pci_dev(i915->drm.dev); + void __iomem *addr; + u32 val; + u8 expected_ver = ip->ver; + u8 expected_rel = ip->rel; + + addr = pci_iomap_range(pdev, 0, offset, sizeof(u32)); + if (drm_WARN_ON(&i915->drm, !addr)) + return; + + val = ioread32(addr); + pci_iounmap(pdev, addr); + + ip->ver = REG_FIELD_GET(GMD_ID_ARCH_MASK, val); + ip->rel = REG_FIELD_GET(GMD_ID_RELEASE_MASK, val); + ip->step = REG_FIELD_GET(GMD_ID_STEP, val); + + /* Sanity check against expected versions from device info */ + if (IP_VER(ip->ver, ip->rel) < IP_VER(expected_ver, expected_rel)) + drm_dbg(&i915->drm, + "Hardware reports GMD IP version %u.%u (REG[0x%x] = 0x%08x) but minimum expected is %u.%u\n", + ip->ver, ip->rel, offset, val, expected_ver, expected_rel); +} + +/* + * Setup the graphics version for the current device. This must be done before + * any code that performs checks on GRAPHICS_VER or DISPLAY_VER, so this + * function should be called very early in the driver initialization sequence. + * + * Regular MMIO access is not yet setup at the point this function is called so + * we peek at the appropriate MMIO offset directly. The GMD_ID register is + * part of an 'always on' power well by design, so we don't need to worry about + * forcewake while reading it. + */ +static void intel_ipver_early_init(struct drm_i915_private *i915) +{ + struct intel_runtime_info *runtime = RUNTIME_INFO(i915); + + if (!HAS_GMD_ID(i915)) { + drm_WARN_ON(&i915->drm, RUNTIME_INFO(i915)->graphics.ip.ver > 12); + /* + * On older platforms, graphics and media share the same ip + * version and release. + */ + RUNTIME_INFO(i915)->media.ip = + RUNTIME_INFO(i915)->graphics.ip; + return; + } + + ip_ver_read(i915, i915_mmio_reg_offset(GMD_ID_GRAPHICS), + &runtime->graphics.ip); + ip_ver_read(i915, i915_mmio_reg_offset(GMD_ID_DISPLAY), + &runtime->display.ip); + ip_ver_read(i915, i915_mmio_reg_offset(GMD_ID_MEDIA), + &runtime->media.ip); +} + +/** + * intel_device_info_runtime_init_early - initialize early runtime info + * @i915: the i915 device + * + * Determine early intel_device_info fields at runtime. This function needs + * to be called before the MMIO has been setup. + */ +void intel_device_info_runtime_init_early(struct drm_i915_private *i915) +{ + intel_ipver_early_init(i915); + intel_device_info_subplatform_init(i915); +} + /** * intel_device_info_runtime_init - initialize runtime info * @dev_priv: the i915 device @@ -415,7 +489,7 @@ void intel_device_info_runtime_init(struct drm_i915_private *dev_priv) if (DISPLAY_VER(dev_priv) >= 11 && (dfsm & ICL_DFSM_DMC_DISABLE)) runtime->has_dmc = 0; - if (DISPLAY_VER(dev_priv) >= 10 && + if (IS_DISPLAY_VER(dev_priv, 10, 12) && (dfsm & GLK_DFSM_DISPLAY_DSC_DISABLE)) runtime->has_dsc = 0; } @@ -442,6 +516,11 @@ void intel_device_info_runtime_init(struct drm_i915_private *dev_priv) runtime->has_dmc = false; runtime->has_dsc = false; } + + /* Disable nuclear pageflip by default on pre-g4x */ + if (!dev_priv->params.nuclear_pageflip && + DISPLAY_VER(dev_priv) < 5 && !IS_G4X(dev_priv)) + dev_priv->drm.driver_features &= ~DRIVER_ATOMIC; } void intel_driver_caps_print(const struct intel_driver_caps *caps, diff --git a/drivers/gpu/drm/i915/intel_device_info.h b/drivers/gpu/drm/i915/intel_device_info.h index d638235e1d26..d588e5fd2eea 100644 --- a/drivers/gpu/drm/i915/intel_device_info.h +++ b/drivers/gpu/drm/i915/intel_device_info.h @@ -146,13 +146,13 @@ enum intel_ppgtt_type { /* Keep has_* in alphabetical order */ \ func(has_64bit_reloc); \ func(has_64k_pages); \ - func(needs_compact_pt); \ func(gpu_reset_clobbers_display); \ func(has_reset_engine); \ func(has_3d_pipeline); \ func(has_4tile); \ func(has_flat_ccs); \ func(has_global_mocs); \ + func(has_gmd_id); \ func(has_gt_uc); \ func(has_heci_pxp); \ func(has_heci_gscfi); \ @@ -164,6 +164,8 @@ enum intel_ppgtt_type { func(has_logical_ring_elsq); \ func(has_media_ratio_mode); \ func(has_mslice_steering); \ + func(has_oa_bpc_reporting); \ + func(has_oa_slice_contrib_limits); \ func(has_one_eu_per_fuse_bit); \ func(has_pxp); \ func(has_rc6); \ @@ -180,6 +182,7 @@ enum intel_ppgtt_type { /* Keep in alphabetical order */ \ func(cursor_needs_physical); \ func(has_cdclk_crawl); \ + func(has_cdclk_squash); \ func(has_ddi); \ func(has_dp_mst); \ func(has_dsb); \ @@ -195,20 +198,25 @@ enum intel_ppgtt_type { func(overlay_needs_physical); \ func(supports_tv); -struct ip_version { +struct intel_ip_version { u8 ver; u8 rel; + u8 step; }; struct intel_runtime_info { + /* + * Single "graphics" IP version that represents + * render, compute and copy behavior. + */ struct { - struct ip_version ip; + struct intel_ip_version ip; } graphics; struct { - struct ip_version ip; + struct intel_ip_version ip; } media; struct { - struct ip_version ip; + struct intel_ip_version ip; } display; /* @@ -308,7 +316,7 @@ struct intel_driver_caps { const char *intel_platform_name(enum intel_platform platform); -void intel_device_info_subplatform_init(struct drm_i915_private *dev_priv); +void intel_device_info_runtime_init_early(struct drm_i915_private *dev_priv); void intel_device_info_runtime_init(struct drm_i915_private *dev_priv); void intel_device_info_print(const struct intel_device_info *info, diff --git a/drivers/gpu/drm/i915/intel_gvt_mmio_table.c b/drivers/gpu/drm/i915/intel_gvt_mmio_table.c index 8279dc580a3e..ce6b3c3b636a 100644 --- a/drivers/gpu/drm/i915/intel_gvt_mmio_table.c +++ b/drivers/gpu/drm/i915/intel_gvt_mmio_table.c @@ -6,6 +6,7 @@ #include "display/intel_audio_regs.h" #include "display/intel_backlight_regs.h" #include "display/intel_dmc_regs.h" +#include "display/intel_dpio_phy.h" #include "display/vlv_dsi_pll_regs.h" #include "gt/intel_gt_regs.h" #include "gvt/gvt.h" @@ -102,7 +103,7 @@ static int iterate_generic_mmio(struct intel_gvt_mmio_table_iter *iter) MMIO_D(_MMIO(0x2438)); MMIO_D(_MMIO(0x243c)); MMIO_D(_MMIO(0x7018)); - MMIO_D(HALF_SLICE_CHICKEN3); + MMIO_D(HSW_HALF_SLICE_CHICKEN3); MMIO_D(GEN7_HALF_SLICE_CHICKEN1); /* display */ MMIO_F(_MMIO(0x60220), 0x20); diff --git a/drivers/gpu/drm/i915/intel_mchbar_regs.h b/drivers/gpu/drm/i915/intel_mchbar_regs.h index ffc702b79579..f93e9af43ac3 100644 --- a/drivers/gpu/drm/i915/intel_mchbar_regs.h +++ b/drivers/gpu/drm/i915/intel_mchbar_regs.h @@ -189,6 +189,21 @@ #define DG1_QCLK_RATIO_MASK REG_GENMASK(9, 2) #define DG1_QCLK_REFERENCE REG_BIT(10) +/* + * *_PACKAGE_POWER_SKU - SKU power and timing parameters. + */ +#define PCU_PACKAGE_POWER_SKU _MMIO(MCHBAR_MIRROR_BASE_SNB + 0x5930) +#define PKG_PKG_TDP GENMASK_ULL(14, 0) +#define PKG_MAX_WIN GENMASK_ULL(54, 48) +#define PKG_MAX_WIN_X GENMASK_ULL(54, 53) +#define PKG_MAX_WIN_Y GENMASK_ULL(52, 48) + +#define PCU_PACKAGE_POWER_SKU_UNIT _MMIO(MCHBAR_MIRROR_BASE_SNB + 0x5938) +#define PKG_PWR_UNIT REG_GENMASK(3, 0) +#define PKG_ENERGY_UNIT REG_GENMASK(12, 8) +#define PKG_TIME_UNIT REG_GENMASK(19, 16) +#define PCU_PACKAGE_ENERGY_STATUS _MMIO(MCHBAR_MIRROR_BASE_SNB + 0x593c) + #define GEN6_GT_PERF_STATUS _MMIO(MCHBAR_MIRROR_BASE_SNB + 0x5948) #define GEN6_RP_STATE_LIMITS _MMIO(MCHBAR_MIRROR_BASE_SNB + 0x5994) #define GEN6_RP_STATE_CAP _MMIO(MCHBAR_MIRROR_BASE_SNB + 0x5998) @@ -198,6 +213,12 @@ #define GEN10_FREQ_INFO_REC _MMIO(MCHBAR_MIRROR_BASE_SNB + 0x5ef0) #define RPE_MASK REG_GENMASK(15, 8) +#define PCU_PACKAGE_RAPL_LIMIT _MMIO(MCHBAR_MIRROR_BASE_SNB + 0x59a0) +#define PKG_PWR_LIM_1 REG_GENMASK(14, 0) +#define PKG_PWR_LIM_1_EN REG_BIT(15) +#define PKG_PWR_LIM_1_TIME REG_GENMASK(23, 17) +#define PKG_PWR_LIM_1_TIME_X REG_GENMASK(23, 22) +#define PKG_PWR_LIM_1_TIME_Y REG_GENMASK(21, 17) /* snb MCH registers for priority tuning */ #define MCH_SSKPD _MMIO(MCHBAR_MIRROR_BASE_SNB + 0x5d10) diff --git a/drivers/gpu/drm/i915/intel_pci_config.h b/drivers/gpu/drm/i915/intel_pci_config.h index 4977a524ce6f..23b8e519f333 100644 --- a/drivers/gpu/drm/i915/intel_pci_config.h +++ b/drivers/gpu/drm/i915/intel_pci_config.h @@ -7,11 +7,29 @@ #define __INTEL_PCI_CONFIG_H__ /* PCI BARs */ -#define GTTMMADR_BAR 0 -#define GEN2_GTTMMADR_BAR 1 -#define GFXMEM_BAR 2 -#define GTT_APERTURE_BAR GFXMEM_BAR -#define GEN12_LMEM_BAR GFXMEM_BAR +#define GEN2_GMADR_BAR 0 +#define GEN2_MMADR_BAR 1 /* MMIO+GTT, despite the name */ +#define GEN2_IO_BAR 2 /* 85x/865 */ + +#define GEN3_MMADR_BAR 0 /* MMIO only */ +#define GEN3_IO_BAR 1 +#define GEN3_GMADR_BAR 2 +#define GEN3_GTTADR_BAR 3 /* GTT only */ + +#define GEN4_GTTMMADR_BAR 0 /* MMIO+GTT */ +#define GEN4_GMADR_BAR 2 +#define GEN4_IO_BAR 4 + +#define GEN12_LMEM_BAR 2 + +static inline int intel_mmio_bar(int graphics_ver) +{ + switch (graphics_ver) { + case 2: return GEN2_MMADR_BAR; + case 3: return GEN3_MMADR_BAR; + default: return GEN4_GTTMMADR_BAR; + } +} /* BSM in include/drm/i915_drm.h */ diff --git a/drivers/gpu/drm/i915/intel_pm.c b/drivers/gpu/drm/i915/intel_pm.c index 8f86f56e7ca4..73c88b1c9545 100644 --- a/drivers/gpu/drm/i915/intel_pm.c +++ b/drivers/gpu/drm/i915/intel_pm.c @@ -30,6 +30,8 @@ #include "display/skl_watermark.h" #include "gt/intel_engine_regs.h" +#include "gt/intel_gt.h" +#include "gt/intel_gt_mcr.h" #include "gt/intel_gt_regs.h" #include "i915_drv.h" @@ -58,25 +60,20 @@ static void gen9_init_clock_gating(struct drm_i915_private *dev_priv) * Must match Sampler, Pixel Back End, and Media. See * WaCompressedResourceSamplerPbeMediaNewHashMode. */ - intel_uncore_write(&dev_priv->uncore, CHICKEN_PAR1_1, - intel_uncore_read(&dev_priv->uncore, CHICKEN_PAR1_1) | - SKL_DE_COMPRESSED_HASH_MODE); + intel_uncore_rmw(&dev_priv->uncore, CHICKEN_PAR1_1, 0, SKL_DE_COMPRESSED_HASH_MODE); } /* See Bspec note for PSR2_CTL bit 31, Wa#828:skl,bxt,kbl,cfl */ - intel_uncore_write(&dev_priv->uncore, CHICKEN_PAR1_1, - intel_uncore_read(&dev_priv->uncore, CHICKEN_PAR1_1) | SKL_EDP_PSR_FIX_RDWRAP); + intel_uncore_rmw(&dev_priv->uncore, CHICKEN_PAR1_1, 0, SKL_EDP_PSR_FIX_RDWRAP); /* WaEnableChickenDCPR:skl,bxt,kbl,glk,cfl */ - intel_uncore_write(&dev_priv->uncore, GEN8_CHICKEN_DCPR_1, - intel_uncore_read(&dev_priv->uncore, GEN8_CHICKEN_DCPR_1) | MASK_WAKEMEM); + intel_uncore_rmw(&dev_priv->uncore, GEN8_CHICKEN_DCPR_1, 0, MASK_WAKEMEM); /* * WaFbcWakeMemOn:skl,bxt,kbl,glk,cfl * Display WA #0859: skl,bxt,kbl,glk,cfl */ - intel_uncore_write(&dev_priv->uncore, DISP_ARB_CTL, intel_uncore_read(&dev_priv->uncore, DISP_ARB_CTL) | - DISP_FBC_MEMORY_WAKE); + intel_uncore_rmw(&dev_priv->uncore, DISP_ARB_CTL, 0, DISP_FBC_MEMORY_WAKE); } static void bxt_init_clock_gating(struct drm_i915_private *dev_priv) @@ -84,15 +81,13 @@ static void bxt_init_clock_gating(struct drm_i915_private *dev_priv) gen9_init_clock_gating(dev_priv); /* WaDisableSDEUnitClockGating:bxt */ - intel_uncore_write(&dev_priv->uncore, GEN8_UCGCTL6, intel_uncore_read(&dev_priv->uncore, GEN8_UCGCTL6) | - GEN8_SDEUNIT_CLOCK_GATE_DISABLE); + intel_uncore_rmw(&dev_priv->uncore, GEN8_UCGCTL6, 0, GEN8_SDEUNIT_CLOCK_GATE_DISABLE); /* * FIXME: * GEN8_HDCUNIT_CLOCK_GATE_DISABLE_HDCREQ applies on 3x6 GT SKUs only. */ - intel_uncore_write(&dev_priv->uncore, GEN8_UCGCTL6, intel_uncore_read(&dev_priv->uncore, GEN8_UCGCTL6) | - GEN8_HDCUNIT_CLOCK_GATE_DISABLE_HDCREQ); + intel_uncore_rmw(&dev_priv->uncore, GEN8_UCGCTL6, 0, GEN8_HDCUNIT_CLOCK_GATE_DISABLE_HDCREQ); /* * Wa: Backlight PWM may stop in the asserted state, causing backlight @@ -113,16 +108,13 @@ static void bxt_init_clock_gating(struct drm_i915_private *dev_priv) * WaFbcTurnOffFbcWatermark:bxt * Display WA #0562: bxt */ - intel_uncore_write(&dev_priv->uncore, DISP_ARB_CTL, intel_uncore_read(&dev_priv->uncore, DISP_ARB_CTL) | - DISP_FBC_WM_DIS); + intel_uncore_rmw(&dev_priv->uncore, DISP_ARB_CTL, 0, DISP_FBC_WM_DIS); /* * WaFbcHighMemBwCorruptionAvoidance:bxt * Display WA #0883: bxt */ - intel_uncore_write(&dev_priv->uncore, ILK_DPFC_CHICKEN(INTEL_FBC_A), - intel_uncore_read(&dev_priv->uncore, ILK_DPFC_CHICKEN(INTEL_FBC_A)) | - DPFC_DISABLE_DUMMY0); + intel_uncore_rmw(&dev_priv->uncore, ILK_DPFC_CHICKEN(INTEL_FBC_A), 0, DPFC_DISABLE_DUMMY0); } static void glk_init_clock_gating(struct drm_i915_private *dev_priv) @@ -895,19 +887,14 @@ static void pnv_update_wm(struct drm_i915_private *dev_priv) wm = intel_calculate_wm(pixel_rate, &pnv_cursor_wm, pnv_display_wm.fifo_size, 4, latency->cursor_sr); - reg = intel_uncore_read(&dev_priv->uncore, DSPFW3); - reg &= ~DSPFW_CURSOR_SR_MASK; - reg |= FW_WM(wm, CURSOR_SR); - intel_uncore_write(&dev_priv->uncore, DSPFW3, reg); + intel_uncore_rmw(&dev_priv->uncore, DSPFW3, DSPFW_CURSOR_SR_MASK, + FW_WM(wm, CURSOR_SR)); /* Display HPLL off SR */ wm = intel_calculate_wm(pixel_rate, &pnv_display_hplloff_wm, pnv_display_hplloff_wm.fifo_size, cpp, latency->display_hpll_disable); - reg = intel_uncore_read(&dev_priv->uncore, DSPFW3); - reg &= ~DSPFW_HPLL_SR_MASK; - reg |= FW_WM(wm, HPLL_SR); - intel_uncore_write(&dev_priv->uncore, DSPFW3, reg); + intel_uncore_rmw(&dev_priv->uncore, DSPFW3, DSPFW_HPLL_SR_MASK, FW_WM(wm, HPLL_SR)); /* cursor HPLL off SR */ wm = intel_calculate_wm(pixel_rate, &pnv_cursor_hplloff_wm, @@ -1337,34 +1324,14 @@ static bool g4x_compute_fbc_en(const struct g4x_wm_state *wm_state, return true; } -static int g4x_compute_pipe_wm(struct intel_atomic_state *state, - struct intel_crtc *crtc) +static int _g4x_compute_pipe_wm(struct intel_crtc_state *crtc_state) { - struct intel_crtc_state *crtc_state = - intel_atomic_get_new_crtc_state(state, crtc); + struct intel_crtc *crtc = to_intel_crtc(crtc_state->uapi.crtc); struct g4x_wm_state *wm_state = &crtc_state->wm.g4x.optimal; u8 active_planes = crtc_state->active_planes & ~BIT(PLANE_CURSOR); const struct g4x_pipe_wm *raw; - const struct intel_plane_state *old_plane_state; - const struct intel_plane_state *new_plane_state; - struct intel_plane *plane; enum plane_id plane_id; - int i, level; - unsigned int dirty = 0; - - for_each_oldnew_intel_plane_in_state(state, plane, - old_plane_state, - new_plane_state, i) { - if (new_plane_state->hw.crtc != &crtc->base && - old_plane_state->hw.crtc != &crtc->base) - continue; - - if (g4x_raw_plane_wm_compute(crtc_state, new_plane_state)) - dirty |= BIT(plane->id); - } - - if (!dirty) - return 0; + int level; level = G4X_WM_LEVEL_NORMAL; if (!g4x_raw_crtc_wm_is_valid(crtc_state, level)) @@ -1417,6 +1384,34 @@ static int g4x_compute_pipe_wm(struct intel_atomic_state *state, return 0; } +static int g4x_compute_pipe_wm(struct intel_atomic_state *state, + struct intel_crtc *crtc) +{ + struct intel_crtc_state *crtc_state = + intel_atomic_get_new_crtc_state(state, crtc); + const struct intel_plane_state *old_plane_state; + const struct intel_plane_state *new_plane_state; + struct intel_plane *plane; + unsigned int dirty = 0; + int i; + + for_each_oldnew_intel_plane_in_state(state, plane, + old_plane_state, + new_plane_state, i) { + if (new_plane_state->hw.crtc != &crtc->base && + old_plane_state->hw.crtc != &crtc->base) + continue; + + if (g4x_raw_plane_wm_compute(crtc_state, new_plane_state)) + dirty |= BIT(plane->id); + } + + if (!dirty) + return 0; + + return _g4x_compute_pipe_wm(crtc_state); +} + static int g4x_compute_intermediate_wm(struct intel_atomic_state *state, struct intel_crtc *crtc) { @@ -1431,7 +1426,7 @@ static int g4x_compute_intermediate_wm(struct intel_atomic_state *state, enum plane_id plane_id; if (!new_crtc_state->hw.active || - drm_atomic_crtc_needs_modeset(&new_crtc_state->uapi)) { + intel_crtc_needs_modeset(new_crtc_state)) { *intermediate = *optimal; intermediate->cxsr = false; @@ -1857,64 +1852,17 @@ static bool vlv_raw_crtc_wm_is_valid(const struct intel_crtc_state *crtc_state, vlv_raw_plane_wm_is_valid(crtc_state, PLANE_CURSOR, level); } -static int vlv_compute_pipe_wm(struct intel_atomic_state *state, - struct intel_crtc *crtc) +static int _vlv_compute_pipe_wm(struct intel_crtc_state *crtc_state) { + struct intel_crtc *crtc = to_intel_crtc(crtc_state->uapi.crtc); struct drm_i915_private *dev_priv = to_i915(crtc->base.dev); - struct intel_crtc_state *crtc_state = - intel_atomic_get_new_crtc_state(state, crtc); struct vlv_wm_state *wm_state = &crtc_state->wm.vlv.optimal; const struct vlv_fifo_state *fifo_state = &crtc_state->wm.vlv.fifo_state; u8 active_planes = crtc_state->active_planes & ~BIT(PLANE_CURSOR); int num_active_planes = hweight8(active_planes); - bool needs_modeset = drm_atomic_crtc_needs_modeset(&crtc_state->uapi); - const struct intel_plane_state *old_plane_state; - const struct intel_plane_state *new_plane_state; - struct intel_plane *plane; enum plane_id plane_id; - int level, ret, i; - unsigned int dirty = 0; - - for_each_oldnew_intel_plane_in_state(state, plane, - old_plane_state, - new_plane_state, i) { - if (new_plane_state->hw.crtc != &crtc->base && - old_plane_state->hw.crtc != &crtc->base) - continue; - - if (vlv_raw_plane_wm_compute(crtc_state, new_plane_state)) - dirty |= BIT(plane->id); - } - - /* - * DSPARB registers may have been reset due to the - * power well being turned off. Make sure we restore - * them to a consistent state even if no primary/sprite - * planes are initially active. - */ - if (needs_modeset) - crtc_state->fifo_changed = true; - - if (!dirty) - return 0; - - /* cursor changes don't warrant a FIFO recompute */ - if (dirty & ~BIT(PLANE_CURSOR)) { - const struct intel_crtc_state *old_crtc_state = - intel_atomic_get_old_crtc_state(state, crtc); - const struct vlv_fifo_state *old_fifo_state = - &old_crtc_state->wm.vlv.fifo_state; - - ret = vlv_compute_fifo(crtc_state); - if (ret) - return ret; - - if (needs_modeset || - memcmp(old_fifo_state, fifo_state, - sizeof(*fifo_state)) != 0) - crtc_state->fifo_changed = true; - } + int level; /* initially allow all levels */ wm_state->num_levels = intel_wm_num_levels(dev_priv); @@ -1961,6 +1909,66 @@ static int vlv_compute_pipe_wm(struct intel_atomic_state *state, return 0; } +static int vlv_compute_pipe_wm(struct intel_atomic_state *state, + struct intel_crtc *crtc) +{ + struct intel_crtc_state *crtc_state = + intel_atomic_get_new_crtc_state(state, crtc); + const struct intel_plane_state *old_plane_state; + const struct intel_plane_state *new_plane_state; + struct intel_plane *plane; + unsigned int dirty = 0; + int i; + + for_each_oldnew_intel_plane_in_state(state, plane, + old_plane_state, + new_plane_state, i) { + if (new_plane_state->hw.crtc != &crtc->base && + old_plane_state->hw.crtc != &crtc->base) + continue; + + if (vlv_raw_plane_wm_compute(crtc_state, new_plane_state)) + dirty |= BIT(plane->id); + } + + /* + * DSPARB registers may have been reset due to the + * power well being turned off. Make sure we restore + * them to a consistent state even if no primary/sprite + * planes are initially active. We also force a FIFO + * recomputation so that we are sure to sanitize the + * FIFO setting we took over from the BIOS even if there + * are no active planes on the crtc. + */ + if (intel_crtc_needs_modeset(crtc_state)) + dirty = ~0; + + if (!dirty) + return 0; + + /* cursor changes don't warrant a FIFO recompute */ + if (dirty & ~BIT(PLANE_CURSOR)) { + const struct intel_crtc_state *old_crtc_state = + intel_atomic_get_old_crtc_state(state, crtc); + const struct vlv_fifo_state *old_fifo_state = + &old_crtc_state->wm.vlv.fifo_state; + const struct vlv_fifo_state *new_fifo_state = + &crtc_state->wm.vlv.fifo_state; + int ret; + + ret = vlv_compute_fifo(crtc_state); + if (ret) + return ret; + + if (intel_crtc_needs_modeset(crtc_state) || + memcmp(old_fifo_state, new_fifo_state, + sizeof(*new_fifo_state)) != 0) + crtc_state->fifo_changed = true; + } + + return _vlv_compute_pipe_wm(crtc_state); +} + #define VLV_FIFO(plane, value) \ (((value) << DSPARB_ ## plane ## _SHIFT_VLV) & DSPARB_ ## plane ## _MASK_VLV) @@ -2075,7 +2083,7 @@ static int vlv_compute_intermediate_wm(struct intel_atomic_state *state, int level; if (!new_crtc_state->hw.active || - drm_atomic_crtc_needs_modeset(&new_crtc_state->uapi)) { + intel_crtc_needs_modeset(new_crtc_state)) { *intermediate = *optimal; intermediate->cxsr = false; @@ -3133,7 +3141,7 @@ static int ilk_compute_intermediate_wm(struct intel_atomic_state *state, */ *a = new_crtc_state->wm.ilk.optimal; if (!new_crtc_state->hw.active || - drm_atomic_crtc_needs_modeset(&new_crtc_state->uapi) || + intel_crtc_needs_modeset(new_crtc_state) || state->skip_intermediate_wm) return 0; @@ -3458,7 +3466,6 @@ static void ilk_write_wm_values(struct drm_i915_private *dev_priv, { struct ilk_wm_values *previous = &dev_priv->display.wm.hw; unsigned int dirty; - u32 val; dirty = ilk_compute_wm_dirty(dev_priv, previous, results); if (!dirty) @@ -3474,32 +3481,20 @@ static void ilk_write_wm_values(struct drm_i915_private *dev_priv, intel_uncore_write(&dev_priv->uncore, WM0_PIPE_ILK(PIPE_C), results->wm_pipe[2]); if (dirty & WM_DIRTY_DDB) { - if (IS_HASWELL(dev_priv) || IS_BROADWELL(dev_priv)) { - val = intel_uncore_read(&dev_priv->uncore, WM_MISC); - if (results->partitioning == INTEL_DDB_PART_1_2) - val &= ~WM_MISC_DATA_PARTITION_5_6; - else - val |= WM_MISC_DATA_PARTITION_5_6; - intel_uncore_write(&dev_priv->uncore, WM_MISC, val); - } else { - val = intel_uncore_read(&dev_priv->uncore, DISP_ARB_CTL2); - if (results->partitioning == INTEL_DDB_PART_1_2) - val &= ~DISP_DATA_PARTITION_5_6; - else - val |= DISP_DATA_PARTITION_5_6; - intel_uncore_write(&dev_priv->uncore, DISP_ARB_CTL2, val); - } - } - - if (dirty & WM_DIRTY_FBC) { - val = intel_uncore_read(&dev_priv->uncore, DISP_ARB_CTL); - if (results->enable_fbc_wm) - val &= ~DISP_FBC_WM_DIS; + if (IS_HASWELL(dev_priv) || IS_BROADWELL(dev_priv)) + intel_uncore_rmw(&dev_priv->uncore, WM_MISC, WM_MISC_DATA_PARTITION_5_6, + results->partitioning == INTEL_DDB_PART_1_2 ? 0 : + WM_MISC_DATA_PARTITION_5_6); else - val |= DISP_FBC_WM_DIS; - intel_uncore_write(&dev_priv->uncore, DISP_ARB_CTL, val); + intel_uncore_rmw(&dev_priv->uncore, DISP_ARB_CTL2, DISP_DATA_PARTITION_5_6, + results->partitioning == INTEL_DDB_PART_1_2 ? 0 : + DISP_DATA_PARTITION_5_6); } + if (dirty & WM_DIRTY_FBC) + intel_uncore_rmw(&dev_priv->uncore, DISP_ARB_CTL, DISP_FBC_WM_DIS, + results->enable_fbc_wm ? 0 : DISP_FBC_WM_DIS); + if (dirty & WM_DIRTY_LP(1) && previous->wm_lp_spr[0] != results->wm_lp_spr[0]) intel_uncore_write(&dev_priv->uncore, WM1S_LP_ILK, results->wm_lp_spr[0]); @@ -3824,6 +3819,8 @@ void g4x_wm_get_hw_state(struct drm_i915_private *dev_priv) plane_id, USHRT_MAX); g4x_raw_fbc_wm_set(crtc_state, level, USHRT_MAX); + g4x_invalidate_wms(crtc, active, level); + crtc_state->wm.g4x.optimal = *active; crtc_state->wm.g4x.intermediate = *active; @@ -3860,37 +3857,30 @@ void g4x_wm_sanitize(struct drm_i915_private *dev_priv) to_intel_crtc_state(crtc->base.state); struct intel_plane_state *plane_state = to_intel_plane_state(plane->base.state); - struct g4x_wm_state *wm_state = &crtc_state->wm.g4x.optimal; enum plane_id plane_id = plane->id; - int level; + int level, num_levels = intel_wm_num_levels(dev_priv); if (plane_state->uapi.visible) continue; - for (level = 0; level < 3; level++) { + for (level = 0; level < num_levels; level++) { struct g4x_pipe_wm *raw = &crtc_state->wm.g4x.raw[level]; raw->plane[plane_id] = 0; - wm_state->wm.plane[plane_id] = 0; - } - if (plane_id == PLANE_PRIMARY) { - for (level = 0; level < 3; level++) { - struct g4x_pipe_wm *raw = - &crtc_state->wm.g4x.raw[level]; + if (plane_id == PLANE_PRIMARY) raw->fbc = 0; - } - - wm_state->sr.fbc = 0; - wm_state->hpll.fbc = 0; - wm_state->fbc_en = false; } } for_each_intel_crtc(&dev_priv->drm, crtc) { struct intel_crtc_state *crtc_state = to_intel_crtc_state(crtc->base.state); + int ret; + + ret = _g4x_compute_pipe_wm(crtc_state); + drm_WARN_ON(&dev_priv->drm, ret); crtc_state->wm.g4x.intermediate = crtc_state->wm.g4x.optimal; @@ -4016,30 +4006,27 @@ void vlv_wm_sanitize(struct drm_i915_private *dev_priv) to_intel_crtc_state(crtc->base.state); struct intel_plane_state *plane_state = to_intel_plane_state(plane->base.state); - struct vlv_wm_state *wm_state = &crtc_state->wm.vlv.optimal; - const struct vlv_fifo_state *fifo_state = - &crtc_state->wm.vlv.fifo_state; enum plane_id plane_id = plane->id; - int level; + int level, num_levels = intel_wm_num_levels(dev_priv); if (plane_state->uapi.visible) continue; - for (level = 0; level < wm_state->num_levels; level++) { + for (level = 0; level < num_levels; level++) { struct g4x_pipe_wm *raw = &crtc_state->wm.vlv.raw[level]; raw->plane[plane_id] = 0; - - wm_state->wm[level].plane[plane_id] = - vlv_invert_wm_value(raw->plane[plane_id], - fifo_state->plane[plane_id]); } } for_each_intel_crtc(&dev_priv->drm, crtc) { struct intel_crtc_state *crtc_state = to_intel_crtc_state(crtc->base.state); + int ret; + + ret = _vlv_compute_pipe_wm(crtc_state); + drm_WARN_ON(&dev_priv->drm, ret); crtc_state->wm.vlv.intermediate = crtc_state->wm.vlv.optimal; @@ -4057,9 +4044,9 @@ void vlv_wm_sanitize(struct drm_i915_private *dev_priv) */ static void ilk_init_lp_watermarks(struct drm_i915_private *dev_priv) { - intel_uncore_write(&dev_priv->uncore, WM3_LP_ILK, intel_uncore_read(&dev_priv->uncore, WM3_LP_ILK) & ~WM_LP_ENABLE); - intel_uncore_write(&dev_priv->uncore, WM2_LP_ILK, intel_uncore_read(&dev_priv->uncore, WM2_LP_ILK) & ~WM_LP_ENABLE); - intel_uncore_write(&dev_priv->uncore, WM1_LP_ILK, intel_uncore_read(&dev_priv->uncore, WM1_LP_ILK) & ~WM_LP_ENABLE); + intel_uncore_rmw(&dev_priv->uncore, WM3_LP_ILK, WM_LP_ENABLE, 0); + intel_uncore_rmw(&dev_priv->uncore, WM2_LP_ILK, WM_LP_ENABLE, 0); + intel_uncore_rmw(&dev_priv->uncore, WM1_LP_ILK, WM_LP_ENABLE, 0); /* * Don't touch WM_LP_SPRITE_ENABLE here. @@ -4113,11 +4100,9 @@ static void g4x_disable_trickle_feed(struct drm_i915_private *dev_priv) enum pipe pipe; for_each_pipe(dev_priv, pipe) { - intel_uncore_write(&dev_priv->uncore, DSPCNTR(pipe), - intel_uncore_read(&dev_priv->uncore, DSPCNTR(pipe)) | - DISP_TRICKLE_FEED_DISABLE); + intel_uncore_rmw(&dev_priv->uncore, DSPCNTR(pipe), 0, DISP_TRICKLE_FEED_DISABLE); - intel_uncore_write(&dev_priv->uncore, DSPSURF(pipe), intel_uncore_read(&dev_priv->uncore, DSPSURF(pipe))); + intel_uncore_rmw(&dev_priv->uncore, DSPSURF(pipe), 0, 0); intel_uncore_posting_read(&dev_priv->uncore, DSPSURF(pipe)); } } @@ -4164,19 +4149,13 @@ static void ilk_init_clock_gating(struct drm_i915_private *dev_priv) */ if (IS_IRONLAKE_M(dev_priv)) { /* WaFbcAsynchFlipDisableFbcQueue:ilk */ - intel_uncore_write(&dev_priv->uncore, ILK_DISPLAY_CHICKEN1, - intel_uncore_read(&dev_priv->uncore, ILK_DISPLAY_CHICKEN1) | - ILK_FBCQ_DIS); - intel_uncore_write(&dev_priv->uncore, ILK_DISPLAY_CHICKEN2, - intel_uncore_read(&dev_priv->uncore, ILK_DISPLAY_CHICKEN2) | - ILK_DPARB_GATE); + intel_uncore_rmw(&dev_priv->uncore, ILK_DISPLAY_CHICKEN1, 0, ILK_FBCQ_DIS); + intel_uncore_rmw(&dev_priv->uncore, ILK_DISPLAY_CHICKEN2, 0, ILK_DPARB_GATE); } intel_uncore_write(&dev_priv->uncore, ILK_DSPCLK_GATE_D, dspclk_gate); - intel_uncore_write(&dev_priv->uncore, ILK_DISPLAY_CHICKEN2, - intel_uncore_read(&dev_priv->uncore, ILK_DISPLAY_CHICKEN2) | - ILK_ELPIN_409_SELECT); + intel_uncore_rmw(&dev_priv->uncore, ILK_DISPLAY_CHICKEN2, 0, ILK_ELPIN_409_SELECT); g4x_disable_trickle_feed(dev_priv); @@ -4196,8 +4175,7 @@ static void cpt_init_clock_gating(struct drm_i915_private *dev_priv) intel_uncore_write(&dev_priv->uncore, SOUTH_DSPCLK_GATE_D, PCH_DPLSUNIT_CLOCK_GATE_DISABLE | PCH_DPLUNIT_CLOCK_GATE_DISABLE | PCH_CPUNIT_CLOCK_GATE_DISABLE); - intel_uncore_write(&dev_priv->uncore, SOUTH_CHICKEN2, intel_uncore_read(&dev_priv->uncore, SOUTH_CHICKEN2) | - DPLS_EDP_PPS_FIX_DIS); + intel_uncore_rmw(&dev_priv->uncore, SOUTH_CHICKEN2, 0, DPLS_EDP_PPS_FIX_DIS); /* The below fixes the weird display corruption, a few pixels shifted * downward, on (only) LVDS of some HP laptops with IVY. */ @@ -4235,9 +4213,7 @@ static void gen6_init_clock_gating(struct drm_i915_private *dev_priv) intel_uncore_write(&dev_priv->uncore, ILK_DSPCLK_GATE_D, dspclk_gate); - intel_uncore_write(&dev_priv->uncore, ILK_DISPLAY_CHICKEN2, - intel_uncore_read(&dev_priv->uncore, ILK_DISPLAY_CHICKEN2) | - ILK_ELPIN_409_SELECT); + intel_uncore_rmw(&dev_priv->uncore, ILK_DISPLAY_CHICKEN2, 0, ILK_ELPIN_409_SELECT); intel_uncore_write(&dev_priv->uncore, GEN6_UCGCTL1, intel_uncore_read(&dev_priv->uncore, GEN6_UCGCTL1) | @@ -4297,14 +4273,12 @@ static void lpt_init_clock_gating(struct drm_i915_private *dev_priv) * disabled when not needed anymore in order to save power. */ if (HAS_PCH_LPT_LP(dev_priv)) - intel_uncore_write(&dev_priv->uncore, SOUTH_DSPCLK_GATE_D, - intel_uncore_read(&dev_priv->uncore, SOUTH_DSPCLK_GATE_D) | - PCH_LP_PARTITION_LEVEL_DISABLE); + intel_uncore_rmw(&dev_priv->uncore, SOUTH_DSPCLK_GATE_D, + 0, PCH_LP_PARTITION_LEVEL_DISABLE); /* WADPOClockGatingDisable:hsw */ - intel_uncore_write(&dev_priv->uncore, TRANS_CHICKEN1(PIPE_A), - intel_uncore_read(&dev_priv->uncore, TRANS_CHICKEN1(PIPE_A)) | - TRANS_CHICKEN1_DP0UNIT_GC_DISABLE); + intel_uncore_rmw(&dev_priv->uncore, TRANS_CHICKEN1(PIPE_A), + 0, TRANS_CHICKEN1_DP0UNIT_GC_DISABLE); } static void lpt_suspend_hw(struct drm_i915_private *dev_priv) @@ -4325,22 +4299,22 @@ static void gen8_set_l3sqc_credits(struct drm_i915_private *dev_priv, u32 val; /* WaTempDisableDOPClkGating:bdw */ - misccpctl = intel_uncore_read(&dev_priv->uncore, GEN7_MISCCPCTL); - intel_uncore_write(&dev_priv->uncore, GEN7_MISCCPCTL, misccpctl & ~GEN7_DOP_CLOCK_GATE_ENABLE); + misccpctl = intel_gt_mcr_multicast_rmw(to_gt(dev_priv), GEN8_MISCCPCTL, + GEN8_DOP_CLOCK_GATE_ENABLE, 0); - val = intel_uncore_read(&dev_priv->uncore, GEN8_L3SQCREG1); + val = intel_gt_mcr_read_any(to_gt(dev_priv), GEN8_L3SQCREG1); val &= ~L3_PRIO_CREDITS_MASK; val |= L3_GENERAL_PRIO_CREDITS(general_prio_credits); val |= L3_HIGH_PRIO_CREDITS(high_prio_credits); - intel_uncore_write(&dev_priv->uncore, GEN8_L3SQCREG1, val); + intel_gt_mcr_multicast_write(to_gt(dev_priv), GEN8_L3SQCREG1, val); /* * Wait at least 100 clocks before re-enabling clock gating. * See the definition of L3SQCREG1 in BSpec. */ - intel_uncore_posting_read(&dev_priv->uncore, GEN8_L3SQCREG1); + intel_gt_mcr_read_any(to_gt(dev_priv), GEN8_L3SQCREG1); udelay(1); - intel_uncore_write(&dev_priv->uncore, GEN7_MISCCPCTL, misccpctl); + intel_gt_mcr_multicast_write(to_gt(dev_priv), GEN8_MISCCPCTL, misccpctl); } static void icl_init_clock_gating(struct drm_i915_private *dev_priv) @@ -4363,8 +4337,7 @@ static void gen12lp_init_clock_gating(struct drm_i915_private *dev_priv) /* Wa_1409825376:tgl (pre-prod)*/ if (IS_TGL_DISPLAY_STEP(dev_priv, STEP_A0, STEP_C0)) - intel_uncore_write(&dev_priv->uncore, GEN9_CLKGATE_DIS_3, intel_uncore_read(&dev_priv->uncore, GEN9_CLKGATE_DIS_3) | - TGL_VRH_GATING_DIS); + intel_uncore_rmw(&dev_priv->uncore, GEN9_CLKGATE_DIS_3, 0, TGL_VRH_GATING_DIS); /* Wa_14013723622:tgl,rkl,dg1,adl-s */ if (DISPLAY_VER(dev_priv) == 12) @@ -4389,8 +4362,7 @@ static void dg1_init_clock_gating(struct drm_i915_private *dev_priv) /* Wa_1409836686:dg1[a0] */ if (IS_DG1_GRAPHICS_STEP(dev_priv, STEP_A0, STEP_B0)) - intel_uncore_write(&dev_priv->uncore, GEN9_CLKGATE_DIS_3, intel_uncore_read(&dev_priv->uncore, GEN9_CLKGATE_DIS_3) | - DPT_GATING_DIS); + intel_uncore_rmw(&dev_priv->uncore, GEN9_CLKGATE_DIS_3, 0, DPT_GATING_DIS); } static void xehpsdv_init_clock_gating(struct drm_i915_private *dev_priv) @@ -4432,8 +4404,7 @@ static void cnp_init_clock_gating(struct drm_i915_private *dev_priv) return; /* Display WA #1181 WaSouthDisplayDisablePWMCGEGating: cnp */ - intel_uncore_write(&dev_priv->uncore, SOUTH_DSPCLK_GATE_D, intel_uncore_read(&dev_priv->uncore, SOUTH_DSPCLK_GATE_D) | - CNP_PWM_CGE_GATING_DISABLE); + intel_uncore_rmw(&dev_priv->uncore, SOUTH_DSPCLK_GATE_D, 0, CNP_PWM_CGE_GATING_DISABLE); } static void cfl_init_clock_gating(struct drm_i915_private *dev_priv) @@ -4442,23 +4413,20 @@ static void cfl_init_clock_gating(struct drm_i915_private *dev_priv) gen9_init_clock_gating(dev_priv); /* WAC6entrylatency:cfl */ - intel_uncore_write(&dev_priv->uncore, FBC_LLC_READ_CTRL, intel_uncore_read(&dev_priv->uncore, FBC_LLC_READ_CTRL) | - FBC_LLC_FULLY_OPEN); + intel_uncore_rmw(&dev_priv->uncore, FBC_LLC_READ_CTRL, 0, FBC_LLC_FULLY_OPEN); /* * WaFbcTurnOffFbcWatermark:cfl * Display WA #0562: cfl */ - intel_uncore_write(&dev_priv->uncore, DISP_ARB_CTL, intel_uncore_read(&dev_priv->uncore, DISP_ARB_CTL) | - DISP_FBC_WM_DIS); + intel_uncore_rmw(&dev_priv->uncore, DISP_ARB_CTL, 0, DISP_FBC_WM_DIS); /* * WaFbcNukeOnHostModify:cfl * Display WA #0873: cfl */ - intel_uncore_write(&dev_priv->uncore, ILK_DPFC_CHICKEN(INTEL_FBC_A), - intel_uncore_read(&dev_priv->uncore, ILK_DPFC_CHICKEN(INTEL_FBC_A)) | - DPFC_NUKE_ON_ANY_MODIFICATION); + intel_uncore_rmw(&dev_priv->uncore, ILK_DPFC_CHICKEN(INTEL_FBC_A), + 0, DPFC_NUKE_ON_ANY_MODIFICATION); } static void kbl_init_clock_gating(struct drm_i915_private *dev_priv) @@ -4466,33 +4434,30 @@ static void kbl_init_clock_gating(struct drm_i915_private *dev_priv) gen9_init_clock_gating(dev_priv); /* WAC6entrylatency:kbl */ - intel_uncore_write(&dev_priv->uncore, FBC_LLC_READ_CTRL, intel_uncore_read(&dev_priv->uncore, FBC_LLC_READ_CTRL) | - FBC_LLC_FULLY_OPEN); + intel_uncore_rmw(&dev_priv->uncore, FBC_LLC_READ_CTRL, 0, FBC_LLC_FULLY_OPEN); /* WaDisableSDEUnitClockGating:kbl */ if (IS_KBL_GRAPHICS_STEP(dev_priv, 0, STEP_C0)) - intel_uncore_write(&dev_priv->uncore, GEN8_UCGCTL6, intel_uncore_read(&dev_priv->uncore, GEN8_UCGCTL6) | - GEN8_SDEUNIT_CLOCK_GATE_DISABLE); + intel_uncore_rmw(&dev_priv->uncore, GEN8_UCGCTL6, + 0, GEN8_SDEUNIT_CLOCK_GATE_DISABLE); /* WaDisableGamClockGating:kbl */ if (IS_KBL_GRAPHICS_STEP(dev_priv, 0, STEP_C0)) - intel_uncore_write(&dev_priv->uncore, GEN6_UCGCTL1, intel_uncore_read(&dev_priv->uncore, GEN6_UCGCTL1) | - GEN6_GAMUNIT_CLOCK_GATE_DISABLE); + intel_uncore_rmw(&dev_priv->uncore, GEN6_UCGCTL1, + 0, GEN6_GAMUNIT_CLOCK_GATE_DISABLE); /* * WaFbcTurnOffFbcWatermark:kbl * Display WA #0562: kbl */ - intel_uncore_write(&dev_priv->uncore, DISP_ARB_CTL, intel_uncore_read(&dev_priv->uncore, DISP_ARB_CTL) | - DISP_FBC_WM_DIS); + intel_uncore_rmw(&dev_priv->uncore, DISP_ARB_CTL, 0, DISP_FBC_WM_DIS); /* * WaFbcNukeOnHostModify:kbl * Display WA #0873: kbl */ - intel_uncore_write(&dev_priv->uncore, ILK_DPFC_CHICKEN(INTEL_FBC_A), - intel_uncore_read(&dev_priv->uncore, ILK_DPFC_CHICKEN(INTEL_FBC_A)) | - DPFC_NUKE_ON_ANY_MODIFICATION); + intel_uncore_rmw(&dev_priv->uncore, ILK_DPFC_CHICKEN(INTEL_FBC_A), + 0, DPFC_NUKE_ON_ANY_MODIFICATION); } static void skl_init_clock_gating(struct drm_i915_private *dev_priv) @@ -4500,35 +4465,30 @@ static void skl_init_clock_gating(struct drm_i915_private *dev_priv) gen9_init_clock_gating(dev_priv); /* WaDisableDopClockGating:skl */ - intel_uncore_write(&dev_priv->uncore, GEN7_MISCCPCTL, intel_uncore_read(&dev_priv->uncore, GEN7_MISCCPCTL) & - ~GEN7_DOP_CLOCK_GATE_ENABLE); + intel_gt_mcr_multicast_rmw(to_gt(dev_priv), GEN8_MISCCPCTL, + GEN8_DOP_CLOCK_GATE_ENABLE, 0); /* WAC6entrylatency:skl */ - intel_uncore_write(&dev_priv->uncore, FBC_LLC_READ_CTRL, intel_uncore_read(&dev_priv->uncore, FBC_LLC_READ_CTRL) | - FBC_LLC_FULLY_OPEN); + intel_uncore_rmw(&dev_priv->uncore, FBC_LLC_READ_CTRL, 0, FBC_LLC_FULLY_OPEN); /* * WaFbcTurnOffFbcWatermark:skl * Display WA #0562: skl */ - intel_uncore_write(&dev_priv->uncore, DISP_ARB_CTL, intel_uncore_read(&dev_priv->uncore, DISP_ARB_CTL) | - DISP_FBC_WM_DIS); + intel_uncore_rmw(&dev_priv->uncore, DISP_ARB_CTL, 0, DISP_FBC_WM_DIS); /* * WaFbcNukeOnHostModify:skl * Display WA #0873: skl */ - intel_uncore_write(&dev_priv->uncore, ILK_DPFC_CHICKEN(INTEL_FBC_A), - intel_uncore_read(&dev_priv->uncore, ILK_DPFC_CHICKEN(INTEL_FBC_A)) | - DPFC_NUKE_ON_ANY_MODIFICATION); + intel_uncore_rmw(&dev_priv->uncore, ILK_DPFC_CHICKEN(INTEL_FBC_A), + 0, DPFC_NUKE_ON_ANY_MODIFICATION); /* * WaFbcHighMemBwCorruptionAvoidance:skl * Display WA #0883: skl */ - intel_uncore_write(&dev_priv->uncore, ILK_DPFC_CHICKEN(INTEL_FBC_A), - intel_uncore_read(&dev_priv->uncore, ILK_DPFC_CHICKEN(INTEL_FBC_A)) | - DPFC_DISABLE_DUMMY0); + intel_uncore_rmw(&dev_priv->uncore, ILK_DPFC_CHICKEN(INTEL_FBC_A), 0, DPFC_DISABLE_DUMMY0); } static void bdw_init_clock_gating(struct drm_i915_private *dev_priv) @@ -4536,43 +4496,37 @@ static void bdw_init_clock_gating(struct drm_i915_private *dev_priv) enum pipe pipe; /* WaFbcAsynchFlipDisableFbcQueue:hsw,bdw */ - intel_uncore_write(&dev_priv->uncore, CHICKEN_PIPESL_1(PIPE_A), - intel_uncore_read(&dev_priv->uncore, CHICKEN_PIPESL_1(PIPE_A)) | - HSW_FBCQ_DIS); + intel_uncore_rmw(&dev_priv->uncore, CHICKEN_PIPESL_1(PIPE_A), 0, HSW_FBCQ_DIS); /* WaSwitchSolVfFArbitrationPriority:bdw */ - intel_uncore_write(&dev_priv->uncore, GAM_ECOCHK, intel_uncore_read(&dev_priv->uncore, GAM_ECOCHK) | HSW_ECOCHK_ARB_PRIO_SOL); + intel_uncore_rmw(&dev_priv->uncore, GAM_ECOCHK, 0, HSW_ECOCHK_ARB_PRIO_SOL); /* WaPsrDPAMaskVBlankInSRD:bdw */ - intel_uncore_write(&dev_priv->uncore, CHICKEN_PAR1_1, - intel_uncore_read(&dev_priv->uncore, CHICKEN_PAR1_1) | DPA_MASK_VBLANK_SRD); + intel_uncore_rmw(&dev_priv->uncore, CHICKEN_PAR1_1, 0, DPA_MASK_VBLANK_SRD); for_each_pipe(dev_priv, pipe) { /* WaPsrDPRSUnmaskVBlankInSRD:bdw */ - intel_uncore_write(&dev_priv->uncore, CHICKEN_PIPESL_1(pipe), - intel_uncore_read(&dev_priv->uncore, CHICKEN_PIPESL_1(pipe)) | - BDW_DPRS_MASK_VBLANK_SRD); + intel_uncore_rmw(&dev_priv->uncore, CHICKEN_PIPESL_1(pipe), + 0, BDW_DPRS_MASK_VBLANK_SRD); } /* WaVSRefCountFullforceMissDisable:bdw */ /* WaDSRefCountFullforceMissDisable:bdw */ - intel_uncore_write(&dev_priv->uncore, GEN7_FF_THREAD_MODE, - intel_uncore_read(&dev_priv->uncore, GEN7_FF_THREAD_MODE) & - ~(GEN8_FF_DS_REF_CNT_FFME | GEN7_FF_VS_REF_CNT_FFME)); + intel_uncore_rmw(&dev_priv->uncore, GEN7_FF_THREAD_MODE, + GEN8_FF_DS_REF_CNT_FFME | GEN7_FF_VS_REF_CNT_FFME, 0); intel_uncore_write(&dev_priv->uncore, RING_PSMI_CTL(RENDER_RING_BASE), _MASKED_BIT_ENABLE(GEN8_RC_SEMA_IDLE_MSG_DISABLE)); /* WaDisableSDEUnitClockGating:bdw */ - intel_uncore_write(&dev_priv->uncore, GEN8_UCGCTL6, intel_uncore_read(&dev_priv->uncore, GEN8_UCGCTL6) | - GEN8_SDEUNIT_CLOCK_GATE_DISABLE); + intel_uncore_rmw(&dev_priv->uncore, GEN8_UCGCTL6, 0, GEN8_SDEUNIT_CLOCK_GATE_DISABLE); /* WaProgramL3SqcReg1Default:bdw */ gen8_set_l3sqc_credits(dev_priv, 30, 2); /* WaKVMNotificationOnConfigChange:bdw */ - intel_uncore_write(&dev_priv->uncore, CHICKEN_PAR2_1, intel_uncore_read(&dev_priv->uncore, CHICKEN_PAR2_1) - | KVM_CONFIG_CHANGE_NOTIFICATION_SELECT); + intel_uncore_rmw(&dev_priv->uncore, CHICKEN_PAR2_1, + 0, KVM_CONFIG_CHANGE_NOTIFICATION_SELECT); lpt_init_clock_gating(dev_priv); @@ -4581,38 +4535,30 @@ static void bdw_init_clock_gating(struct drm_i915_private *dev_priv) * Also see the CHICKEN2 write in bdw_init_workarounds() to disable DOP * clock gating. */ - intel_uncore_write(&dev_priv->uncore, GEN6_UCGCTL1, - intel_uncore_read(&dev_priv->uncore, GEN6_UCGCTL1) | GEN6_EU_TCUNIT_CLOCK_GATE_DISABLE); + intel_uncore_rmw(&dev_priv->uncore, GEN6_UCGCTL1, 0, GEN6_EU_TCUNIT_CLOCK_GATE_DISABLE); } static void hsw_init_clock_gating(struct drm_i915_private *dev_priv) { /* WaFbcAsynchFlipDisableFbcQueue:hsw,bdw */ - intel_uncore_write(&dev_priv->uncore, CHICKEN_PIPESL_1(PIPE_A), - intel_uncore_read(&dev_priv->uncore, CHICKEN_PIPESL_1(PIPE_A)) | - HSW_FBCQ_DIS); + intel_uncore_rmw(&dev_priv->uncore, CHICKEN_PIPESL_1(PIPE_A), 0, HSW_FBCQ_DIS); /* This is required by WaCatErrorRejectionIssue:hsw */ - intel_uncore_write(&dev_priv->uncore, GEN7_SQ_CHICKEN_MBCUNIT_CONFIG, - intel_uncore_read(&dev_priv->uncore, GEN7_SQ_CHICKEN_MBCUNIT_CONFIG) | - GEN7_SQ_CHICKEN_MBCUNIT_SQINTMOB); + intel_uncore_rmw(&dev_priv->uncore, GEN7_SQ_CHICKEN_MBCUNIT_CONFIG, + 0, GEN7_SQ_CHICKEN_MBCUNIT_SQINTMOB); /* WaSwitchSolVfFArbitrationPriority:hsw */ - intel_uncore_write(&dev_priv->uncore, GAM_ECOCHK, intel_uncore_read(&dev_priv->uncore, GAM_ECOCHK) | HSW_ECOCHK_ARB_PRIO_SOL); + intel_uncore_rmw(&dev_priv->uncore, GAM_ECOCHK, 0, HSW_ECOCHK_ARB_PRIO_SOL); lpt_init_clock_gating(dev_priv); } static void ivb_init_clock_gating(struct drm_i915_private *dev_priv) { - u32 snpcr; - intel_uncore_write(&dev_priv->uncore, ILK_DSPCLK_GATE_D, ILK_VRHUNIT_CLOCK_GATE_DISABLE); /* WaFbcAsynchFlipDisableFbcQueue:ivb */ - intel_uncore_write(&dev_priv->uncore, ILK_DISPLAY_CHICKEN1, - intel_uncore_read(&dev_priv->uncore, ILK_DISPLAY_CHICKEN1) | - ILK_FBCQ_DIS); + intel_uncore_rmw(&dev_priv->uncore, ILK_DISPLAY_CHICKEN1, 0, ILK_FBCQ_DIS); /* WaDisableBackToBackFlipFix:ivb */ intel_uncore_write(&dev_priv->uncore, IVB_CHICKEN3, @@ -4638,16 +4584,13 @@ static void ivb_init_clock_gating(struct drm_i915_private *dev_priv) GEN6_RCZUNIT_CLOCK_GATE_DISABLE); /* This is required by WaCatErrorRejectionIssue:ivb */ - intel_uncore_write(&dev_priv->uncore, GEN7_SQ_CHICKEN_MBCUNIT_CONFIG, - intel_uncore_read(&dev_priv->uncore, GEN7_SQ_CHICKEN_MBCUNIT_CONFIG) | - GEN7_SQ_CHICKEN_MBCUNIT_SQINTMOB); + intel_uncore_rmw(&dev_priv->uncore, GEN7_SQ_CHICKEN_MBCUNIT_CONFIG, + 0, GEN7_SQ_CHICKEN_MBCUNIT_SQINTMOB); g4x_disable_trickle_feed(dev_priv); - snpcr = intel_uncore_read(&dev_priv->uncore, GEN6_MBCUNIT_SNPCR); - snpcr &= ~GEN6_MBC_SNPCR_MASK; - snpcr |= GEN6_MBC_SNPCR_MED; - intel_uncore_write(&dev_priv->uncore, GEN6_MBCUNIT_SNPCR, snpcr); + intel_uncore_rmw(&dev_priv->uncore, GEN6_MBCUNIT_SNPCR, GEN6_MBC_SNPCR_MASK, + GEN6_MBC_SNPCR_MED); if (!HAS_PCH_NOP(dev_priv)) cpt_init_clock_gating(dev_priv); @@ -4667,9 +4610,8 @@ static void vlv_init_clock_gating(struct drm_i915_private *dev_priv) _MASKED_BIT_ENABLE(DOP_CLOCK_GATING_DISABLE)); /* This is required by WaCatErrorRejectionIssue:vlv */ - intel_uncore_write(&dev_priv->uncore, GEN7_SQ_CHICKEN_MBCUNIT_CONFIG, - intel_uncore_read(&dev_priv->uncore, GEN7_SQ_CHICKEN_MBCUNIT_CONFIG) | - GEN7_SQ_CHICKEN_MBCUNIT_SQINTMOB); + intel_uncore_rmw(&dev_priv->uncore, GEN7_SQ_CHICKEN_MBCUNIT_CONFIG, + 0, GEN7_SQ_CHICKEN_MBCUNIT_SQINTMOB); /* * According to the spec, bit 13 (RCZUNIT) must be set on IVB. @@ -4681,8 +4623,7 @@ static void vlv_init_clock_gating(struct drm_i915_private *dev_priv) /* WaDisableL3Bank2xClockGate:vlv * Disabling L3 clock gating- MMIO 940c[25] = 1 * Set bit 25, to disable L3_BANK_2x_CLK_GATING */ - intel_uncore_write(&dev_priv->uncore, GEN7_UCGCTL4, - intel_uncore_read(&dev_priv->uncore, GEN7_UCGCTL4) | GEN7_L3BANK2X_CLOCK_GATE_DISABLE); + intel_uncore_rmw(&dev_priv->uncore, GEN7_UCGCTL4, 0, GEN7_L3BANK2X_CLOCK_GATE_DISABLE); /* * WaDisableVLVClockGating_VBIIssue:vlv @@ -4696,21 +4637,18 @@ static void chv_init_clock_gating(struct drm_i915_private *dev_priv) { /* WaVSRefCountFullforceMissDisable:chv */ /* WaDSRefCountFullforceMissDisable:chv */ - intel_uncore_write(&dev_priv->uncore, GEN7_FF_THREAD_MODE, - intel_uncore_read(&dev_priv->uncore, GEN7_FF_THREAD_MODE) & - ~(GEN8_FF_DS_REF_CNT_FFME | GEN7_FF_VS_REF_CNT_FFME)); + intel_uncore_rmw(&dev_priv->uncore, GEN7_FF_THREAD_MODE, + GEN8_FF_DS_REF_CNT_FFME | GEN7_FF_VS_REF_CNT_FFME, 0); /* WaDisableSemaphoreAndSyncFlipWait:chv */ intel_uncore_write(&dev_priv->uncore, RING_PSMI_CTL(RENDER_RING_BASE), _MASKED_BIT_ENABLE(GEN8_RC_SEMA_IDLE_MSG_DISABLE)); /* WaDisableCSUnitClockGating:chv */ - intel_uncore_write(&dev_priv->uncore, GEN6_UCGCTL1, intel_uncore_read(&dev_priv->uncore, GEN6_UCGCTL1) | - GEN6_CSUNIT_CLOCK_GATE_DISABLE); + intel_uncore_rmw(&dev_priv->uncore, GEN6_UCGCTL1, 0, GEN6_CSUNIT_CLOCK_GATE_DISABLE); /* WaDisableSDEUnitClockGating:chv */ - intel_uncore_write(&dev_priv->uncore, GEN8_UCGCTL6, intel_uncore_read(&dev_priv->uncore, GEN8_UCGCTL6) | - GEN8_SDEUNIT_CLOCK_GATE_DISABLE); + intel_uncore_rmw(&dev_priv->uncore, GEN8_UCGCTL6, 0, GEN8_SDEUNIT_CLOCK_GATE_DISABLE); /* * WaProgramL3SqcReg1Default:chv diff --git a/drivers/gpu/drm/i915/intel_region_ttm.c b/drivers/gpu/drm/i915/intel_region_ttm.c index 575d67bc6ffe..cf89d0c2a2d9 100644 --- a/drivers/gpu/drm/i915/intel_region_ttm.c +++ b/drivers/gpu/drm/i915/intel_region_ttm.c @@ -244,7 +244,7 @@ void intel_region_ttm_resource_free(struct intel_memory_region *mem, struct ttm_resource_manager *man = mem->region_private; struct ttm_buffer_object mock_bo = {}; - mock_bo.base.size = res->num_pages << PAGE_SHIFT; + mock_bo.base.size = res->size; mock_bo.bdev = &mem->i915->bdev; res->bo = &mock_bo; diff --git a/drivers/gpu/drm/i915/intel_runtime_pm.c b/drivers/gpu/drm/i915/intel_runtime_pm.c index 744cca507946..129746713d07 100644 --- a/drivers/gpu/drm/i915/intel_runtime_pm.c +++ b/drivers/gpu/drm/i915/intel_runtime_pm.c @@ -633,6 +633,8 @@ void intel_runtime_pm_driver_release(struct intel_runtime_pm *rpm) runtime_pm); int count = atomic_read(&rpm->wakeref_count); + intel_wakeref_auto_fini(&rpm->userfault_wakeref); + drm_WARN(&i915->drm, count, "i915 raw-wakerefs=%d wakelocks=%d on cleanup\n", intel_rpm_raw_wakeref_count(count), @@ -652,4 +654,7 @@ void intel_runtime_pm_init_early(struct intel_runtime_pm *rpm) rpm->available = HAS_RUNTIME_PM(i915); init_intel_runtime_pm_wakeref(rpm); + INIT_LIST_HEAD(&rpm->lmem_userfault_list); + spin_lock_init(&rpm->lmem_userfault_lock); + intel_wakeref_auto_init(&rpm->userfault_wakeref, rpm); } diff --git a/drivers/gpu/drm/i915/intel_runtime_pm.h b/drivers/gpu/drm/i915/intel_runtime_pm.h index d9160e3ff4af..98b8b28baaa1 100644 --- a/drivers/gpu/drm/i915/intel_runtime_pm.h +++ b/drivers/gpu/drm/i915/intel_runtime_pm.h @@ -53,6 +53,28 @@ struct intel_runtime_pm { bool irqs_enabled; bool no_wakeref_tracking; + /* + * Protects access to lmem usefault list. + * It is required, if we are outside of the runtime suspend path, + * access to @lmem_userfault_list requires always first grabbing the + * runtime pm, to ensure we can't race against runtime suspend. + * Once we have that we also need to grab @lmem_userfault_lock, + * at which point we have exclusive access. + * The runtime suspend path is special since it doesn't really hold any locks, + * but instead has exclusive access by virtue of all other accesses requiring + * holding the runtime pm wakeref. + */ + spinlock_t lmem_userfault_lock; + + /* + * Keep list of userfaulted gem obj, which require to release their + * mmap mappings at runtime suspend path. + */ + struct list_head lmem_userfault_list; + + /* Manual runtime pm autosuspend delay for user GGTT/lmem mmaps */ + struct intel_wakeref_auto userfault_wakeref; + #if IS_ENABLED(CONFIG_DRM_I915_DEBUG_RUNTIME_PM) /* * To aide detection of wakeref leaks and general misuse, we diff --git a/drivers/gpu/drm/i915/intel_step.c b/drivers/gpu/drm/i915/intel_step.c index 42b3133d8387..84a6fe736a3b 100644 --- a/drivers/gpu/drm/i915/intel_step.c +++ b/drivers/gpu/drm/i915/intel_step.c @@ -131,10 +131,27 @@ static const struct intel_step_info adls_rpls_revids[] = { [0xC] = { COMMON_GT_MEDIA_STEP(D0), .display_step = STEP_C0 }, }; +static const struct intel_step_info adlp_rplp_revids[] = { + [0x4] = { COMMON_GT_MEDIA_STEP(C0), .display_step = STEP_E0 }, +}; + static const struct intel_step_info adlp_n_revids[] = { [0x0] = { COMMON_GT_MEDIA_STEP(A0), .display_step = STEP_D0 }, }; +static u8 gmd_to_intel_step(struct drm_i915_private *i915, + struct intel_ip_version *gmd) +{ + u8 step = gmd->step + STEP_A0; + + if (step >= STEP_FUTURE) { + drm_dbg(&i915->drm, "Using future steppings\n"); + return STEP_FUTURE; + } + + return step; +} + static void pvc_step_init(struct drm_i915_private *i915, int pci_revid); void intel_step_init(struct drm_i915_private *i915) @@ -144,6 +161,18 @@ void intel_step_init(struct drm_i915_private *i915) int revid = INTEL_REVID(i915); struct intel_step_info step = {}; + if (HAS_GMD_ID(i915)) { + step.graphics_step = gmd_to_intel_step(i915, + &RUNTIME_INFO(i915)->graphics.ip); + step.media_step = gmd_to_intel_step(i915, + &RUNTIME_INFO(i915)->media.ip); + step.display_step = gmd_to_intel_step(i915, + &RUNTIME_INFO(i915)->display.ip); + RUNTIME_INFO(i915)->step = step; + + return; + } + if (IS_PONTEVECCHIO(i915)) { pvc_step_init(i915, revid); return; @@ -162,6 +191,9 @@ void intel_step_init(struct drm_i915_private *i915) } else if (IS_ADLP_N(i915)) { revids = adlp_n_revids; size = ARRAY_SIZE(adlp_n_revids); + } else if (IS_ADLP_RPLP(i915)) { + revids = adlp_rplp_revids; + size = ARRAY_SIZE(adlp_rplp_revids); } else if (IS_ALDERLAKE_P(i915)) { revids = adlp_revids; size = ARRAY_SIZE(adlp_revids); diff --git a/drivers/gpu/drm/i915/intel_step.h b/drivers/gpu/drm/i915/intel_step.h index a6b12bfa9744..96dfca4cba73 100644 --- a/drivers/gpu/drm/i915/intel_step.h +++ b/drivers/gpu/drm/i915/intel_step.h @@ -11,6 +11,10 @@ struct drm_i915_private; struct intel_step_info { + /* + * It is expected to have 4 number steps per letter. Deviation from + * the expectation breaks gmd_to_intel_step(). + */ u8 graphics_step; /* Represents the compute tile on Xe_HPC */ u8 display_step; u8 media_step; @@ -23,21 +27,43 @@ struct intel_step_info { func(A0) \ func(A1) \ func(A2) \ + func(A3) \ func(B0) \ func(B1) \ func(B2) \ func(B3) \ func(C0) \ func(C1) \ + func(C2) \ + func(C3) \ func(D0) \ func(D1) \ + func(D2) \ + func(D3) \ func(E0) \ + func(E1) \ + func(E2) \ + func(E3) \ func(F0) \ + func(F1) \ + func(F2) \ + func(F3) \ func(G0) \ + func(G1) \ + func(G2) \ + func(G3) \ func(H0) \ + func(H1) \ + func(H2) \ + func(H3) \ func(I0) \ func(I1) \ - func(J0) + func(I2) \ + func(I3) \ + func(J0) \ + func(J1) \ + func(J2) \ + func(J3) /* * Symbolic steppings that do not match the hardware. These are valid both as gt diff --git a/drivers/gpu/drm/i915/intel_uncore.c b/drivers/gpu/drm/i915/intel_uncore.c index 5cd423c7b646..8006a6c61466 100644 --- a/drivers/gpu/drm/i915/intel_uncore.c +++ b/drivers/gpu/drm/i915/intel_uncore.c @@ -29,6 +29,7 @@ #include "i915_drv.h" #include "i915_iosf_mbi.h" +#include "i915_reg.h" #include "i915_trace.h" #include "i915_vgpu.h" #include "intel_pm.h" @@ -104,6 +105,7 @@ static const char * const forcewake_domain_names[] = { "vebox1", "vebox2", "vebox3", + "gsc", }; const char * @@ -177,8 +179,9 @@ static inline void fw_domain_wait_ack_clear(const struct intel_uncore_forcewake_domain *d) { if (wait_ack_clear(d, FORCEWAKE_KERNEL)) { - DRM_ERROR("%s: timed out waiting for forcewake ack to clear.\n", - intel_uncore_forcewake_domain_to_str(d->id)); + drm_err(&d->uncore->i915->drm, + "%s: timed out waiting for forcewake ack to clear.\n", + intel_uncore_forcewake_domain_to_str(d->id)); add_taint_for_CI(d->uncore->i915, TAINT_WARN); /* CI now unreliable */ } } @@ -225,11 +228,12 @@ fw_domain_wait_ack_with_fallback(const struct intel_uncore_forcewake_domain *d, fw_clear(d, FORCEWAKE_KERNEL_FALLBACK); } while (!ack_detected && pass++ < 10); - DRM_DEBUG_DRIVER("%s had to use fallback to %s ack, 0x%x (passes %u)\n", - intel_uncore_forcewake_domain_to_str(d->id), - type == ACK_SET ? "set" : "clear", - fw_ack(d), - pass); + drm_dbg(&d->uncore->i915->drm, + "%s had to use fallback to %s ack, 0x%x (passes %u)\n", + intel_uncore_forcewake_domain_to_str(d->id), + type == ACK_SET ? "set" : "clear", + fw_ack(d), + pass); return ack_detected ? 0 : -ETIMEDOUT; } @@ -254,8 +258,9 @@ static inline void fw_domain_wait_ack_set(const struct intel_uncore_forcewake_domain *d) { if (wait_ack_set(d, FORCEWAKE_KERNEL)) { - DRM_ERROR("%s: timed out waiting for forcewake ack request.\n", - intel_uncore_forcewake_domain_to_str(d->id)); + drm_err(&d->uncore->i915->drm, + "%s: timed out waiting for forcewake ack request.\n", + intel_uncore_forcewake_domain_to_str(d->id)); add_taint_for_CI(d->uncore->i915, TAINT_WARN); /* CI now unreliable */ } } @@ -888,10 +893,13 @@ void assert_forcewakes_active(struct intel_uncore *uncore, spin_unlock_irq(&uncore->lock); } -/* We give fast paths for the really cool registers */ +/* + * We give fast paths for the really cool registers. The second range includes + * media domains (and the GSC starting from Xe_LPM+) + */ #define NEEDS_FORCE_WAKE(reg) ({ \ u32 __reg = (reg); \ - __reg < 0x40000 || __reg >= GEN11_BSD_RING_BASE; \ + __reg < 0x40000 || __reg >= 0x116000; \ }) static int fw_range_cmp(u32 offset, const struct intel_forcewake_range *entry) @@ -1131,6 +1139,45 @@ static const struct i915_range pvc_shadowed_regs[] = { { .start = 0x1F8510, .end = 0x1F8550 }, }; +static const struct i915_range mtl_shadowed_regs[] = { + { .start = 0x2030, .end = 0x2030 }, + { .start = 0x2510, .end = 0x2550 }, + { .start = 0xA008, .end = 0xA00C }, + { .start = 0xA188, .end = 0xA188 }, + { .start = 0xA278, .end = 0xA278 }, + { .start = 0xA540, .end = 0xA56C }, + { .start = 0xC050, .end = 0xC050 }, + { .start = 0xC340, .end = 0xC340 }, + { .start = 0xC4C8, .end = 0xC4C8 }, + { .start = 0xC4E0, .end = 0xC4E0 }, + { .start = 0xC600, .end = 0xC600 }, + { .start = 0xC658, .end = 0xC658 }, + { .start = 0xCFD4, .end = 0xCFDC }, + { .start = 0x22030, .end = 0x22030 }, + { .start = 0x22510, .end = 0x22550 }, +}; + +static const struct i915_range xelpmp_shadowed_regs[] = { + { .start = 0x1C0030, .end = 0x1C0030 }, + { .start = 0x1C0510, .end = 0x1C0550 }, + { .start = 0x1C8030, .end = 0x1C8030 }, + { .start = 0x1C8510, .end = 0x1C8550 }, + { .start = 0x1D0030, .end = 0x1D0030 }, + { .start = 0x1D0510, .end = 0x1D0550 }, + { .start = 0x38A008, .end = 0x38A00C }, + { .start = 0x38A188, .end = 0x38A188 }, + { .start = 0x38A278, .end = 0x38A278 }, + { .start = 0x38A540, .end = 0x38A56C }, + { .start = 0x38A618, .end = 0x38A618 }, + { .start = 0x38C050, .end = 0x38C050 }, + { .start = 0x38C340, .end = 0x38C340 }, + { .start = 0x38C4C8, .end = 0x38C4C8 }, + { .start = 0x38C4E0, .end = 0x38C4E4 }, + { .start = 0x38C600, .end = 0x38C600 }, + { .start = 0x38C658, .end = 0x38C658 }, + { .start = 0x38CFD4, .end = 0x38CFDC }, +}; + static int mmio_range_cmp(u32 key, const struct i915_range *range) { if (key < range->start) @@ -1639,25 +1686,27 @@ static const struct intel_forcewake_range __pvc_fw_ranges[] = { GEN_FW_RANGE(0x12000, 0x12fff, 0), /* 0x12000 - 0x127ff: always on 0x12800 - 0x12fff: reserved */ - GEN_FW_RANGE(0x13000, 0x23fff, FORCEWAKE_GT), /* + GEN_FW_RANGE(0x13000, 0x19fff, FORCEWAKE_GT), /* 0x13000 - 0x135ff: gt 0x13600 - 0x147ff: reserved 0x14800 - 0x153ff: gt - 0x15400 - 0x19fff: reserved - 0x1a000 - 0x1ffff: gt - 0x20000 - 0x21fff: reserved - 0x22000 - 0x23fff: gt */ + 0x15400 - 0x19fff: reserved */ + GEN_FW_RANGE(0x1a000, 0x21fff, FORCEWAKE_RENDER), /* + 0x1a000 - 0x1ffff: render + 0x20000 - 0x21fff: reserved */ + GEN_FW_RANGE(0x22000, 0x23fff, FORCEWAKE_GT), GEN_FW_RANGE(0x24000, 0x2417f, 0), /* 24000 - 0x2407f: always on 24080 - 0x2417f: reserved */ - GEN_FW_RANGE(0x24180, 0x3ffff, FORCEWAKE_GT), /* + GEN_FW_RANGE(0x24180, 0x25fff, FORCEWAKE_GT), /* 0x24180 - 0x241ff: gt 0x24200 - 0x251ff: reserved 0x25200 - 0x252ff: gt - 0x25300 - 0x25fff: reserved - 0x26000 - 0x27fff: gt - 0x28000 - 0x2ffff: reserved - 0x30000 - 0x3ffff: gt */ + 0x25300 - 0x25fff: reserved */ + GEN_FW_RANGE(0x26000, 0x2ffff, FORCEWAKE_RENDER), /* + 0x26000 - 0x27fff: render + 0x28000 - 0x2ffff: reserved */ + GEN_FW_RANGE(0x30000, 0x3ffff, FORCEWAKE_GT), GEN_FW_RANGE(0x40000, 0x1bffff, 0), GEN_FW_RANGE(0x1c0000, 0x1c3fff, FORCEWAKE_MEDIA_VDBOX0), /* 0x1c0000 - 0x1c2bff: VD0 @@ -1679,6 +1728,162 @@ static const struct intel_forcewake_range __pvc_fw_ranges[] = { GEN_FW_RANGE(0x3e0000, 0x3effff, FORCEWAKE_GT), }; +static const struct intel_forcewake_range __mtl_fw_ranges[] = { + GEN_FW_RANGE(0x0, 0xaff, 0), + GEN_FW_RANGE(0xb00, 0xbff, FORCEWAKE_GT), + GEN_FW_RANGE(0xc00, 0xfff, 0), + GEN_FW_RANGE(0x1000, 0x1fff, FORCEWAKE_GT), + GEN_FW_RANGE(0x2000, 0x26ff, FORCEWAKE_RENDER), + GEN_FW_RANGE(0x2700, 0x2fff, FORCEWAKE_GT), + GEN_FW_RANGE(0x3000, 0x3fff, FORCEWAKE_RENDER), + GEN_FW_RANGE(0x4000, 0x51ff, FORCEWAKE_GT), /* + 0x4000 - 0x48ff: render + 0x4900 - 0x51ff: reserved */ + GEN_FW_RANGE(0x5200, 0x7fff, FORCEWAKE_RENDER), /* + 0x5200 - 0x53ff: render + 0x5400 - 0x54ff: reserved + 0x5500 - 0x7fff: render */ + GEN_FW_RANGE(0x8000, 0x813f, FORCEWAKE_GT), + GEN_FW_RANGE(0x8140, 0x817f, FORCEWAKE_RENDER), /* + 0x8140 - 0x815f: render + 0x8160 - 0x817f: reserved */ + GEN_FW_RANGE(0x8180, 0x81ff, 0), + GEN_FW_RANGE(0x8200, 0x94cf, FORCEWAKE_GT), /* + 0x8200 - 0x87ff: gt + 0x8800 - 0x8dff: reserved + 0x8e00 - 0x8f7f: gt + 0x8f80 - 0x8fff: reserved + 0x9000 - 0x947f: gt + 0x9480 - 0x94cf: reserved */ + GEN_FW_RANGE(0x94d0, 0x955f, FORCEWAKE_RENDER), + GEN_FW_RANGE(0x9560, 0x967f, 0), /* + 0x9560 - 0x95ff: always on + 0x9600 - 0x967f: reserved */ + GEN_FW_RANGE(0x9680, 0x97ff, FORCEWAKE_RENDER), /* + 0x9680 - 0x96ff: render + 0x9700 - 0x97ff: reserved */ + GEN_FW_RANGE(0x9800, 0xcfff, FORCEWAKE_GT), /* + 0x9800 - 0xb4ff: gt + 0xb500 - 0xbfff: reserved + 0xc000 - 0xcfff: gt */ + GEN_FW_RANGE(0xd000, 0xd7ff, 0), /* + 0xd000 - 0xd3ff: always on + 0xd400 - 0xd7ff: reserved */ + GEN_FW_RANGE(0xd800, 0xd87f, FORCEWAKE_RENDER), + GEN_FW_RANGE(0xd880, 0xdbff, FORCEWAKE_GT), + GEN_FW_RANGE(0xdc00, 0xdcff, FORCEWAKE_RENDER), + GEN_FW_RANGE(0xdd00, 0xde7f, FORCEWAKE_GT), /* + 0xdd00 - 0xddff: gt + 0xde00 - 0xde7f: reserved */ + GEN_FW_RANGE(0xde80, 0xe8ff, FORCEWAKE_RENDER), /* + 0xde80 - 0xdfff: render + 0xe000 - 0xe0ff: reserved + 0xe100 - 0xe8ff: render */ + GEN_FW_RANGE(0xe900, 0xe9ff, FORCEWAKE_GT), + GEN_FW_RANGE(0xea00, 0x147ff, 0), /* + 0xea00 - 0x11fff: reserved + 0x12000 - 0x127ff: always on + 0x12800 - 0x147ff: reserved */ + GEN_FW_RANGE(0x14800, 0x19fff, FORCEWAKE_GT), /* + 0x14800 - 0x153ff: gt + 0x15400 - 0x19fff: reserved */ + GEN_FW_RANGE(0x1a000, 0x21fff, FORCEWAKE_RENDER), /* + 0x1a000 - 0x1bfff: render + 0x1c000 - 0x21fff: reserved */ + GEN_FW_RANGE(0x22000, 0x23fff, FORCEWAKE_GT), + GEN_FW_RANGE(0x24000, 0x2ffff, 0), /* + 0x24000 - 0x2407f: always on + 0x24080 - 0x2ffff: reserved */ + GEN_FW_RANGE(0x30000, 0x3ffff, FORCEWAKE_GT) +}; + +/* + * Note that the register ranges here are the final offsets after + * translation of the GSI block to the 0x380000 offset. + * + * NOTE: There are a couple MCR ranges near the bottom of this table + * that need to power up either VD0 or VD2 depending on which replicated + * instance of the register we're trying to access. Our forcewake logic + * at the moment doesn't have a good way to take steering into consideration, + * and the driver doesn't even access any registers in those ranges today, + * so for now we just mark those ranges as FORCEWAKE_ALL. That will ensure + * proper operation if we do start using the ranges in the future, and we + * can determine at that time whether it's worth adding extra complexity to + * the forcewake handling to take steering into consideration. + */ +static const struct intel_forcewake_range __xelpmp_fw_ranges[] = { + GEN_FW_RANGE(0x0, 0x115fff, 0), /* render GT range */ + GEN_FW_RANGE(0x116000, 0x11ffff, FORCEWAKE_GSC), /* + 0x116000 - 0x117fff: gsc + 0x118000 - 0x119fff: reserved + 0x11a000 - 0x11efff: gsc + 0x11f000 - 0x11ffff: reserved */ + GEN_FW_RANGE(0x120000, 0x1bffff, 0), /* non-GT range */ + GEN_FW_RANGE(0x1c0000, 0x1c7fff, FORCEWAKE_MEDIA_VDBOX0), /* + 0x1c0000 - 0x1c3dff: VD0 + 0x1c3e00 - 0x1c3eff: reserved + 0x1c3f00 - 0x1c3fff: VD0 + 0x1c4000 - 0x1c7fff: reserved */ + GEN_FW_RANGE(0x1c8000, 0x1cbfff, FORCEWAKE_MEDIA_VEBOX0), /* + 0x1c8000 - 0x1ca0ff: VE0 + 0x1ca100 - 0x1cbfff: reserved */ + GEN_FW_RANGE(0x1cc000, 0x1cffff, FORCEWAKE_MEDIA_VDBOX0), /* + 0x1cc000 - 0x1cdfff: VD0 + 0x1ce000 - 0x1cffff: reserved */ + GEN_FW_RANGE(0x1d0000, 0x1d7fff, FORCEWAKE_MEDIA_VDBOX2), /* + 0x1d0000 - 0x1d3dff: VD2 + 0x1d3e00 - 0x1d3eff: reserved + 0x1d4000 - 0x1d7fff: VD2 */ + GEN_FW_RANGE(0x1d8000, 0x1da0ff, FORCEWAKE_MEDIA_VEBOX1), + GEN_FW_RANGE(0x1da100, 0x380aff, 0), /* + 0x1da100 - 0x23ffff: reserved + 0x240000 - 0x37ffff: non-GT range + 0x380000 - 0x380aff: reserved */ + GEN_FW_RANGE(0x380b00, 0x380bff, FORCEWAKE_GT), + GEN_FW_RANGE(0x380c00, 0x380fff, 0), + GEN_FW_RANGE(0x381000, 0x38817f, FORCEWAKE_GT), /* + 0x381000 - 0x381fff: gt + 0x382000 - 0x383fff: reserved + 0x384000 - 0x384aff: gt + 0x384b00 - 0x3851ff: reserved + 0x385200 - 0x3871ff: gt + 0x387200 - 0x387fff: reserved + 0x388000 - 0x38813f: gt + 0x388140 - 0x38817f: reserved */ + GEN_FW_RANGE(0x388180, 0x3882ff, 0), /* + 0x388180 - 0x3881ff: always on + 0x388200 - 0x3882ff: reserved */ + GEN_FW_RANGE(0x388300, 0x38955f, FORCEWAKE_GT), /* + 0x388300 - 0x38887f: gt + 0x388880 - 0x388fff: reserved + 0x389000 - 0x38947f: gt + 0x389480 - 0x38955f: reserved */ + GEN_FW_RANGE(0x389560, 0x389fff, 0), /* + 0x389560 - 0x3895ff: always on + 0x389600 - 0x389fff: reserved */ + GEN_FW_RANGE(0x38a000, 0x38cfff, FORCEWAKE_GT), /* + 0x38a000 - 0x38afff: gt + 0x38b000 - 0x38bfff: reserved + 0x38c000 - 0x38cfff: gt */ + GEN_FW_RANGE(0x38d000, 0x38d11f, 0), + GEN_FW_RANGE(0x38d120, 0x391fff, FORCEWAKE_GT), /* + 0x38d120 - 0x38dfff: gt + 0x38e000 - 0x38efff: reserved + 0x38f000 - 0x38ffff: gt + 0x389000 - 0x391fff: reserved */ + GEN_FW_RANGE(0x392000, 0x392fff, 0), /* + 0x392000 - 0x3927ff: always on + 0x392800 - 0x292fff: reserved */ + GEN_FW_RANGE(0x393000, 0x3931ff, FORCEWAKE_GT), + GEN_FW_RANGE(0x393200, 0x39323f, FORCEWAKE_ALL), /* instance-based, see note above */ + GEN_FW_RANGE(0x393240, 0x3933ff, FORCEWAKE_GT), + GEN_FW_RANGE(0x393400, 0x3934ff, FORCEWAKE_ALL), /* instance-based, see note above */ + GEN_FW_RANGE(0x393500, 0x393c7f, 0), /* + 0x393500 - 0x393bff: reserved + 0x393c00 - 0x393c7f: always on */ + GEN_FW_RANGE(0x393c80, 0x393dff, FORCEWAKE_GT), +}; + static void ilk_dummy_write(struct intel_uncore *uncore) { @@ -2021,6 +2226,7 @@ static int __fw_domain_init(struct intel_uncore *uncore, BUILD_BUG_ON(FORCEWAKE_MEDIA_VEBOX1 != (1 << FW_DOMAIN_ID_MEDIA_VEBOX1)); BUILD_BUG_ON(FORCEWAKE_MEDIA_VEBOX2 != (1 << FW_DOMAIN_ID_MEDIA_VEBOX2)); BUILD_BUG_ON(FORCEWAKE_MEDIA_VEBOX3 != (1 << FW_DOMAIN_ID_MEDIA_VEBOX3)); + BUILD_BUG_ON(FORCEWAKE_GSC != (1 << FW_DOMAIN_ID_GSC)); d->mask = BIT(domain_id); @@ -2085,17 +2291,26 @@ static int intel_uncore_fw_domains_init(struct intel_uncore *uncore) (ret ?: (ret = __fw_domain_init((uncore__), (id__), (set__), (ack__)))) if (GRAPHICS_VER(i915) >= 11) { - /* we'll prune the domains of missing engines later */ - intel_engine_mask_t emask = RUNTIME_INFO(i915)->platform_engine_mask; + intel_engine_mask_t emask; int i; + /* we'll prune the domains of missing engines later */ + emask = uncore->gt->info.engine_mask; + uncore->fw_get_funcs = &uncore_get_fallback; - fw_domain_init(uncore, FW_DOMAIN_ID_RENDER, - FORCEWAKE_RENDER_GEN9, - FORCEWAKE_ACK_RENDER_GEN9); - fw_domain_init(uncore, FW_DOMAIN_ID_GT, - FORCEWAKE_GT_GEN9, - FORCEWAKE_ACK_GT_GEN9); + if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 70)) + fw_domain_init(uncore, FW_DOMAIN_ID_GT, + FORCEWAKE_GT_GEN9, + FORCEWAKE_ACK_GT_MTL); + else + fw_domain_init(uncore, FW_DOMAIN_ID_GT, + FORCEWAKE_GT_GEN9, + FORCEWAKE_ACK_GT_GEN9); + + if (RCS_MASK(uncore->gt) || CCS_MASK(uncore->gt)) + fw_domain_init(uncore, FW_DOMAIN_ID_RENDER, + FORCEWAKE_RENDER_GEN9, + FORCEWAKE_ACK_RENDER_GEN9); for (i = 0; i < I915_MAX_VCS; i++) { if (!__HAS_ENGINE(emask, _VCS(i))) @@ -2113,6 +2328,10 @@ static int intel_uncore_fw_domains_init(struct intel_uncore *uncore) FORCEWAKE_MEDIA_VEBOX_GEN11(i), FORCEWAKE_ACK_MEDIA_VEBOX_GEN11(i)); } + + if (uncore->gt->type == GT_MEDIA) + fw_domain_init(uncore, FW_DOMAIN_ID_GSC, + FORCEWAKE_REQ_GSC, FORCEWAKE_ACK_GSC); } else if (IS_GRAPHICS_VER(i915, 9, 10)) { uncore->fw_get_funcs = &uncore_get_fallback; fw_domain_init(uncore, FW_DOMAIN_ID_RENDER, @@ -2300,6 +2519,22 @@ static void uncore_raw_init(struct intel_uncore *uncore) } } +static int uncore_media_forcewake_init(struct intel_uncore *uncore) +{ + struct drm_i915_private *i915 = uncore->i915; + + if (MEDIA_VER(i915) >= 13) { + ASSIGN_FW_DOMAINS_TABLE(uncore, __xelpmp_fw_ranges); + ASSIGN_SHADOW_TABLE(uncore, xelpmp_shadowed_regs); + ASSIGN_WRITE_MMIO_VFUNCS(uncore, fwtable); + } else { + MISSING_CASE(MEDIA_VER(i915)); + return -ENODEV; + } + + return 0; +} + static int uncore_forcewake_init(struct intel_uncore *uncore) { struct drm_i915_private *i915 = uncore->i915; @@ -2314,7 +2549,14 @@ static int uncore_forcewake_init(struct intel_uncore *uncore) ASSIGN_READ_MMIO_VFUNCS(uncore, fwtable); - if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 60)) { + if (uncore->gt->type == GT_MEDIA) + return uncore_media_forcewake_init(uncore); + + if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 70)) { + ASSIGN_FW_DOMAINS_TABLE(uncore, __mtl_fw_ranges); + ASSIGN_SHADOW_TABLE(uncore, mtl_shadowed_regs); + ASSIGN_WRITE_MMIO_VFUNCS(uncore, fwtable); + } else if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 60)) { ASSIGN_FW_DOMAINS_TABLE(uncore, __pvc_fw_ranges); ASSIGN_SHADOW_TABLE(uncore, pvc_shadowed_regs); ASSIGN_WRITE_MMIO_VFUNCS(uncore, fwtable); diff --git a/drivers/gpu/drm/i915/intel_uncore.h b/drivers/gpu/drm/i915/intel_uncore.h index 5022bac80b67..e9e38490815d 100644 --- a/drivers/gpu/drm/i915/intel_uncore.h +++ b/drivers/gpu/drm/i915/intel_uncore.h @@ -62,6 +62,7 @@ enum forcewake_domain_id { FW_DOMAIN_ID_MEDIA_VEBOX1, FW_DOMAIN_ID_MEDIA_VEBOX2, FW_DOMAIN_ID_MEDIA_VEBOX3, + FW_DOMAIN_ID_GSC, FW_DOMAIN_ID_COUNT }; @@ -82,6 +83,7 @@ enum forcewake_domains { FORCEWAKE_MEDIA_VEBOX1 = BIT(FW_DOMAIN_ID_MEDIA_VEBOX1), FORCEWAKE_MEDIA_VEBOX2 = BIT(FW_DOMAIN_ID_MEDIA_VEBOX2), FORCEWAKE_MEDIA_VEBOX3 = BIT(FW_DOMAIN_ID_MEDIA_VEBOX3), + FORCEWAKE_GSC = BIT(FW_DOMAIN_ID_GSC), FORCEWAKE_ALL = BIT(FW_DOMAIN_ID_COUNT) - 1, }; @@ -380,20 +382,6 @@ __uncore_write(write_notrace, 32, l, false) */ __uncore_read(read64, 64, q, true) -static inline u64 -intel_uncore_read64_2x32(struct intel_uncore *uncore, - i915_reg_t lower_reg, i915_reg_t upper_reg) -{ - u32 upper, lower, old_upper, loop = 0; - upper = intel_uncore_read(uncore, upper_reg); - do { - old_upper = upper; - lower = intel_uncore_read(uncore, lower_reg); - upper = intel_uncore_read(uncore, upper_reg); - } while (upper != old_upper && loop++ < 2); - return (u64)upper << 32 | lower; -} - #define intel_uncore_posting_read(...) ((void)intel_uncore_read_notrace(__VA_ARGS__)) #define intel_uncore_posting_read16(...) ((void)intel_uncore_read16_notrace(__VA_ARGS__)) @@ -431,15 +419,15 @@ intel_uncore_read64_2x32(struct intel_uncore *uncore, #define intel_uncore_write64_fw(...) __raw_uncore_write64(__VA_ARGS__) #define intel_uncore_posting_read_fw(...) ((void)intel_uncore_read_fw(__VA_ARGS__)) -static inline void intel_uncore_rmw(struct intel_uncore *uncore, - i915_reg_t reg, u32 clear, u32 set) +static inline u32 intel_uncore_rmw(struct intel_uncore *uncore, + i915_reg_t reg, u32 clear, u32 set) { u32 old, val; old = intel_uncore_read(uncore, reg); val = (old & ~clear) | set; - if (val != old) - intel_uncore_write(uncore, reg, val); + intel_uncore_write(uncore, reg, val); + return old; } static inline void intel_uncore_rmw_fw(struct intel_uncore *uncore, @@ -453,6 +441,36 @@ static inline void intel_uncore_rmw_fw(struct intel_uncore *uncore, intel_uncore_write_fw(uncore, reg, val); } +static inline u64 +intel_uncore_read64_2x32(struct intel_uncore *uncore, + i915_reg_t lower_reg, i915_reg_t upper_reg) +{ + u32 upper, lower, old_upper, loop = 0; + enum forcewake_domains fw_domains; + unsigned long flags; + + fw_domains = intel_uncore_forcewake_for_reg(uncore, lower_reg, + FW_REG_READ); + + fw_domains |= intel_uncore_forcewake_for_reg(uncore, upper_reg, + FW_REG_READ); + + spin_lock_irqsave(&uncore->lock, flags); + intel_uncore_forcewake_get__locked(uncore, fw_domains); + + upper = intel_uncore_read_fw(uncore, upper_reg); + do { + old_upper = upper; + lower = intel_uncore_read_fw(uncore, lower_reg); + upper = intel_uncore_read_fw(uncore, upper_reg); + } while (upper != old_upper && loop++ < 2); + + intel_uncore_forcewake_put__locked(uncore, fw_domains); + spin_unlock_irqrestore(&uncore->lock, flags); + + return (u64)upper << 32 | lower; +} + static inline int intel_uncore_write_and_verify(struct intel_uncore *uncore, i915_reg_t reg, u32 val, u32 mask, u32 expected_val) diff --git a/drivers/gpu/drm/i915/pxp/intel_pxp.c b/drivers/gpu/drm/i915/pxp/intel_pxp.c index 69cdaaddc4a9..5efe61f67546 100644 --- a/drivers/gpu/drm/i915/pxp/intel_pxp.c +++ b/drivers/gpu/drm/i915/pxp/intel_pxp.c @@ -103,19 +103,15 @@ static int create_vcs_context(struct intel_pxp *pxp) static void destroy_vcs_context(struct intel_pxp *pxp) { - intel_engine_destroy_pinned_context(fetch_and_zero(&pxp->ce)); + if (pxp->ce) + intel_engine_destroy_pinned_context(fetch_and_zero(&pxp->ce)); } -void intel_pxp_init(struct intel_pxp *pxp) +static void pxp_init_full(struct intel_pxp *pxp) { struct intel_gt *gt = pxp_to_gt(pxp); int ret; - if (!HAS_PXP(gt->i915)) - return; - - mutex_init(&pxp->tee_mutex); - /* * we'll use the completion to check if there is a termination pending, * so we start it as completed and we reinit it when a termination @@ -124,8 +120,7 @@ void intel_pxp_init(struct intel_pxp *pxp) init_completion(&pxp->termination); complete_all(&pxp->termination); - mutex_init(&pxp->arb_mutex); - INIT_WORK(&pxp->session_work, intel_pxp_session_work); + intel_pxp_session_management_init(pxp); ret = create_vcs_context(pxp); if (ret) @@ -143,11 +138,26 @@ out_context: destroy_vcs_context(pxp); } -void intel_pxp_fini(struct intel_pxp *pxp) +void intel_pxp_init(struct intel_pxp *pxp) { - if (!intel_pxp_is_enabled(pxp)) + struct intel_gt *gt = pxp_to_gt(pxp); + + /* we rely on the mei PXP module */ + if (!IS_ENABLED(CONFIG_INTEL_MEI_PXP)) return; + /* + * If HuC is loaded by GSC but PXP is disabled, we can skip the init of + * the full PXP session/object management and just init the tee channel. + */ + if (HAS_PXP(gt->i915)) + pxp_init_full(pxp); + else if (intel_huc_is_loaded_by_gsc(>->uc.huc) && intel_uc_uses_huc(>->uc)) + intel_pxp_tee_component_init(pxp); +} + +void intel_pxp_fini(struct intel_pxp *pxp) +{ pxp->arb_is_valid = false; intel_pxp_tee_component_fini(pxp); diff --git a/drivers/gpu/drm/i915/pxp/intel_pxp.h b/drivers/gpu/drm/i915/pxp/intel_pxp.h index 73847e535cab..2da309088c6d 100644 --- a/drivers/gpu/drm/i915/pxp/intel_pxp.h +++ b/drivers/gpu/drm/i915/pxp/intel_pxp.h @@ -12,7 +12,6 @@ struct intel_pxp; struct drm_i915_gem_object; -#ifdef CONFIG_DRM_I915_PXP struct intel_gt *pxp_to_gt(const struct intel_pxp *pxp); bool intel_pxp_is_enabled(const struct intel_pxp *pxp); bool intel_pxp_is_active(const struct intel_pxp *pxp); @@ -32,36 +31,5 @@ int intel_pxp_key_check(struct intel_pxp *pxp, bool assign); void intel_pxp_invalidate(struct intel_pxp *pxp); -#else -static inline void intel_pxp_init(struct intel_pxp *pxp) -{ -} - -static inline void intel_pxp_fini(struct intel_pxp *pxp) -{ -} - -static inline int intel_pxp_start(struct intel_pxp *pxp) -{ - return -ENODEV; -} - -static inline bool intel_pxp_is_enabled(const struct intel_pxp *pxp) -{ - return false; -} - -static inline bool intel_pxp_is_active(const struct intel_pxp *pxp) -{ - return false; -} - -static inline int intel_pxp_key_check(struct intel_pxp *pxp, - struct drm_i915_gem_object *obj, - bool assign) -{ - return -ENODEV; -} -#endif #endif /* __INTEL_PXP_H__ */ diff --git a/drivers/gpu/drm/i915/pxp/intel_pxp_cmd_interface_42.h b/drivers/gpu/drm/i915/pxp/intel_pxp_cmd_interface_42.h new file mode 100644 index 000000000000..739f9072fa5f --- /dev/null +++ b/drivers/gpu/drm/i915/pxp/intel_pxp_cmd_interface_42.h @@ -0,0 +1,28 @@ +/* SPDX-License-Identifier: MIT */ +/* + * Copyright(c) 2020, Intel Corporation. All rights reserved. + */ + +#ifndef __INTEL_PXP_FW_INTERFACE_42_H__ +#define __INTEL_PXP_FW_INTERFACE_42_H__ + +#include <linux/types.h> +#include "intel_pxp_cmd_interface_cmn.h" + +/* PXP-Opcode for Init Session */ +#define PXP42_CMDID_INIT_SESSION 0x1e + +/* PXP-Input-Packet: Init Session (Arb-Session) */ +struct pxp42_create_arb_in { + struct pxp_cmd_header header; + u32 protection_mode; +#define PXP42_ARB_SESSION_MODE_HEAVY 0x2 + u32 session_id; +} __packed; + +/* PXP-Output-Packet: Init Session */ +struct pxp42_create_arb_out { + struct pxp_cmd_header header; +} __packed; + +#endif /* __INTEL_PXP_FW_INTERFACE_42_H__ */ diff --git a/drivers/gpu/drm/i915/pxp/intel_pxp_cmd_interface_43.h b/drivers/gpu/drm/i915/pxp/intel_pxp_cmd_interface_43.h new file mode 100644 index 000000000000..ad67e3f49c20 --- /dev/null +++ b/drivers/gpu/drm/i915/pxp/intel_pxp_cmd_interface_43.h @@ -0,0 +1,26 @@ +/* SPDX-License-Identifier: MIT */ +/* + * Copyright(c) 2022, Intel Corporation. All rights reserved. + */ + +#ifndef __INTEL_PXP_FW_INTERFACE_43_H__ +#define __INTEL_PXP_FW_INTERFACE_43_H__ + +#include <linux/types.h> +#include "intel_pxp_cmd_interface_cmn.h" + +/* PXP-Cmd-Op definitions */ +#define PXP43_CMDID_START_HUC_AUTH 0x0000003A + +/* PXP-Input-Packet: HUC-Authentication */ +struct pxp43_start_huc_auth_in { + struct pxp_cmd_header header; + __le64 huc_base_address; +} __packed; + +/* PXP-Output-Packet: HUC-Authentication */ +struct pxp43_start_huc_auth_out { + struct pxp_cmd_header header; +} __packed; + +#endif /* __INTEL_PXP_FW_INTERFACE_43_H__ */ diff --git a/drivers/gpu/drm/i915/pxp/intel_pxp_cmd_interface_cmn.h b/drivers/gpu/drm/i915/pxp/intel_pxp_cmd_interface_cmn.h new file mode 100644 index 000000000000..c2f23394f9b8 --- /dev/null +++ b/drivers/gpu/drm/i915/pxp/intel_pxp_cmd_interface_cmn.h @@ -0,0 +1,35 @@ +/* SPDX-License-Identifier: MIT */ +/* + * Copyright(c) 2022, Intel Corporation. All rights reserved. + */ + +#ifndef __INTEL_PXP_FW_INTERFACE_CMN_H__ +#define __INTEL_PXP_FW_INTERFACE_CMN_H__ + +#include <linux/types.h> + +#define PXP_APIVER(x, y) (((x) & 0xFFFF) << 16 | ((y) & 0xFFFF)) + +/* + * there are a lot of status codes for PXP, but we only define the cross-API + * common ones that we actually can handle in the kernel driver. Other failure + * codes should be printed to error msg for debug. + */ +enum pxp_status { + PXP_STATUS_SUCCESS = 0x0, + PXP_STATUS_OP_NOT_PERMITTED = 0x4013 +}; + +/* Common PXP FW message header */ +struct pxp_cmd_header { + u32 api_version; + u32 command_id; + union { + u32 status; /* out */ + u32 stream_id; /* in */ + }; + /* Length of the message (excluding the header) */ + u32 buffer_len; +} __packed; + +#endif /* __INTEL_PXP_FW_INTERFACE_CMN_H__ */ diff --git a/drivers/gpu/drm/i915/pxp/intel_pxp_huc.c b/drivers/gpu/drm/i915/pxp/intel_pxp_huc.c new file mode 100644 index 000000000000..2e1165522950 --- /dev/null +++ b/drivers/gpu/drm/i915/pxp/intel_pxp_huc.c @@ -0,0 +1,70 @@ +// SPDX-License-Identifier: MIT +/* + * Copyright(c) 2021-2022, Intel Corporation. All rights reserved. + */ + +#include <drm/i915_drm.h> + +#include "i915_drv.h" + +#include "gem/i915_gem_region.h" +#include "gt/intel_gt.h" + +#include "intel_pxp.h" +#include "intel_pxp_huc.h" +#include "intel_pxp_tee.h" +#include "intel_pxp_types.h" +#include "intel_pxp_cmd_interface_43.h" + +int intel_pxp_huc_load_and_auth(struct intel_pxp *pxp) +{ + struct intel_gt *gt = pxp_to_gt(pxp); + struct intel_huc *huc = >->uc.huc; + struct pxp43_start_huc_auth_in huc_in = {0}; + struct pxp43_start_huc_auth_out huc_out = {0}; + dma_addr_t huc_phys_addr; + u8 client_id = 0; + u8 fence_id = 0; + int err; + + if (!pxp->pxp_component) + return -ENODEV; + + huc_phys_addr = i915_gem_object_get_dma_address(huc->fw.obj, 0); + + /* write the PXP message into the lmem (the sg list) */ + huc_in.header.api_version = PXP_APIVER(4, 3); + huc_in.header.command_id = PXP43_CMDID_START_HUC_AUTH; + huc_in.header.status = 0; + huc_in.header.buffer_len = sizeof(huc_in.huc_base_address); + huc_in.huc_base_address = huc_phys_addr; + + err = intel_pxp_tee_stream_message(pxp, client_id, fence_id, + &huc_in, sizeof(huc_in), + &huc_out, sizeof(huc_out)); + if (err < 0) { + drm_err(>->i915->drm, + "Failed to send HuC load and auth command to GSC [%d]!\n", + err); + return err; + } + + /* + * HuC does sometimes survive suspend/resume (it depends on how "deep" + * a sleep state the device reaches) so we can end up here on resume + * with HuC already loaded, in which case the GSC will return + * PXP_STATUS_OP_NOT_PERMITTED. We can therefore consider the GuC + * correctly transferred in this scenario; if the same error is ever + * returned with HuC not loaded we'll still catch it when we check the + * authentication bit later. + */ + if (huc_out.header.status != PXP_STATUS_SUCCESS && + huc_out.header.status != PXP_STATUS_OP_NOT_PERMITTED) { + drm_err(>->i915->drm, + "HuC load failed with GSC error = 0x%x\n", + huc_out.header.status); + return -EPROTO; + } + + return 0; +} diff --git a/drivers/gpu/drm/i915/pxp/intel_pxp_huc.h b/drivers/gpu/drm/i915/pxp/intel_pxp_huc.h new file mode 100644 index 000000000000..e40847a91c39 --- /dev/null +++ b/drivers/gpu/drm/i915/pxp/intel_pxp_huc.h @@ -0,0 +1,13 @@ +/* SPDX-License-Identifier: MIT */ +/* + * Copyright(c) 2021-2022, Intel Corporation. All rights reserved. + */ + +#ifndef __INTEL_PXP_HUC_H__ +#define __INTEL_PXP_HUC_H__ + +struct intel_pxp; + +int intel_pxp_huc_load_and_auth(struct intel_pxp *pxp); + +#endif /* __INTEL_PXP_HUC_H__ */ diff --git a/drivers/gpu/drm/i915/pxp/intel_pxp_irq.h b/drivers/gpu/drm/i915/pxp/intel_pxp_irq.h index 8b5793654844..8c292dc86f68 100644 --- a/drivers/gpu/drm/i915/pxp/intel_pxp_irq.h +++ b/drivers/gpu/drm/i915/pxp/intel_pxp_irq.h @@ -27,6 +27,14 @@ void intel_pxp_irq_handler(struct intel_pxp *pxp, u16 iir); static inline void intel_pxp_irq_handler(struct intel_pxp *pxp, u16 iir) { } + +static inline void intel_pxp_irq_enable(struct intel_pxp *pxp) +{ +} + +static inline void intel_pxp_irq_disable(struct intel_pxp *pxp) +{ +} #endif #endif /* __INTEL_PXP_IRQ_H__ */ diff --git a/drivers/gpu/drm/i915/pxp/intel_pxp_session.c b/drivers/gpu/drm/i915/pxp/intel_pxp_session.c index 1bb5b5249157..85572360c71a 100644 --- a/drivers/gpu/drm/i915/pxp/intel_pxp_session.c +++ b/drivers/gpu/drm/i915/pxp/intel_pxp_session.c @@ -77,6 +77,7 @@ static int pxp_create_arb_session(struct intel_pxp *pxp) drm_err(>->i915->drm, "arb session failed to go in play\n"); return ret; } + drm_dbg(>->i915->drm, "PXP ARB session is alive\n"); if (!++pxp->key_instance) ++pxp->key_instance; @@ -137,7 +138,7 @@ static void pxp_terminate_complete(struct intel_pxp *pxp) complete_all(&pxp->termination); } -void intel_pxp_session_work(struct work_struct *work) +static void pxp_session_work(struct work_struct *work) { struct intel_pxp *pxp = container_of(work, typeof(*pxp), session_work); struct intel_gt *gt = pxp_to_gt(pxp); @@ -172,3 +173,9 @@ void intel_pxp_session_work(struct work_struct *work) intel_runtime_pm_put(gt->uncore->rpm, wakeref); } + +void intel_pxp_session_management_init(struct intel_pxp *pxp) +{ + mutex_init(&pxp->arb_mutex); + INIT_WORK(&pxp->session_work, pxp_session_work); +} diff --git a/drivers/gpu/drm/i915/pxp/intel_pxp_session.h b/drivers/gpu/drm/i915/pxp/intel_pxp_session.h index ba4c9d2b94b7..903ac52cffa1 100644 --- a/drivers/gpu/drm/i915/pxp/intel_pxp_session.h +++ b/drivers/gpu/drm/i915/pxp/intel_pxp_session.h @@ -8,8 +8,13 @@ #include <linux/types.h> -struct work_struct; - -void intel_pxp_session_work(struct work_struct *work); +struct intel_pxp; +#ifdef CONFIG_DRM_I915_PXP +void intel_pxp_session_management_init(struct intel_pxp *pxp); +#else +static inline void intel_pxp_session_management_init(struct intel_pxp *pxp) +{ +} +#endif #endif /* __INTEL_PXP_SESSION_H__ */ diff --git a/drivers/gpu/drm/i915/pxp/intel_pxp_tee.c b/drivers/gpu/drm/i915/pxp/intel_pxp_tee.c index 4b6f5655fab5..b0c9170b1395 100644 --- a/drivers/gpu/drm/i915/pxp/intel_pxp_tee.c +++ b/drivers/gpu/drm/i915/pxp/intel_pxp_tee.c @@ -8,11 +8,14 @@ #include <drm/i915_pxp_tee_interface.h> #include <drm/i915_component.h> +#include "gem/i915_gem_lmem.h" + #include "i915_drv.h" #include "intel_pxp.h" #include "intel_pxp_session.h" #include "intel_pxp_tee.h" -#include "intel_pxp_tee_interface.h" +#include "intel_pxp_cmd_interface_42.h" +#include "intel_pxp_huc.h" static inline struct intel_pxp *i915_dev_to_pxp(struct device *i915_kdev) { @@ -69,6 +72,47 @@ unlock: return ret; } +int intel_pxp_tee_stream_message(struct intel_pxp *pxp, + u8 client_id, u32 fence_id, + void *msg_in, size_t msg_in_len, + void *msg_out, size_t msg_out_len) +{ + /* TODO: for bigger objects we need to use a sg of 4k pages */ + const size_t max_msg_size = PAGE_SIZE; + struct drm_i915_private *i915 = pxp_to_gt(pxp)->i915; + struct i915_pxp_component *pxp_component = pxp->pxp_component; + unsigned int offset = 0; + struct scatterlist *sg; + int ret; + + if (msg_in_len > max_msg_size || msg_out_len > max_msg_size) + return -ENOSPC; + + mutex_lock(&pxp->tee_mutex); + + if (unlikely(!pxp_component || !pxp_component->ops->gsc_command)) { + ret = -ENODEV; + goto unlock; + } + + GEM_BUG_ON(!pxp->stream_cmd.obj); + + sg = i915_gem_object_get_sg_dma(pxp->stream_cmd.obj, 0, &offset); + + memcpy(pxp->stream_cmd.vaddr, msg_in, msg_in_len); + + ret = pxp_component->ops->gsc_command(pxp_component->tee_dev, client_id, + fence_id, sg, msg_in_len, sg); + if (ret < 0) + drm_err(&i915->drm, "Failed to send PXP TEE gsc command\n"); + else + memcpy(msg_out, pxp->stream_cmd.vaddr, msg_out_len); + +unlock: + mutex_unlock(&pxp->tee_mutex); + return ret; +} + /** * i915_pxp_tee_component_bind - bind function to pass the function pointers to pxp_tee * @i915_kdev: pointer to i915 kernel device @@ -84,24 +128,36 @@ static int i915_pxp_tee_component_bind(struct device *i915_kdev, { struct drm_i915_private *i915 = kdev_to_i915(i915_kdev); struct intel_pxp *pxp = i915_dev_to_pxp(i915_kdev); + struct intel_uc *uc = &pxp_to_gt(pxp)->uc; intel_wakeref_t wakeref; + int ret = 0; mutex_lock(&pxp->tee_mutex); pxp->pxp_component = data; pxp->pxp_component->tee_dev = tee_kdev; mutex_unlock(&pxp->tee_mutex); + if (intel_uc_uses_huc(uc) && intel_huc_is_loaded_by_gsc(&uc->huc)) { + with_intel_runtime_pm(&i915->runtime_pm, wakeref) { + /* load huc via pxp */ + ret = intel_huc_fw_load_and_auth_via_gsc(&uc->huc); + if (ret < 0) + drm_err(&i915->drm, "failed to load huc via gsc %d\n", ret); + } + } + /* if we are suspended, the HW will be re-initialized on resume */ wakeref = intel_runtime_pm_get_if_in_use(&i915->runtime_pm); if (!wakeref) return 0; /* the component is required to fully start the PXP HW */ - intel_pxp_init_hw(pxp); + if (intel_pxp_is_enabled(pxp)) + intel_pxp_init_hw(pxp); intel_runtime_pm_put(&i915->runtime_pm, wakeref); - return 0; + return ret; } static void i915_pxp_tee_component_unbind(struct device *i915_kdev, @@ -111,8 +167,9 @@ static void i915_pxp_tee_component_unbind(struct device *i915_kdev, struct intel_pxp *pxp = i915_dev_to_pxp(i915_kdev); intel_wakeref_t wakeref; - with_intel_runtime_pm_if_in_use(&i915->runtime_pm, wakeref) - intel_pxp_fini_hw(pxp); + if (intel_pxp_is_enabled(pxp)) + with_intel_runtime_pm_if_in_use(&i915->runtime_pm, wakeref) + intel_pxp_fini_hw(pxp); mutex_lock(&pxp->tee_mutex); pxp->pxp_component = NULL; @@ -124,22 +181,92 @@ static const struct component_ops i915_pxp_tee_component_ops = { .unbind = i915_pxp_tee_component_unbind, }; +static int alloc_streaming_command(struct intel_pxp *pxp) +{ + struct drm_i915_private *i915 = pxp_to_gt(pxp)->i915; + struct drm_i915_gem_object *obj = NULL; + void *cmd; + int err; + + pxp->stream_cmd.obj = NULL; + pxp->stream_cmd.vaddr = NULL; + + if (!IS_DGFX(i915)) + return 0; + + /* allocate lmem object of one page for PXP command memory and store it */ + obj = i915_gem_object_create_lmem(i915, PAGE_SIZE, I915_BO_ALLOC_CONTIGUOUS); + if (IS_ERR(obj)) { + drm_err(&i915->drm, "Failed to allocate pxp streaming command!\n"); + return PTR_ERR(obj); + } + + err = i915_gem_object_pin_pages_unlocked(obj); + if (err) { + drm_err(&i915->drm, "Failed to pin gsc message page!\n"); + goto out_put; + } + + /* map the lmem into the virtual memory pointer */ + cmd = i915_gem_object_pin_map_unlocked(obj, i915_coherent_map_type(i915, obj, true)); + if (IS_ERR(cmd)) { + drm_err(&i915->drm, "Failed to map gsc message page!\n"); + err = PTR_ERR(cmd); + goto out_unpin; + } + + memset(cmd, 0, obj->base.size); + + pxp->stream_cmd.obj = obj; + pxp->stream_cmd.vaddr = cmd; + + return 0; + +out_unpin: + i915_gem_object_unpin_pages(obj); +out_put: + i915_gem_object_put(obj); + return err; +} + +static void free_streaming_command(struct intel_pxp *pxp) +{ + struct drm_i915_gem_object *obj = fetch_and_zero(&pxp->stream_cmd.obj); + + if (!obj) + return; + + i915_gem_object_unpin_map(obj); + i915_gem_object_unpin_pages(obj); + i915_gem_object_put(obj); +} + int intel_pxp_tee_component_init(struct intel_pxp *pxp) { int ret; struct intel_gt *gt = pxp_to_gt(pxp); struct drm_i915_private *i915 = gt->i915; + mutex_init(&pxp->tee_mutex); + + ret = alloc_streaming_command(pxp); + if (ret) + return ret; + ret = component_add_typed(i915->drm.dev, &i915_pxp_tee_component_ops, I915_COMPONENT_PXP); if (ret < 0) { drm_err(&i915->drm, "Failed to add PXP component (%d)\n", ret); - return ret; + goto out_free; } pxp->pxp_component_added = true; return 0; + +out_free: + free_streaming_command(pxp); + return ret; } void intel_pxp_tee_component_fini(struct intel_pxp *pxp) @@ -151,20 +278,22 @@ void intel_pxp_tee_component_fini(struct intel_pxp *pxp) component_del(i915->drm.dev, &i915_pxp_tee_component_ops); pxp->pxp_component_added = false; + + free_streaming_command(pxp); } int intel_pxp_tee_cmd_create_arb_session(struct intel_pxp *pxp, int arb_session_id) { struct drm_i915_private *i915 = pxp_to_gt(pxp)->i915; - struct pxp_tee_create_arb_in msg_in = {0}; - struct pxp_tee_create_arb_out msg_out = {0}; + struct pxp42_create_arb_in msg_in = {0}; + struct pxp42_create_arb_out msg_out = {0}; int ret; - msg_in.header.api_version = PXP_TEE_APIVER; - msg_in.header.command_id = PXP_TEE_ARB_CMDID; + msg_in.header.api_version = PXP_APIVER(4, 2); + msg_in.header.command_id = PXP42_CMDID_INIT_SESSION; msg_in.header.buffer_len = sizeof(msg_in) - sizeof(msg_in.header); - msg_in.protection_mode = PXP_TEE_ARB_PROTECTION_MODE; + msg_in.protection_mode = PXP42_ARB_SESSION_MODE_HEAVY; msg_in.session_id = arb_session_id; ret = intel_pxp_tee_io_message(pxp, @@ -174,6 +303,9 @@ int intel_pxp_tee_cmd_create_arb_session(struct intel_pxp *pxp, if (ret) drm_err(&i915->drm, "Failed to send tee msg ret=[%d]\n", ret); + else if (msg_out.header.status != 0x0) + drm_warn(&i915->drm, "PXP firmware failed arb session init request ret=[0x%08x]\n", + msg_out.header.status); return ret; } diff --git a/drivers/gpu/drm/i915/pxp/intel_pxp_tee.h b/drivers/gpu/drm/i915/pxp/intel_pxp_tee.h index c136053ce340..aeb3dfe7ce96 100644 --- a/drivers/gpu/drm/i915/pxp/intel_pxp_tee.h +++ b/drivers/gpu/drm/i915/pxp/intel_pxp_tee.h @@ -14,4 +14,9 @@ void intel_pxp_tee_component_fini(struct intel_pxp *pxp); int intel_pxp_tee_cmd_create_arb_session(struct intel_pxp *pxp, int arb_session_id); +int intel_pxp_tee_stream_message(struct intel_pxp *pxp, + u8 client_id, u32 fence_id, + void *msg_in, size_t msg_in_len, + void *msg_out, size_t msg_out_len); + #endif /* __INTEL_PXP_TEE_H__ */ diff --git a/drivers/gpu/drm/i915/pxp/intel_pxp_tee_interface.h b/drivers/gpu/drm/i915/pxp/intel_pxp_tee_interface.h deleted file mode 100644 index 36e9b0868f5c..000000000000 --- a/drivers/gpu/drm/i915/pxp/intel_pxp_tee_interface.h +++ /dev/null @@ -1,36 +0,0 @@ -/* SPDX-License-Identifier: MIT */ -/* - * Copyright(c) 2020, Intel Corporation. All rights reserved. - */ - -#ifndef __INTEL_PXP_TEE_INTERFACE_H__ -#define __INTEL_PXP_TEE_INTERFACE_H__ - -#include <linux/types.h> - -#define PXP_TEE_APIVER 0x40002 -#define PXP_TEE_ARB_CMDID 0x1e -#define PXP_TEE_ARB_PROTECTION_MODE 0x2 - -/* PXP TEE message header */ -struct pxp_tee_cmd_header { - u32 api_version; - u32 command_id; - u32 status; - /* Length of the message (excluding the header) */ - u32 buffer_len; -} __packed; - -/* PXP TEE message input to create a arbitrary session */ -struct pxp_tee_create_arb_in { - struct pxp_tee_cmd_header header; - u32 protection_mode; - u32 session_id; -} __packed; - -/* PXP TEE message output to create a arbitrary session */ -struct pxp_tee_create_arb_out { - struct pxp_tee_cmd_header header; -} __packed; - -#endif /* __INTEL_PXP_TEE_INTERFACE_H__ */ diff --git a/drivers/gpu/drm/i915/pxp/intel_pxp_types.h b/drivers/gpu/drm/i915/pxp/intel_pxp_types.h index 7ce5f37ee12e..f74b1e11a505 100644 --- a/drivers/gpu/drm/i915/pxp/intel_pxp_types.h +++ b/drivers/gpu/drm/i915/pxp/intel_pxp_types.h @@ -53,6 +53,12 @@ struct intel_pxp { /** @tee_mutex: protects the tee channel binding and messaging. */ struct mutex tee_mutex; + /** @stream_cmd: LMEM obj used to send stream PXP commands to the GSC */ + struct { + struct drm_i915_gem_object *obj; /* contains PXP command memory */ + void *vaddr; /* virtual memory for PXP command */ + } stream_cmd; + /** * @hw_state_invalidated: if the HW perceives an attack on the integrity * of the encryption it will invalidate the keys and expect SW to diff --git a/drivers/gpu/drm/i915/selftests/i915_gem_gtt.c b/drivers/gpu/drm/i915/selftests/i915_gem_gtt.c index e050a2de5fd1..eae7d947d7de 100644 --- a/drivers/gpu/drm/i915/selftests/i915_gem_gtt.c +++ b/drivers/gpu/drm/i915/selftests/i915_gem_gtt.c @@ -27,6 +27,7 @@ #include "gem/i915_gem_context.h" #include "gem/i915_gem_internal.h" +#include "gem/i915_gem_lmem.h" #include "gem/i915_gem_region.h" #include "gem/selftests/mock_context.h" #include "gt/intel_context.h" @@ -60,7 +61,6 @@ static int fake_get_pages(struct drm_i915_gem_object *obj) #define PFN_BIAS 0x1000 struct sg_table *pages; struct scatterlist *sg; - unsigned int sg_page_sizes; typeof(obj->base.size) rem; pages = kmalloc(sizeof(*pages), GFP); @@ -73,7 +73,6 @@ static int fake_get_pages(struct drm_i915_gem_object *obj) return -ENOMEM; } - sg_page_sizes = 0; rem = obj->base.size; for (sg = pages->sgl; sg; sg = sg_next(sg)) { unsigned long len = min_t(typeof(rem), rem, BIT(31)); @@ -82,13 +81,12 @@ static int fake_get_pages(struct drm_i915_gem_object *obj) sg_set_page(sg, pfn_to_page(PFN_BIAS), len, 0); sg_dma_address(sg) = page_to_phys(sg_page(sg)); sg_dma_len(sg) = len; - sg_page_sizes |= len; rem -= len; } GEM_BUG_ON(rem); - __i915_gem_object_set_pages(obj, pages, sg_page_sizes); + __i915_gem_object_set_pages(obj, pages); return 0; #undef GFP @@ -1113,15 +1111,8 @@ static int misaligned_case(struct i915_address_space *vm, struct intel_memory_re expected_node_size = expected_vma_size; if (HAS_64K_PAGES(vm->i915) && i915_gem_object_is_lmem(obj)) { - /* - * The compact-pt should expand lmem node to 2MB for the ppGTT, - * for all other cases we should only expect 64K. - */ expected_vma_size = round_up(size, I915_GTT_PAGE_SIZE_64K); - if (NEEDS_COMPACT_PT(vm->i915) && !i915_is_ggtt(vm)) - expected_node_size = round_up(size, I915_GTT_PAGE_SIZE_2M); - else - expected_node_size = round_up(size, I915_GTT_PAGE_SIZE_64K); + expected_node_size = round_up(size, I915_GTT_PAGE_SIZE_64K); } if (vma->size != expected_vma_size || vma->node.size != expected_node_size) { diff --git a/drivers/gpu/drm/i915/selftests/i915_perf.c b/drivers/gpu/drm/i915/selftests/i915_perf.c index 429c6d73b159..24dde5531423 100644 --- a/drivers/gpu/drm/i915/selftests/i915_perf.c +++ b/drivers/gpu/drm/i915/selftests/i915_perf.c @@ -102,6 +102,12 @@ test_stream(struct i915_perf *perf) I915_OA_FORMAT_A32u40_A4u32_B8_C8 : I915_OA_FORMAT_C4_B8, }; struct i915_perf_stream *stream; + struct intel_gt *gt; + + if (!props.engine) + return NULL; + + gt = props.engine->gt; if (!oa_config) return NULL; @@ -116,12 +122,12 @@ test_stream(struct i915_perf *perf) stream->perf = perf; - mutex_lock(&perf->lock); + mutex_lock(>->perf.lock); if (i915_oa_stream_init(stream, ¶m, &props)) { kfree(stream); stream = NULL; } - mutex_unlock(&perf->lock); + mutex_unlock(>->perf.lock); i915_oa_config_put(oa_config); @@ -130,11 +136,11 @@ test_stream(struct i915_perf *perf) static void stream_destroy(struct i915_perf_stream *stream) { - struct i915_perf *perf = stream->perf; + struct intel_gt *gt = stream->engine->gt; - mutex_lock(&perf->lock); + mutex_lock(>->perf.lock); i915_perf_destroy_locked(stream); - mutex_unlock(&perf->lock); + mutex_unlock(>->perf.lock); } static int live_sanitycheck(void *arg) diff --git a/drivers/gpu/drm/i915/selftests/i915_request.c b/drivers/gpu/drm/i915/selftests/i915_request.c index 818a4909c1f3..0daa8669181d 100644 --- a/drivers/gpu/drm/i915/selftests/i915_request.c +++ b/drivers/gpu/drm/i915/selftests/i915_request.c @@ -299,9 +299,18 @@ __live_request_alloc(struct intel_context *ce) return intel_context_create_request(ce); } -static int __igt_breadcrumbs_smoketest(void *arg) +struct smoke_thread { + struct kthread_worker *worker; + struct kthread_work work; + struct smoketest *t; + bool stop; + int result; +}; + +static void __igt_breadcrumbs_smoketest(struct kthread_work *work) { - struct smoketest *t = arg; + struct smoke_thread *thread = container_of(work, typeof(*thread), work); + struct smoketest *t = thread->t; const unsigned int max_batch = min(t->ncontexts, t->max_batch) - 1; const unsigned int total = 4 * t->ncontexts + 1; unsigned int num_waits = 0, num_fences = 0; @@ -320,8 +329,10 @@ static int __igt_breadcrumbs_smoketest(void *arg) */ requests = kcalloc(total, sizeof(*requests), GFP_KERNEL); - if (!requests) - return -ENOMEM; + if (!requests) { + thread->result = -ENOMEM; + return; + } order = i915_random_order(total, &prng); if (!order) { @@ -329,7 +340,7 @@ static int __igt_breadcrumbs_smoketest(void *arg) goto out_requests; } - while (!kthread_should_stop()) { + while (!READ_ONCE(thread->stop)) { struct i915_sw_fence *submit, *wait; unsigned int n, count; @@ -437,7 +448,7 @@ static int __igt_breadcrumbs_smoketest(void *arg) kfree(order); out_requests: kfree(requests); - return err; + thread->result = err; } static int mock_breadcrumbs_smoketest(void *arg) @@ -450,7 +461,7 @@ static int mock_breadcrumbs_smoketest(void *arg) .request_alloc = __mock_request_alloc }; unsigned int ncpus = num_online_cpus(); - struct task_struct **threads; + struct smoke_thread *threads; unsigned int n; int ret = 0; @@ -479,28 +490,37 @@ static int mock_breadcrumbs_smoketest(void *arg) } for (n = 0; n < ncpus; n++) { - threads[n] = kthread_run(__igt_breadcrumbs_smoketest, - &t, "igt/%d", n); - if (IS_ERR(threads[n])) { - ret = PTR_ERR(threads[n]); + struct kthread_worker *worker; + + worker = kthread_create_worker(0, "igt/%d", n); + if (IS_ERR(worker)) { + ret = PTR_ERR(worker); ncpus = n; break; } - get_task_struct(threads[n]); + threads[n].worker = worker; + threads[n].t = &t; + threads[n].stop = false; + threads[n].result = 0; + + kthread_init_work(&threads[n].work, + __igt_breadcrumbs_smoketest); + kthread_queue_work(worker, &threads[n].work); } - yield(); /* start all threads before we begin */ msleep(jiffies_to_msecs(i915_selftest.timeout_jiffies)); for (n = 0; n < ncpus; n++) { int err; - err = kthread_stop(threads[n]); + WRITE_ONCE(threads[n].stop, true); + kthread_flush_work(&threads[n].work); + err = READ_ONCE(threads[n].result); if (err < 0 && !ret) ret = err; - put_task_struct(threads[n]); + kthread_destroy_worker(threads[n].worker); } pr_info("Completed %lu waits for %lu fence across %d cpus\n", atomic_long_read(&t.num_waits), @@ -1203,9 +1223,7 @@ static int live_all_engines(void *arg) goto out_request; } - err = i915_request_await_object(request[idx], batch->obj, 0); - if (err == 0) - err = i915_vma_move_to_active(batch, request[idx], 0); + err = i915_vma_move_to_active(batch, request[idx], 0); GEM_BUG_ON(err); err = engine->emit_bb_start(request[idx], @@ -1332,10 +1350,7 @@ static int live_sequential_engines(void *arg) } } - err = i915_request_await_object(request[idx], - batch->obj, false); - if (err == 0) - err = i915_vma_move_to_active(batch, request[idx], 0); + err = i915_vma_move_to_active(batch, request[idx], 0); GEM_BUG_ON(err); err = engine->emit_bb_start(request[idx], @@ -1419,9 +1434,18 @@ out_free: return err; } -static int __live_parallel_engine1(void *arg) +struct parallel_thread { + struct kthread_worker *worker; + struct kthread_work work; + struct intel_engine_cs *engine; + int result; +}; + +static void __live_parallel_engine1(struct kthread_work *work) { - struct intel_engine_cs *engine = arg; + struct parallel_thread *thread = + container_of(work, typeof(*thread), work); + struct intel_engine_cs *engine = thread->engine; IGT_TIMEOUT(end_time); unsigned long count; int err = 0; @@ -1452,12 +1476,14 @@ static int __live_parallel_engine1(void *arg) intel_engine_pm_put(engine); pr_info("%s: %lu request + sync\n", engine->name, count); - return err; + thread->result = err; } -static int __live_parallel_engineN(void *arg) +static void __live_parallel_engineN(struct kthread_work *work) { - struct intel_engine_cs *engine = arg; + struct parallel_thread *thread = + container_of(work, typeof(*thread), work); + struct intel_engine_cs *engine = thread->engine; IGT_TIMEOUT(end_time); unsigned long count; int err = 0; @@ -1479,7 +1505,7 @@ static int __live_parallel_engineN(void *arg) intel_engine_pm_put(engine); pr_info("%s: %lu requests\n", engine->name, count); - return err; + thread->result = err; } static bool wake_all(struct drm_i915_private *i915) @@ -1505,9 +1531,11 @@ static int wait_for_all(struct drm_i915_private *i915) return -ETIME; } -static int __live_parallel_spin(void *arg) +static void __live_parallel_spin(struct kthread_work *work) { - struct intel_engine_cs *engine = arg; + struct parallel_thread *thread = + container_of(work, typeof(*thread), work); + struct intel_engine_cs *engine = thread->engine; struct igt_spinner spin; struct i915_request *rq; int err = 0; @@ -1520,7 +1548,8 @@ static int __live_parallel_spin(void *arg) if (igt_spinner_init(&spin, engine->gt)) { wake_all(engine->i915); - return -ENOMEM; + thread->result = -ENOMEM; + return; } intel_engine_pm_get(engine); @@ -1553,22 +1582,22 @@ static int __live_parallel_spin(void *arg) out_spin: igt_spinner_fini(&spin); - return err; + thread->result = err; } static int live_parallel_engines(void *arg) { struct drm_i915_private *i915 = arg; - static int (* const func[])(void *arg) = { + static void (* const func[])(struct kthread_work *) = { __live_parallel_engine1, __live_parallel_engineN, __live_parallel_spin, NULL, }; const unsigned int nengines = num_uabi_engines(i915); + struct parallel_thread *threads; struct intel_engine_cs *engine; - int (* const *fn)(void *arg); - struct task_struct **tsk; + void (* const *fn)(struct kthread_work *); int err = 0; /* @@ -1576,8 +1605,8 @@ static int live_parallel_engines(void *arg) * tests that we load up the system maximally. */ - tsk = kcalloc(nengines, sizeof(*tsk), GFP_KERNEL); - if (!tsk) + threads = kcalloc(nengines, sizeof(*threads), GFP_KERNEL); + if (!threads) return -ENOMEM; for (fn = func; !err && *fn; fn++) { @@ -1594,37 +1623,44 @@ static int live_parallel_engines(void *arg) idx = 0; for_each_uabi_engine(engine, i915) { - tsk[idx] = kthread_run(*fn, engine, - "igt/parallel:%s", - engine->name); - if (IS_ERR(tsk[idx])) { - err = PTR_ERR(tsk[idx]); + struct kthread_worker *worker; + + worker = kthread_create_worker(0, "igt/parallel:%s", + engine->name); + if (IS_ERR(worker)) { + err = PTR_ERR(worker); break; } - get_task_struct(tsk[idx++]); - } - yield(); /* start all threads before we kthread_stop() */ + threads[idx].worker = worker; + threads[idx].result = 0; + threads[idx].engine = engine; + + kthread_init_work(&threads[idx].work, *fn); + kthread_queue_work(worker, &threads[idx].work); + idx++; + } idx = 0; for_each_uabi_engine(engine, i915) { int status; - if (IS_ERR(tsk[idx])) + if (!threads[idx].worker) break; - status = kthread_stop(tsk[idx]); + kthread_flush_work(&threads[idx].work); + status = READ_ONCE(threads[idx].result); if (status && !err) err = status; - put_task_struct(tsk[idx++]); + kthread_destroy_worker(threads[idx++].worker); } if (igt_live_test_end(&t)) err = -EIO; } - kfree(tsk); + kfree(threads); return err; } @@ -1669,10 +1705,11 @@ static int live_breadcrumbs_smoketest(void *arg) { struct drm_i915_private *i915 = arg; const unsigned int nengines = num_uabi_engines(i915); - const unsigned int ncpus = num_online_cpus(); + const unsigned int ncpus = /* saturate with nengines * ncpus */ + max_t(int, 2, DIV_ROUND_UP(num_online_cpus(), nengines)); unsigned long num_waits, num_fences; struct intel_engine_cs *engine; - struct task_struct **threads; + struct smoke_thread *threads; struct igt_live_test live; intel_wakeref_t wakeref; struct smoketest *smoke; @@ -1741,28 +1778,31 @@ static int live_breadcrumbs_smoketest(void *arg) goto out_flush; } /* One ring interleaved between requests from all cpus */ - smoke[idx].max_batch /= num_online_cpus() + 1; + smoke[idx].max_batch /= ncpus + 1; pr_debug("Limiting batches to %d requests on %s\n", smoke[idx].max_batch, engine->name); for (n = 0; n < ncpus; n++) { - struct task_struct *tsk; + unsigned int i = idx * ncpus + n; + struct kthread_worker *worker; - tsk = kthread_run(__igt_breadcrumbs_smoketest, - &smoke[idx], "igt/%d.%d", idx, n); - if (IS_ERR(tsk)) { - ret = PTR_ERR(tsk); + worker = kthread_create_worker(0, "igt/%d.%d", idx, n); + if (IS_ERR(worker)) { + ret = PTR_ERR(worker); goto out_flush; } - get_task_struct(tsk); - threads[idx * ncpus + n] = tsk; + threads[i].worker = worker; + threads[i].t = &smoke[idx]; + + kthread_init_work(&threads[i].work, + __igt_breadcrumbs_smoketest); + kthread_queue_work(worker, &threads[i].work); } idx++; } - yield(); /* start all threads before we begin */ msleep(jiffies_to_msecs(i915_selftest.timeout_jiffies)); out_flush: @@ -1771,17 +1811,19 @@ out_flush: num_fences = 0; for_each_uabi_engine(engine, i915) { for (n = 0; n < ncpus; n++) { - struct task_struct *tsk = threads[idx * ncpus + n]; + unsigned int i = idx * ncpus + n; int err; - if (!tsk) + if (!threads[i].worker) continue; - err = kthread_stop(tsk); + WRITE_ONCE(threads[i].stop, true); + kthread_flush_work(&threads[i].work); + err = READ_ONCE(threads[i].result); if (err < 0 && !ret) ret = err; - put_task_struct(tsk); + kthread_destroy_worker(threads[i].worker); } num_waits += atomic_long_read(&smoke[idx].num_waits); @@ -2891,9 +2933,18 @@ out: return err; } -static int p_sync0(void *arg) +struct p_thread { + struct perf_stats p; + struct kthread_worker *worker; + struct kthread_work work; + struct intel_engine_cs *engine; + int result; +}; + +static void p_sync0(struct kthread_work *work) { - struct perf_stats *p = arg; + struct p_thread *thread = container_of(work, typeof(*thread), work); + struct perf_stats *p = &thread->p; struct intel_engine_cs *engine = p->engine; struct intel_context *ce; IGT_TIMEOUT(end_time); @@ -2902,13 +2953,16 @@ static int p_sync0(void *arg) int err = 0; ce = intel_context_create(engine); - if (IS_ERR(ce)) - return PTR_ERR(ce); + if (IS_ERR(ce)) { + thread->result = PTR_ERR(ce); + return; + } err = intel_context_pin(ce); if (err) { intel_context_put(ce); - return err; + thread->result = err; + return; } if (intel_engine_supports_stats(engine)) { @@ -2958,12 +3012,13 @@ static int p_sync0(void *arg) intel_context_unpin(ce); intel_context_put(ce); - return err; + thread->result = err; } -static int p_sync1(void *arg) +static void p_sync1(struct kthread_work *work) { - struct perf_stats *p = arg; + struct p_thread *thread = container_of(work, typeof(*thread), work); + struct perf_stats *p = &thread->p; struct intel_engine_cs *engine = p->engine; struct i915_request *prev = NULL; struct intel_context *ce; @@ -2973,13 +3028,16 @@ static int p_sync1(void *arg) int err = 0; ce = intel_context_create(engine); - if (IS_ERR(ce)) - return PTR_ERR(ce); + if (IS_ERR(ce)) { + thread->result = PTR_ERR(ce); + return; + } err = intel_context_pin(ce); if (err) { intel_context_put(ce); - return err; + thread->result = err; + return; } if (intel_engine_supports_stats(engine)) { @@ -3031,12 +3089,13 @@ static int p_sync1(void *arg) intel_context_unpin(ce); intel_context_put(ce); - return err; + thread->result = err; } -static int p_many(void *arg) +static void p_many(struct kthread_work *work) { - struct perf_stats *p = arg; + struct p_thread *thread = container_of(work, typeof(*thread), work); + struct perf_stats *p = &thread->p; struct intel_engine_cs *engine = p->engine; struct intel_context *ce; IGT_TIMEOUT(end_time); @@ -3045,13 +3104,16 @@ static int p_many(void *arg) bool busy; ce = intel_context_create(engine); - if (IS_ERR(ce)) - return PTR_ERR(ce); + if (IS_ERR(ce)) { + thread->result = PTR_ERR(ce); + return; + } err = intel_context_pin(ce); if (err) { intel_context_put(ce); - return err; + thread->result = err; + return; } if (intel_engine_supports_stats(engine)) { @@ -3092,26 +3154,23 @@ static int p_many(void *arg) intel_context_unpin(ce); intel_context_put(ce); - return err; + thread->result = err; } static int perf_parallel_engines(void *arg) { struct drm_i915_private *i915 = arg; - static int (* const func[])(void *arg) = { + static void (* const func[])(struct kthread_work *) = { p_sync0, p_sync1, p_many, NULL, }; const unsigned int nengines = num_uabi_engines(i915); + void (* const *fn)(struct kthread_work *); struct intel_engine_cs *engine; - int (* const *fn)(void *arg); struct pm_qos_request qos; - struct { - struct perf_stats p; - struct task_struct *tsk; - } *engines; + struct p_thread *engines; int err = 0; engines = kcalloc(nengines, sizeof(*engines), GFP_KERNEL); @@ -3134,36 +3193,45 @@ static int perf_parallel_engines(void *arg) idx = 0; for_each_uabi_engine(engine, i915) { + struct kthread_worker *worker; + intel_engine_pm_get(engine); memset(&engines[idx].p, 0, sizeof(engines[idx].p)); - engines[idx].p.engine = engine; - engines[idx].tsk = kthread_run(*fn, &engines[idx].p, - "igt:%s", engine->name); - if (IS_ERR(engines[idx].tsk)) { - err = PTR_ERR(engines[idx].tsk); + worker = kthread_create_worker(0, "igt:%s", + engine->name); + if (IS_ERR(worker)) { + err = PTR_ERR(worker); intel_engine_pm_put(engine); break; } - get_task_struct(engines[idx++].tsk); - } + engines[idx].worker = worker; + engines[idx].result = 0; + engines[idx].p.engine = engine; + engines[idx].engine = engine; - yield(); /* start all threads before we kthread_stop() */ + kthread_init_work(&engines[idx].work, *fn); + kthread_queue_work(worker, &engines[idx].work); + idx++; + } idx = 0; for_each_uabi_engine(engine, i915) { int status; - if (IS_ERR(engines[idx].tsk)) + if (!engines[idx].worker) break; - status = kthread_stop(engines[idx].tsk); + kthread_flush_work(&engines[idx].work); + status = READ_ONCE(engines[idx].result); if (status && !err) err = status; intel_engine_pm_put(engine); - put_task_struct(engines[idx++].tsk); + + kthread_destroy_worker(engines[idx].worker); + idx++; } if (igt_live_test_end(&t)) diff --git a/drivers/gpu/drm/i915/selftests/igt_spinner.c b/drivers/gpu/drm/i915/selftests/igt_spinner.c index 0c22594ae274..16978ac59797 100644 --- a/drivers/gpu/drm/i915/selftests/igt_spinner.c +++ b/drivers/gpu/drm/i915/selftests/igt_spinner.c @@ -119,22 +119,6 @@ static u64 hws_address(const struct i915_vma *hws, return hws->node.start + seqno_offset(rq->fence.context); } -static int move_to_active(struct i915_vma *vma, - struct i915_request *rq, - unsigned int flags) -{ - int err; - - i915_vma_lock(vma); - err = i915_request_await_object(rq, vma->obj, - flags & EXEC_OBJECT_WRITE); - if (err == 0) - err = i915_vma_move_to_active(vma, rq, flags); - i915_vma_unlock(vma); - - return err; -} - struct i915_request * igt_spinner_create_request(struct igt_spinner *spin, struct intel_context *ce, @@ -165,11 +149,11 @@ igt_spinner_create_request(struct igt_spinner *spin, if (IS_ERR(rq)) return ERR_CAST(rq); - err = move_to_active(vma, rq, 0); + err = igt_vma_move_to_active_unlocked(vma, rq, 0); if (err) goto cancel_rq; - err = move_to_active(hws, rq, 0); + err = igt_vma_move_to_active_unlocked(hws, rq, 0); if (err) goto cancel_rq; diff --git a/drivers/gpu/drm/i915/selftests/intel_uncore.c b/drivers/gpu/drm/i915/selftests/intel_uncore.c index fda9bb79c049..e4281508d580 100644 --- a/drivers/gpu/drm/i915/selftests/intel_uncore.c +++ b/drivers/gpu/drm/i915/selftests/intel_uncore.c @@ -70,6 +70,8 @@ static int intel_shadow_table_check(void) { gen12_shadowed_regs, ARRAY_SIZE(gen12_shadowed_regs) }, { dg2_shadowed_regs, ARRAY_SIZE(dg2_shadowed_regs) }, { pvc_shadowed_regs, ARRAY_SIZE(pvc_shadowed_regs) }, + { mtl_shadowed_regs, ARRAY_SIZE(mtl_shadowed_regs) }, + { xelpmp_shadowed_regs, ARRAY_SIZE(xelpmp_shadowed_regs) }, }; const struct i915_range *range; unsigned int i, j; @@ -117,6 +119,8 @@ int intel_uncore_mock_selftests(void) { __gen12_fw_ranges, ARRAY_SIZE(__gen12_fw_ranges), true }, { __xehp_fw_ranges, ARRAY_SIZE(__xehp_fw_ranges), true }, { __pvc_fw_ranges, ARRAY_SIZE(__pvc_fw_ranges), true }, + { __mtl_fw_ranges, ARRAY_SIZE(__mtl_fw_ranges), true }, + { __xelpmp_fw_ranges, ARRAY_SIZE(__xelpmp_fw_ranges), true }, }; int err, i; diff --git a/drivers/gpu/drm/i915/selftests/mock_gem_device.c b/drivers/gpu/drm/i915/selftests/mock_gem_device.c index fff11c90f1fa..f6a7c0bd2955 100644 --- a/drivers/gpu/drm/i915/selftests/mock_gem_device.c +++ b/drivers/gpu/drm/i915/selftests/mock_gem_device.c @@ -67,7 +67,6 @@ static void mock_device_release(struct drm_device *dev) intel_gt_driver_remove(to_gt(i915)); i915_gem_drain_workqueue(i915); - i915_gem_drain_freed_objects(i915); mock_fini_ggtt(to_gt(i915)->ggtt); destroy_workqueue(i915->wq); diff --git a/drivers/gpu/drm/i915/selftests/mock_region.c b/drivers/gpu/drm/i915/selftests/mock_region.c index bac21fe84ca5..6324eb32d4dd 100644 --- a/drivers/gpu/drm/i915/selftests/mock_region.c +++ b/drivers/gpu/drm/i915/selftests/mock_region.c @@ -41,7 +41,7 @@ static int mock_region_get_pages(struct drm_i915_gem_object *obj) } pages = &obj->mm.rsgt->table; - __i915_gem_object_set_pages(obj, pages, i915_sg_dma_sizes(pages->sgl)); + __i915_gem_object_set_pages(obj, pages); return 0; diff --git a/drivers/gpu/drm/i915/vlv_sideband.c b/drivers/gpu/drm/i915/vlv_sideband.c index c26001300ebd..6eea6e1a99c0 100644 --- a/drivers/gpu/drm/i915/vlv_sideband.c +++ b/drivers/gpu/drm/i915/vlv_sideband.c @@ -8,6 +8,8 @@ #include "i915_reg.h" #include "vlv_sideband.h" +#include "display/intel_dpio_phy.h" + /* * IOSF sideband, see VLV2_SidebandMsg_HAS.docx and * VLV_VLV2_PUNIT_HAS_0.8.docx diff --git a/drivers/gpu/drm/i915/vlv_suspend.c b/drivers/gpu/drm/i915/vlv_suspend.c index 664fde244f59..02e63ed77f60 100644 --- a/drivers/gpu/drm/i915/vlv_suspend.c +++ b/drivers/gpu/drm/i915/vlv_suspend.c @@ -194,7 +194,6 @@ static void vlv_restore_gunit_s0ix_state(struct drm_i915_private *i915) { struct vlv_s0ix_state *s = i915->vlv_s0ix_state; struct intel_uncore *uncore = &i915->uncore; - u32 val; int i; if (!s) @@ -262,15 +261,11 @@ static void vlv_restore_gunit_s0ix_state(struct drm_i915_private *i915) * be restored, as they are used to control the s0ix suspend/resume * sequence by the caller. */ - val = intel_uncore_read(uncore, VLV_GTLC_WAKE_CTRL); - val &= VLV_GTLC_ALLOWWAKEREQ; - val |= s->gtlc_wake_ctrl & ~VLV_GTLC_ALLOWWAKEREQ; - intel_uncore_write(uncore, VLV_GTLC_WAKE_CTRL, val); + intel_uncore_rmw(uncore, VLV_GTLC_WAKE_CTRL, ~VLV_GTLC_ALLOWWAKEREQ, + s->gtlc_wake_ctrl & ~VLV_GTLC_ALLOWWAKEREQ); - val = intel_uncore_read(uncore, VLV_GTLC_SURVIVABILITY_REG); - val &= VLV_GFX_CLK_FORCE_ON_BIT; - val |= s->gtlc_survive & ~VLV_GFX_CLK_FORCE_ON_BIT; - intel_uncore_write(uncore, VLV_GTLC_SURVIVABILITY_REG, val); + intel_uncore_rmw(uncore, VLV_GTLC_SURVIVABILITY_REG, ~VLV_GFX_CLK_FORCE_ON_BIT, + s->gtlc_survive & ~VLV_GFX_CLK_FORCE_ON_BIT); intel_uncore_write(uncore, VLV_PMWGICZ, s->pmwgicz); @@ -308,14 +303,10 @@ static int vlv_wait_for_pw_status(struct drm_i915_private *i915, static int vlv_force_gfx_clock(struct drm_i915_private *i915, bool force_on) { struct intel_uncore *uncore = &i915->uncore; - u32 val; int err; - val = intel_uncore_read(uncore, VLV_GTLC_SURVIVABILITY_REG); - val &= ~VLV_GFX_CLK_FORCE_ON_BIT; - if (force_on) - val |= VLV_GFX_CLK_FORCE_ON_BIT; - intel_uncore_write(uncore, VLV_GTLC_SURVIVABILITY_REG, val); + intel_uncore_rmw(uncore, VLV_GTLC_SURVIVABILITY_REG, VLV_GFX_CLK_FORCE_ON_BIT, + force_on ? VLV_GFX_CLK_FORCE_ON_BIT : 0); if (!force_on) return 0; @@ -340,11 +331,8 @@ static int vlv_allow_gt_wake(struct drm_i915_private *i915, bool allow) u32 val; int err; - val = intel_uncore_read(uncore, VLV_GTLC_WAKE_CTRL); - val &= ~VLV_GTLC_ALLOWWAKEREQ; - if (allow) - val |= VLV_GTLC_ALLOWWAKEREQ; - intel_uncore_write(uncore, VLV_GTLC_WAKE_CTRL, val); + intel_uncore_rmw(uncore, VLV_GTLC_WAKE_CTRL, VLV_GTLC_ALLOWWAKEREQ, + allow ? VLV_GTLC_ALLOWWAKEREQ : 0); intel_uncore_posting_read(uncore, VLV_GTLC_WAKE_CTRL); mask = VLV_GTLC_ALLOWWAKEACK; |