On Thu, 2021-09-09 at 16:38 -0300, Jason Gunthorpe wrote:
This addresses Cornelia's remark on the earlier patch that ccw has a confusing lifecycle. While it doesn't seem like the original attempt was functionally wrong, the result can be made better with a lot of further work.
I thought I'd take a stab at seeing how this works with the hardware before looking at the code much. git couldn't apply patches 1, 6, or 9 to 5.15-rc1, but I was able to hand-fit them into place. Shutting down the guest and de-configuring a device ends up bringing my whole system down. I haven't looked at this any further; hopefully something jumps to mind for you:
[ 64.585347] vfio_ccw 0.0.08fe: MDEV: Unregistering [ 64.585357] illegal operation: 0001 ilc:1 [#1] SMP [ 64.585362] Modules linked in: vhost_vsock vmw_vsock_virtio_transport_common vsock vhost [ 64.585364] vfio_ccw_mdev b50bbd4b-eab8-4f8c-9f0c-3cf636f936b9: Relaying device request to user (#0) [ 64.585364] vhost_iotlb lcs ctcm fsm kvm xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp iptable_mangle iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_filter bridge stp llc dm_multipath dm_mod s390_trng eadm_sch zcrypt_cex4 qeth_l2 vfio_ccw mdev vfio_iommu_type1 vfio configfs zram zsmalloc ip_tables x_tables mlx5_core ghash_s390 prng aes_s390 des_s390 libdes sha3_512_s390 sha3_256_s390 sha512_s390 sha256_s390 sha1_s390 sha_common pkey zcrypt rng_core autofs4 [ 64.585392] CPU: 14 PID: 4487 Comm: qemu-system-s39 Kdump: loaded Not tainted 5.15.0-rc1 #1 [ 64.585395] Hardware name: IBM 3906 M05 780 (LPAR) [ 64.585396] Krnl PSW : 0704c00180000000 0000000000000002 (0x2) [ 64.585404] R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:0 PM:0 RI:0 EA:3 [ 64.585407] Krnl GPRS: 0000000000000001 0000000000000000 00000000005f4800 0000000000000004 [ 64.585410] 0000000000000000 0000000000000002 0000000000000000 000002aa3e65085e [ 64.585412] 000000008de09100 0000000000003b6f 000003ff8017fa08 00000000005f4800 [ 64.585413] 0000000081450000 000003ff7c032310 000003ff80179db0 000003800bf53da0 [ 64.585418] Krnl Code:#0000000000000000: 0000 illegal >0000000000000002: 0000 illegal 0000000000000004: 0000 illegal 0000000000000006: 0000 illegal 0000000000000008: 0000 illegal 000000000000000a: 0000 illegal 000000000000000c: 0000 illegal 000000000000000e: 0000 illegal [ 64.585462] Call Trace: [ 64.585464] [<0000000000000002>] 0x2 [ 64.585467] ([<000003ff80179d74>] vfio_ccw_mdev_ioctl+0x84/0x318 [vfio_ccw]) [ 64.585476] [<00000000bb7adda6>] __s390x_sys_ioctl+0xbe/0x100 [ 64.585481] [<00000000bbfbf5e4>] __do_syscall+0x1bc/0x1e8 [ 64.585488] [<00000000bbfcc8d8>] system_call+0x78/0xa0
Eric
Reorganize the driver so that the mdev owns the private memory and controls the lifecycle, not the css_driver. The memory associated with the css_driver lifecycle is only the mdev_parent/mdev_type registration.
Along the way we change when the sch is quiescent or not to be linked to the open/close_device lifetime of the vfio_device, which is sort of what it was tring to do already, just not completely.
The troublesome racey lifecycle of the css_driver callbacks is made clear with simple vfio_device refcounting so a callback is only delivered into a registered vfio_device and has obvious correctness.
Move the only per-css_driver state, the "available instance" counter, into the core code and share that logic with many of the other drivers. The value is kept in the mdev_type memory.
v2:
- Clean up the lifecycle in ccw with 7 new patches
- Rebase
v1: https://lore.kernel.org/all/7-v2-7667f42c9bad+935-vfio3_jgg@nvidia.com
Jason Gunthorpe (9): vfio/ccw: Use functions for alloc/free of the vfio_ccw_private vfio/ccw: Pass vfio_ccw_private not mdev_device to various functions vfio/ccw: Convert to use vfio_register_group_dev() vfio/ccw: Make the FSM complete and synchronize it to the mdev vfio/mdev: Consolidate all the device_api sysfs into the core code vfio/mdev: Add mdev available instance checking to the core vfio/ccw: Remove private->mdev vfio: Export vfio_device_try_get() vfio/ccw: Move the lifecycle of the struct vfio_ccw_private to the mdev
drivers/gpu/drm/i915/gvt/kvmgt.c | 9 +- drivers/s390/cio/vfio_ccw_drv.c | 282 +++++++++++------------- -- drivers/s390/cio/vfio_ccw_fsm.c | 152 ++++++++++---- drivers/s390/cio/vfio_ccw_ops.c | 240 ++++++++++------------ drivers/s390/cio/vfio_ccw_private.h | 42 +++- drivers/s390/crypto/vfio_ap_ops.c | 41 +--- drivers/s390/crypto/vfio_ap_private.h | 2 - drivers/vfio/mdev/mdev_core.c | 13 +- drivers/vfio/mdev/mdev_private.h | 2 + drivers/vfio/mdev/mdev_sysfs.c | 64 +++++- drivers/vfio/vfio.c | 3 +- include/linux/mdev.h | 13 +- include/linux/vfio.h | 1 + samples/vfio-mdev/mbochs.c | 9 +- samples/vfio-mdev/mdpy.c | 31 +-- samples/vfio-mdev/mtty.c | 10 +- 16 files changed, 470 insertions(+), 444 deletions(-)