vfio: Stall vfio_del_group_dev() for container group detach

When the user unbinds the last device of a group from a vfio bus
driver, the devices within that group should be available for other
purposes.  We currently have a race that makes this generally, but
not always true.  The device can be unbound from the vfio bus driver,
but remaining IOMMU context of the group attached to the container
can result in errors as the next driver configures DMA for the device.

Wait for the group to be detached from the IOMMU backend before
allowing the bus driver remove callback to complete.

Signed-off-by: Alex Williamson <alex.williamson@redhat.com>

diff --git a/drivers/vfio/vfio.c b/drivers/vfio/vfio.c
index 4ee4f361fe9fc32b1a68aabe362cd459c5ccabf1..f5a86f651f38e12874069281735169a1be26bede 100644 (file)
--- a/drivers/vfio/vfio.c
+++ b/drivers/vfio/vfio.c
@@ -85,6 +85,7 @@ struct vfio_group {
        struct list_head                unbound_list;
        struct mutex                    unbound_lock;
        atomic_t                        opened;
+       wait_queue_head_t               container_q;
        bool                            noiommu;
        struct kvm                      *kvm;
        struct blocking_notifier_head   notifier;
@@ -338,6 +339,7 @@ static struct vfio_group *vfio_create_group(struct iommu_group *iommu_group)
        mutex_init(&group->unbound_lock);
        atomic_set(&group->container_users, 0);
        atomic_set(&group->opened, 0);
+       init_waitqueue_head(&group->container_q);
        group->iommu_group = iommu_group;
 #ifdef CONFIG_VFIO_NOIOMMU
        group->noiommu = (iommu_group_get_iommudata(iommu_group) == &noiommu);
@@ -994,6 +996,23 @@ void *vfio_del_group_dev(struct device *dev)
                }
        } while (ret <= 0);
 
+       /*
+        * In order to support multiple devices per group, devices can be
+        * plucked from the group while other devices in the group are still
+        * in use.  The container persists with this group and those remaining
+        * devices still attached.  If the user creates an isolation violation
+        * by binding this device to another driver while the group is still in
+        * use, that's their fault.  However, in the case of removing the last,
+        * or potentially the only, device in the group there can be no other
+        * in-use devices in the group.  The user has done their due diligence
+        * and we should lay no claims to those devices.  In order to do that,
+        * we need to make sure the group is detached from the container.
+        * Without this stall, we're potentially racing with a user process
+        * that may attempt to immediately bind this device to another driver.
+        */
+       if (list_empty(&group->device_list))
+               wait_event(group->container_q, !group->container);
+
        vfio_group_put(group);
 
        return device_data;
@@ -1299,6 +1318,7 @@ static void __vfio_group_unset_container(struct vfio_group *group)
                                          group->iommu_group);
 
        group->container = NULL;
+       wake_up(&group->container_q);
        list_del(&group->container_next);
 
        /* Detaching the last group deprivileges a container, remove iommu */