powerpc/powernv/npu: Move tlb flush before launching ATSD

The nest MMU tlb flush needs to happen before the GPU translation
shootdown is launched to avoid the GPU refilling its tlb with stale
nmmu translations prior to the nmmu flush completing.

Fixes: 1ab66d1fbada ("powerpc/powernv: Introduce address translation services for Nvlink2")
Cc: stable@vger.kernel.org # v4.12+
Signed-off-by: Alistair Popple <alistair@popple.id.au>
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>

diff --git a/arch/powerpc/platforms/powernv/npu-dma.c b/arch/powerpc/platforms/powernv/npu-dma.c
index b5d960d6db3d0b18d33273b31ac67e03caebd02b..3d4f879e687cae0c84ea1c9b48e2de7d9ba860c9 100644 (file)
--- a/arch/powerpc/platforms/powernv/npu-dma.c
+++ b/arch/powerpc/platforms/powernv/npu-dma.c
@@ -545,6 +545,12 @@ static void mmio_invalidate(struct npu_context *npu_context, int va,
        struct mmio_atsd_reg mmio_atsd_reg[NV_MAX_NPUS];
        unsigned long pid = npu_context->mm->context.id;
 
+       /*
+        * Unfortunately the nest mmu does not support flushing specific
+        * addresses so we have to flush the whole mm.
+        */
+       flush_tlb_mm(npu_context->mm);
+
        /*
         * Loop over all the NPUs this process is active on and launch
         * an invalidate.
@@ -576,12 +582,6 @@ static void mmio_invalidate(struct npu_context *npu_context, int va,
                }
        }
 
-       /*
-        * Unfortunately the nest mmu does not support flushing specific
-        * addresses so we have to flush the whole mm.
-        */
-       flush_tlb_mm(npu_context->mm);
-
        mmio_invalidate_wait(mmio_atsd_reg, flush);
        if (flush)
                /* Wait for the flush to complete */