bpf/xdp: non-map redirect can avoid calling ndo_xdp_flush

This is the first real user of the XDP_XMIT_FLUSH flag.

As pointed out many times, XDP_REDIRECT without using BPF maps is
significant slower than the map variant.  This is primary due to the
lack of bulking, as the ndo_xdp_flush operation is required after each
frame (to avoid frames hanging on the egress device).

It is still possible to optimize this case.  Instead of invoking two
NDO indirect calls, which are very expensive with CONFIG_RETPOLINE,
instead instruct ndo_xdp_xmit to flush via XDP_XMIT_FLUSH flag.

Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Acked-by: Song Liu <songliubraving@fb.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>

diff --git a/net/core/filter.c b/net/core/filter.c
index 56e40dafdde7097986989fb06f29cfe8bfceacf6..a72ea9f610100c9b40c1ca48cb20f6c0a68e3dbe 100644 (file)
--- a/net/core/filter.c
+++ b/net/core/filter.c
@@ -3056,10 +3056,9 @@ static int __bpf_tx_xdp(struct net_device *dev,
        if (unlikely(!xdpf))
                return -EOVERFLOW;
 
-       sent = dev->netdev_ops->ndo_xdp_xmit(dev, 1, &xdpf, 0);
+       sent = dev->netdev_ops->ndo_xdp_xmit(dev, 1, &xdpf, XDP_XMIT_FLUSH);
        if (sent <= 0)
                return sent;
-       dev->netdev_ops->ndo_xdp_flush(dev);
        return 0;
 }