1 POWER9 eXternal Interrupt Virtualization Engine (XIVE Gen1)
2 ==========================================================
4 Device types supported:
5 KVM_DEV_TYPE_XIVE POWER9 XIVE Interrupt Controller generation 1
7 This device acts as a VM interrupt controller. It provides the KVM
8 interface to configure the interrupt sources of a VM in the underlying
9 POWER9 XIVE interrupt controller.
11 Only one XIVE instance may be instantiated. A guest XIVE device
12 requires a POWER9 host and the guest OS should have support for the
13 XIVE native exploitation interrupt mode. If not, it should run using
14 the legacy interrupt mode, referred as XICS (POWER7/8).
18 The KVM device exposes different MMIO ranges of the XIVE HW which
19 are required for interrupt management. These are exposed to the
20 guest in VMAs populated with a custom VM fault handler.
22 1. Thread Interrupt Management Area (TIMA)
24 Each thread has an associated Thread Interrupt Management context
25 composed of a set of registers. These registers let the thread
26 handle priority management and interrupt acknowledgment. The most
29 - Interrupt Pending Buffer (IPB)
30 - Current Processor Priority (CPPR)
31 - Notification Source Register (NSR)
33 They are exposed to software in four different pages each proposing
34 a view with a different privilege. The first page is for the
35 physical thread context and the second for the hypervisor. Only the
36 third (operating system) and the fourth (user level) are exposed the
39 2. Event State Buffer (ESB)
41 Each source is associated with an Event State Buffer (ESB) with
42 either a pair of even/odd pair of pages which provides commands to
43 manage the source: to trigger, to EOI, to turn off the source for
46 3. Device pass-through
48 When a device is passed-through into the guest, the source
49 interrupts are from a different HW controller (PHB4) and the ESB
50 pages exposed to the guest should accommadate this change.
52 The passthru_irq helpers, kvmppc_xive_set_mapped() and
53 kvmppc_xive_clr_mapped() are called when the device HW irqs are
54 mapped into or unmapped from the guest IRQ number space. The KVM
55 device extends these helpers to clear the ESB pages of the guest IRQ
56 number being mapped and then lets the VM fault handler repopulate.
57 The handler will insert the ESB page corresponding to the HW
58 interrupt of the device being passed-through or the initial IPI ESB
59 page if the device has being removed.
61 The ESB remapping is fully transparent to the guest and the OS
62 device driver. All handling is done within VFIO and the above
67 1. KVM_DEV_XIVE_GRP_CTRL
68 Provides global controls on the device
70 1.1 KVM_DEV_XIVE_RESET (write only)
71 Resets the interrupt controller configuration for sources and event
72 queues. To be used by kexec and kdump.
75 1.2 KVM_DEV_XIVE_EQ_SYNC (write only)
76 Sync all the sources and queues and mark the EQ pages dirty. This
77 to make sure that a consistent memory state is captured when
81 2. KVM_DEV_XIVE_GRP_SOURCE (write only)
82 Initializes a new source in the XIVE device and mask it.
84 Interrupt source number (64-bit)
85 The kvm_device_attr.addr points to a __u64 value:
86 bits: | 63 .... 2 | 1 | 0
87 values: | unused | level | type
89 - level: assertion level in case of an LSI.
91 -E2BIG: Interrupt source number is out of range
92 -ENOMEM: Could not create a new source block
93 -EFAULT: Invalid user pointer for attr->addr.
94 -ENXIO: Could not allocate underlying HW interrupt
96 3. KVM_DEV_XIVE_GRP_SOURCE_CONFIG (write only)
97 Configures source targeting
99 Interrupt source number (64-bit)
100 The kvm_device_attr.addr points to a __u64 value:
101 bits: | 63 .... 33 | 32 | 31 .. 3 | 2 .. 0
102 values: | eisn | mask | server | priority
103 - priority: 0-7 interrupt priority level
104 - server: CPU number chosen to handle the interrupt
105 - mask: mask flag (unused)
106 - eisn: Effective Interrupt Source Number
108 -ENOENT: Unknown source number
109 -EINVAL: Not initialized source number
110 -EINVAL: Invalid priority
111 -EINVAL: Invalid CPU number.
112 -EFAULT: Invalid user pointer for attr->addr.
113 -ENXIO: CPU event queues not configured or configuration of the
114 underlying HW interrupt failed
115 -EBUSY: No CPU available to serve interrupt
117 4. KVM_DEV_XIVE_GRP_EQ_CONFIG (read-write)
118 Configures an event queue of a CPU
120 EQ descriptor identifier (64-bit)
121 The EQ descriptor identifier is a tuple (server, priority) :
122 bits: | 63 .... 32 | 31 .. 3 | 2 .. 0
123 values: | unused | server | priority
124 The kvm_device_attr.addr points to :
125 struct kvm_ppc_xive_eq {
134 KVM_XIVE_EQ_ALWAYS_NOTIFY (required)
135 forces notification without using the coalescing mechanism
136 provided by the XIVE END ESBs.
137 - qshift: queue size (power of 2)
138 - qaddr: real address of queue
139 - qtoggle: current queue toggle bit
140 - qindex: current queue index
141 - pad: reserved for future use
143 -ENOENT: Invalid CPU number
144 -EINVAL: Invalid priority
145 -EINVAL: Invalid flags
146 -EINVAL: Invalid queue size
147 -EINVAL: Invalid queue address
148 -EFAULT: Invalid user pointer for attr->addr.
149 -EIO: Configuration of the underlying HW failed
151 5. KVM_DEV_XIVE_GRP_SOURCE_SYNC (write only)
152 Synchronize the source to flush event notifications
154 Interrupt source number (64-bit)
156 -ENOENT: Unknown source number
157 -EINVAL: Not initialized source number
161 The XIVE IC maintains VP interrupt state in an internal structure
162 called the NVT. When a VP is not dispatched on a HW processor
163 thread, this structure can be updated by HW if the VP is the target
164 of an event notification.
166 It is important for migration to capture the cached IPB from the NVT
167 as it synthesizes the priorities of the pending interrupts. We
168 capture a bit more to report debug information.
170 KVM_REG_PPC_VP_STATE (2 * 64bits)
171 bits: | 63 .... 32 | 31 .... 0 |
172 values: | TIMA word0 | TIMA word1 |
173 bits: | 127 .......... 64 |
178 Saving the state of a VM using the XIVE native exploitation mode
179 should follow a specific sequence. When the VM is stopped :
181 1. Mask all sources (PQ=01) to stop the flow of events.
183 2. Sync the XIVE device with the KVM control KVM_DEV_XIVE_EQ_SYNC to
184 flush any in-flight event notification and to stabilize the EQs. At
185 this stage, the EQ pages are marked dirty to make sure they are
186 transferred in the migration sequence.
188 3. Capture the state of the source targeting, the EQs configuration
189 and the state of thread interrupt context registers.
193 1. Restore the EQ configuration. As targeting depends on it.
195 3. Restore the thread interrupt contexts
196 4. Restore the source states