7 The goal of firmware-assisted dump is to enable the dump of
8 a crashed system, and to do so from a fully-reset system, and
9 to minimize the total elapsed time until the system is back
12 - Firmware assisted dump (fadump) infrastructure is intended to replace
13 the existing phyp assisted dump.
14 - Fadump uses the same firmware interfaces and memory reservation model
15 as phyp assisted dump.
16 - Unlike phyp dump, fadump exports the memory dump through /proc/vmcore
17 in the ELF format in the same way as kdump. This helps us reuse the
18 kdump infrastructure for dump capture and filtering.
19 - Unlike phyp dump, userspace tool does not need to refer any sysfs
20 interface while reading /proc/vmcore.
21 - Unlike phyp dump, fadump allows user to release all the memory reserved
22 for dump, with a single operation of echo 1 > /sys/kernel/fadump_release_mem.
23 - Once enabled through kernel boot parameter, fadump can be
24 started/stopped through /sys/kernel/fadump_registered interface (see
25 sysfs files section below) and can be easily integrated with kdump
26 service start/stop init scripts.
28 Comparing with kdump or other strategies, firmware-assisted
29 dump offers several strong, practical advantages:
31 - Unlike kdump, the system has been reset, and loaded
32 with a fresh copy of the kernel. In particular,
33 PCI and I/O devices have been reinitialized and are
34 in a clean, consistent state.
35 - Once the dump is copied out, the memory that held the dump
36 is immediately available to the running kernel. And therefore,
37 unlike kdump, fadump doesn't need a 2nd reboot to get back
38 the system to the production configuration.
40 The above can only be accomplished by coordination with,
41 and assistance from the Power firmware. The procedure is
44 - The first kernel registers the sections of memory with the
45 Power firmware for dump preservation during OS initialization.
46 These registered sections of memory are reserved by the first
47 kernel during early boot.
49 - When a system crashes, the Power firmware will save
50 the low memory (boot memory of size larger of 5% of system RAM
51 or 256MB) of RAM to the previous registered region. It will
52 also save system registers, and hardware PTE's.
55 The term 'boot memory' means size of the low memory chunk
56 that is required for a kernel to boot successfully when
57 booted with restricted memory. By default, the boot memory
58 size will be the larger of 5% of system RAM or 256MB.
59 Alternatively, user can also specify boot memory size
60 through boot parameter 'crashkernel=' which will override
61 the default calculated size. Use this option if default
62 boot memory size is not sufficient for second kernel to
63 boot successfully. For syntax of crashkernel= parameter,
64 refer to Documentation/admin-guide/kdump/kdump.rst. If any offset is
65 provided in crashkernel= parameter, it will be ignored
66 as fadump uses a predefined offset to reserve memory
67 for boot memory dump preservation in case of a crash.
69 - After the low memory (boot memory) area has been saved, the
70 firmware will reset PCI and other hardware state. It will
71 *not* clear the RAM. It will then launch the bootloader, as
74 - The freshly booted kernel will notice that there is a new
75 node (ibm,dump-kernel) in the device tree, indicating that
76 there is crash data available from a previous boot. During
77 the early boot OS will reserve rest of the memory above
78 boot memory size effectively booting with restricted memory
79 size. This will make sure that the second kernel will not
80 touch any of the dump memory area.
82 - User-space tools will read /proc/vmcore to obtain the contents
83 of memory, which holds the previous crashed kernel dump in ELF
84 format. The userspace tools may copy this info to disk, or
85 network, nas, san, iscsi, etc. as desired.
87 - Once the userspace tool is done saving dump, it will echo
88 '1' to /sys/kernel/fadump_release_mem to release the reserved
89 memory back to general use, except the memory required for
90 next firmware-assisted dump registration.
94 # echo 1 > /sys/kernel/fadump_release_mem
96 Please note that the firmware-assisted dump feature
97 is only available on Power6 and above systems with recent
100 Implementation details:
101 -----------------------
103 During boot, a check is made to see if firmware supports
104 this feature on that particular machine. If it does, then
105 we check to see if an active dump is waiting for us. If yes
106 then everything but boot memory size of RAM is reserved during
107 early boot (See Fig. 2). This area is released once we finish
108 collecting the dump from user land scripts (e.g. kdump scripts)
109 that are run. If there is dump data, then the
110 /sys/kernel/fadump_release_mem file is created, and the reserved
113 If there is no waiting dump data, then only the memory required
114 to hold CPU state, HPTE region, boot memory dump and elfcore
115 header, is usually reserved at an offset greater than boot memory
116 size (see Fig. 1). This area is *not* released: this region will
117 be kept permanently reserved, so that it can act as a receptacle
118 for a copy of the boot memory content in addition to CPU state
119 and HPTE region, in the case a crash does occur. Since this reserved
120 memory area is used only after the system crash, there is no point in
121 blocking this significant chunk of memory from production kernel.
122 Hence, the implementation uses the Linux kernel's Contiguous Memory
123 Allocator (CMA) for memory reservation if CMA is configured for kernel.
124 With CMA reservation this memory will be available for applications to
125 use it, while kernel is prevented from using it. With this fadump will
126 still be able to capture all of the kernel memory and most of the user
127 space memory except the user pages that were present in CMA region::
129 o Memory Reservation during first kernel
131 Low memory Top of memory
133 | | |<--Reserved dump area -->| |
134 V V | Permanent Reservation | V
135 +-----------+----------/ /---+---+----+-----------+----+------+
136 | | |CPU|HPTE| DUMP |ELF | |
137 +-----------+----------/ /---+---+----+-----------+----+------+
141 -------------------------------------------
142 Boot memory content gets transferred to
143 reserved area by firmware at the time of
147 o Memory Reservation during second kernel after crash
149 Low memory Top of memory
151 | |<------------- Reserved dump area ----------- -->|
153 +-----------+----------/ /---+---+----+-----------+----+------+
154 | | |CPU|HPTE| DUMP |ELF | |
155 +-----------+----------/ /---+---+----+-----------+----+------+
158 Used by second /proc/vmcore
162 Currently the dump will be copied from /proc/vmcore to a
163 a new file upon user intervention. The dump data available through
164 /proc/vmcore will be in ELF format. Hence the existing kdump
165 infrastructure (kdump scripts) to save the dump works fine with
168 The tools to examine the dump will be same as the ones
171 How to enable firmware-assisted dump (fadump):
172 ----------------------------------------------
174 1. Set config option CONFIG_FA_DUMP=y and build kernel.
175 2. Boot into linux kernel with 'fadump=on' kernel cmdline option.
176 By default, fadump reserved memory will be initialized as CMA area.
177 Alternatively, user can boot linux kernel with 'fadump=nocma' to
178 prevent fadump to use CMA.
179 3. Optionally, user can also set 'crashkernel=' kernel cmdline
180 to specify size of the memory to reserve for boot memory dump
184 1. 'fadump_reserve_mem=' parameter has been deprecated. Instead
185 use 'crashkernel=' to specify size of the memory to reserve
186 for boot memory dump preservation.
187 2. If firmware-assisted dump fails to reserve memory then it
188 will fallback to existing kdump mechanism if 'crashkernel='
189 option is set at kernel cmdline.
190 3. if user wants to capture all of user space memory and ok with
191 reserved memory not available to production system, then
192 'fadump=nocma' kernel parameter can be used to fallback to
198 Firmware-assisted dump feature uses sysfs file system to hold
199 the control files and debugfs file to display memory reserved region.
201 Here is the list of files under kernel sysfs:
203 /sys/kernel/fadump_enabled
204 This is used to display the fadump status.
206 - 0 = fadump is disabled
207 - 1 = fadump is enabled
209 This interface can be used by kdump init scripts to identify if
210 fadump is enabled in the kernel and act accordingly.
212 /sys/kernel/fadump_registered
213 This is used to display the fadump registration status as well
214 as to control (start/stop) the fadump registration.
216 - 0 = fadump is not registered.
217 - 1 = fadump is registered and ready to handle system crash.
219 To register fadump echo 1 > /sys/kernel/fadump_registered and
220 echo 0 > /sys/kernel/fadump_registered for un-register and stop the
221 fadump. Once the fadump is un-registered, the system crash will not
222 be handled and vmcore will not be captured. This interface can be
223 easily integrated with kdump service start/stop.
225 /sys/kernel/fadump_release_mem
226 This file is available only when fadump is active during
227 second kernel. This is used to release the reserved memory
228 region that are held for saving crash dump. To release the
229 reserved memory echo 1 to it::
231 echo 1 > /sys/kernel/fadump_release_mem
233 After echo 1, the content of the /sys/kernel/debug/powerpc/fadump_region
234 file will change to reflect the new memory reservations.
236 The existing userspace tools (kdump infrastructure) can be easily
237 enhanced to use this interface to release the memory reserved for
238 dump and continue without 2nd reboot.
240 Here is the list of files under powerpc debugfs:
241 (Assuming debugfs is mounted on /sys/kernel/debug directory.)
243 /sys/kernel/debug/powerpc/fadump_region
244 This file shows the reserved memory regions if fadump is
245 enabled otherwise this file is empty. The output format
248 <region>: [<start>-<end>] <reserved-size> bytes, Dumped: <dump-size>
251 Contents when fadump is registered during first kernel::
253 # cat /sys/kernel/debug/powerpc/fadump_region
254 CPU : [0x0000006ffb0000-0x0000006fff001f] 0x40020 bytes, Dumped: 0x0
255 HPTE: [0x0000006fff0020-0x0000006fff101f] 0x1000 bytes, Dumped: 0x0
256 DUMP: [0x0000006fff1020-0x0000007fff101f] 0x10000000 bytes, Dumped: 0x0
258 Contents when fadump is active during second kernel::
260 # cat /sys/kernel/debug/powerpc/fadump_region
261 CPU : [0x0000006ffb0000-0x0000006fff001f] 0x40020 bytes, Dumped: 0x40020
262 HPTE: [0x0000006fff0020-0x0000006fff101f] 0x1000 bytes, Dumped: 0x1000
263 DUMP: [0x0000006fff1020-0x0000007fff101f] 0x10000000 bytes, Dumped: 0x10000000
264 : [0x00000010000000-0x0000006ffaffff] 0x5ffb0000 bytes, Dumped: 0x5ffb0000
267 Please refer to Documentation/filesystems/debugfs.txt on
268 how to mount the debugfs filesystem.
273 - Need to come up with the better approach to find out more
274 accurate boot memory size that is required for a kernel to
275 boot successfully when booted with restricted memory.
276 - The fadump implementation introduces a fadump crash info structure
277 in the scratch area before the ELF core header. The idea of introducing
278 this structure is to pass some important crash info data to the second
279 kernel which will help second kernel to populate ELF core header with
280 correct data before it gets exported through /proc/vmcore. The current
281 design implementation does not address a possibility of introducing
282 additional fields (in future) to this structure without affecting
283 compatibility. Need to come up with the better approach to address this.
285 The possible approaches are:
287 1. Introduce version field for version tracking, bump up the version
288 whenever a new field is added to the structure in future. The version
289 field can be used to find out what fields are valid for the current
290 version of the structure.
291 2. Reserve the area of predefined size (say PAGE_SIZE) for this
292 structure and have unused area as reserved (initialized to zero)
293 for future field additions.
295 The advantage of approach 1 over 2 is we don't need to reserve extra space.
297 Author: Mahesh Salgaonkar <mahesh@linux.vnet.ibm.com>
299 This document is based on the original documentation written for phyp
301 assisted dump by Linas Vepstas and Manish Ahuja.