6 :Author: Tejun Heo <tj@kernel.org>
8 This is the authoritative documentation on the design, interface and
9 conventions of cgroup v2. It describes all userland-visible aspects
10 of cgroup including core and specific controller behaviors. All
11 future changes must be reflected in this document. Documentation for
12 v1 is available under Documentation/cgroup-v1/.
21 2-2. Organizing Processes and Threads
24 2-3. [Un]populated Notification
25 2-4. Controlling Controllers
26 2-4-1. Enabling and Disabling
27 2-4-2. Top-down Constraint
28 2-4-3. No Internal Process Constraint
30 2-5-1. Model of Delegation
31 2-5-2. Delegation Containment
33 2-6-1. Organize Once and Control
34 2-6-2. Avoid Name Collisions
35 3. Resource Distribution Models
43 4-3. Core Interface Files
46 5-1-1. CPU Interface Files
48 5-2-1. Memory Interface Files
49 5-2-2. Usage Guidelines
50 5-2-3. Memory Ownership
52 5-3-1. IO Interface Files
55 5-4-1. PID Interface Files
58 5-6-1. RDMA Interface Files
63 6-2. The Root and Views
64 6-3. Migration and setns(2)
65 6-4. Interaction with Other Namespaces
66 P. Information on Kernel Programming
67 P-1. Filesystem Support for Writeback
68 D. Deprecated v1 Core Features
69 R. Issues with v1 and Rationales for v2
70 R-1. Multiple Hierarchies
71 R-2. Thread Granularity
72 R-3. Competition Between Inner Nodes and Threads
73 R-4. Other Interface Issues
74 R-5. Controller Issues and Remedies
84 "cgroup" stands for "control group" and is never capitalized. The
85 singular form is used to designate the whole feature and also as a
86 qualifier as in "cgroup controllers". When explicitly referring to
87 multiple individual control groups, the plural form "cgroups" is used.
93 cgroup is a mechanism to organize processes hierarchically and
94 distribute system resources along the hierarchy in a controlled and
97 cgroup is largely composed of two parts - the core and controllers.
98 cgroup core is primarily responsible for hierarchically organizing
99 processes. A cgroup controller is usually responsible for
100 distributing a specific type of system resource along the hierarchy
101 although there are utility controllers which serve purposes other than
102 resource distribution.
104 cgroups form a tree structure and every process in the system belongs
105 to one and only one cgroup. All threads of a process belong to the
106 same cgroup. On creation, all processes are put in the cgroup that
107 the parent process belongs to at the time. A process can be migrated
108 to another cgroup. Migration of a process doesn't affect already
109 existing descendant processes.
111 Following certain structural constraints, controllers may be enabled or
112 disabled selectively on a cgroup. All controller behaviors are
113 hierarchical - if a controller is enabled on a cgroup, it affects all
114 processes which belong to the cgroups consisting the inclusive
115 sub-hierarchy of the cgroup. When a controller is enabled on a nested
116 cgroup, it always restricts the resource distribution further. The
117 restrictions set closer to the root in the hierarchy can not be
118 overridden from further away.
127 Unlike v1, cgroup v2 has only single hierarchy. The cgroup v2
128 hierarchy can be mounted with the following mount command::
130 # mount -t cgroup2 none $MOUNT_POINT
132 cgroup2 filesystem has the magic number 0x63677270 ("cgrp"). All
133 controllers which support v2 and are not bound to a v1 hierarchy are
134 automatically bound to the v2 hierarchy and show up at the root.
135 Controllers which are not in active use in the v2 hierarchy can be
136 bound to other hierarchies. This allows mixing v2 hierarchy with the
137 legacy v1 multiple hierarchies in a fully backward compatible way.
139 A controller can be moved across hierarchies only after the controller
140 is no longer referenced in its current hierarchy. Because per-cgroup
141 controller states are destroyed asynchronously and controllers may
142 have lingering references, a controller may not show up immediately on
143 the v2 hierarchy after the final umount of the previous hierarchy.
144 Similarly, a controller should be fully disabled to be moved out of
145 the unified hierarchy and it may take some time for the disabled
146 controller to become available for other hierarchies; furthermore, due
147 to inter-controller dependencies, other controllers may need to be
150 While useful for development and manual configurations, moving
151 controllers dynamically between the v2 and other hierarchies is
152 strongly discouraged for production use. It is recommended to decide
153 the hierarchies and controller associations before starting using the
154 controllers after system boot.
156 During transition to v2, system management software might still
157 automount the v1 cgroup filesystem and so hijack all controllers
158 during boot, before manual intervention is possible. To make testing
159 and experimenting easier, the kernel parameter cgroup_no_v1= allows
160 disabling controllers in v1 and make them always available in v2.
162 cgroup v2 currently supports the following mount options.
166 Consider cgroup namespaces as delegation boundaries. This
167 option is system wide and can only be set on mount or modified
168 through remount from the init namespace. The mount option is
169 ignored on non-init namespace mounts. Please refer to the
170 Delegation section for details.
173 Organizing Processes and Threads
174 --------------------------------
179 Initially, only the root cgroup exists to which all processes belong.
180 A child cgroup can be created by creating a sub-directory::
184 A given cgroup may have multiple child cgroups forming a tree
185 structure. Each cgroup has a read-writable interface file
186 "cgroup.procs". When read, it lists the PIDs of all processes which
187 belong to the cgroup one-per-line. The PIDs are not ordered and the
188 same PID may show up more than once if the process got moved to
189 another cgroup and then back or the PID got recycled while reading.
191 A process can be migrated into a cgroup by writing its PID to the
192 target cgroup's "cgroup.procs" file. Only one process can be migrated
193 on a single write(2) call. If a process is composed of multiple
194 threads, writing the PID of any thread migrates all threads of the
197 When a process forks a child process, the new process is born into the
198 cgroup that the forking process belongs to at the time of the
199 operation. After exit, a process stays associated with the cgroup
200 that it belonged to at the time of exit until it's reaped; however, a
201 zombie process does not appear in "cgroup.procs" and thus can't be
202 moved to another cgroup.
204 A cgroup which doesn't have any children or live processes can be
205 destroyed by removing the directory. Note that a cgroup which doesn't
206 have any children and is associated only with zombie processes is
207 considered empty and can be removed::
211 "/proc/$PID/cgroup" lists a process's cgroup membership. If legacy
212 cgroup is in use in the system, this file may contain multiple lines,
213 one for each hierarchy. The entry for cgroup v2 is always in the
216 # cat /proc/842/cgroup
218 0::/test-cgroup/test-cgroup-nested
220 If the process becomes a zombie and the cgroup it was associated with
221 is removed subsequently, " (deleted)" is appended to the path::
223 # cat /proc/842/cgroup
225 0::/test-cgroup/test-cgroup-nested (deleted)
231 cgroup v2 supports thread granularity for a subset of controllers to
232 support use cases requiring hierarchical resource distribution across
233 the threads of a group of processes. By default, all threads of a
234 process belong to the same cgroup, which also serves as the resource
235 domain to host resource consumptions which are not specific to a
236 process or thread. The thread mode allows threads to be spread across
237 a subtree while still maintaining the common resource domain for them.
239 Controllers which support thread mode are called threaded controllers.
240 The ones which don't are called domain controllers.
242 Marking a cgroup threaded makes it join the resource domain of its
243 parent as a threaded cgroup. The parent may be another threaded
244 cgroup whose resource domain is further up in the hierarchy. The root
245 of a threaded subtree, that is, the nearest ancestor which is not
246 threaded, is called threaded domain or thread root interchangeably and
247 serves as the resource domain for the entire subtree.
249 Inside a threaded subtree, threads of a process can be put in
250 different cgroups and are not subject to the no internal process
251 constraint - threaded controllers can be enabled on non-leaf cgroups
252 whether they have threads in them or not.
254 As the threaded domain cgroup hosts all the domain resource
255 consumptions of the subtree, it is considered to have internal
256 resource consumptions whether there are processes in it or not and
257 can't have populated child cgroups which aren't threaded. Because the
258 root cgroup is not subject to no internal process constraint, it can
259 serve both as a threaded domain and a parent to domain cgroups.
261 The current operation mode or type of the cgroup is shown in the
262 "cgroup.type" file which indicates whether the cgroup is a normal
263 domain, a domain which is serving as the domain of a threaded subtree,
264 or a threaded cgroup.
266 On creation, a cgroup is always a domain cgroup and can be made
267 threaded by writing "threaded" to the "cgroup.type" file. The
268 operation is single direction::
270 # echo threaded > cgroup.type
272 Once threaded, the cgroup can't be made a domain again. To enable the
273 thread mode, the following conditions must be met.
275 - As the cgroup will join the parent's resource domain. The parent
276 must either be a valid (threaded) domain or a threaded cgroup.
278 - When the parent is an unthreaded domain, it must not have any domain
279 controllers enabled or populated domain children. The root is
280 exempt from this requirement.
282 Topology-wise, a cgroup can be in an invalid state. Please consider
283 the following toplogy::
285 A (threaded domain) - B (threaded) - C (domain, just created)
287 C is created as a domain but isn't connected to a parent which can
288 host child domains. C can't be used until it is turned into a
289 threaded cgroup. "cgroup.type" file will report "domain (invalid)" in
290 these cases. Operations which fail due to invalid topology use
291 EOPNOTSUPP as the errno.
293 A domain cgroup is turned into a threaded domain when one of its child
294 cgroup becomes threaded or threaded controllers are enabled in the
295 "cgroup.subtree_control" file while there are processes in the cgroup.
296 A threaded domain reverts to a normal domain when the conditions
299 When read, "cgroup.threads" contains the list of the thread IDs of all
300 threads in the cgroup. Except that the operations are per-thread
301 instead of per-process, "cgroup.threads" has the same format and
302 behaves the same way as "cgroup.procs". While "cgroup.threads" can be
303 written to in any cgroup, as it can only move threads inside the same
304 threaded domain, its operations are confined inside each threaded
307 The threaded domain cgroup serves as the resource domain for the whole
308 subtree, and, while the threads can be scattered across the subtree,
309 all the processes are considered to be in the threaded domain cgroup.
310 "cgroup.procs" in a threaded domain cgroup contains the PIDs of all
311 processes in the subtree and is not readable in the subtree proper.
312 However, "cgroup.procs" can be written to from anywhere in the subtree
313 to migrate all threads of the matching process to the cgroup.
315 Only threaded controllers can be enabled in a threaded subtree. When
316 a threaded controller is enabled inside a threaded subtree, it only
317 accounts for and controls resource consumptions associated with the
318 threads in the cgroup and its descendants. All consumptions which
319 aren't tied to a specific thread belong to the threaded domain cgroup.
321 Because a threaded subtree is exempt from no internal process
322 constraint, a threaded controller must be able to handle competition
323 between threads in a non-leaf cgroup and its child cgroups. Each
324 threaded controller defines how such competitions are handled.
327 [Un]populated Notification
328 --------------------------
330 Each non-root cgroup has a "cgroup.events" file which contains
331 "populated" field indicating whether the cgroup's sub-hierarchy has
332 live processes in it. Its value is 0 if there is no live process in
333 the cgroup and its descendants; otherwise, 1. poll and [id]notify
334 events are triggered when the value changes. This can be used, for
335 example, to start a clean-up operation after all processes of a given
336 sub-hierarchy have exited. The populated state updates and
337 notifications are recursive. Consider the following sub-hierarchy
338 where the numbers in the parentheses represent the numbers of processes
344 A, B and C's "populated" fields would be 1 while D's 0. After the one
345 process in C exits, B and C's "populated" fields would flip to "0" and
346 file modified events will be generated on the "cgroup.events" files of
350 Controlling Controllers
351 -----------------------
353 Enabling and Disabling
354 ~~~~~~~~~~~~~~~~~~~~~~
356 Each cgroup has a "cgroup.controllers" file which lists all
357 controllers available for the cgroup to enable::
359 # cat cgroup.controllers
362 No controller is enabled by default. Controllers can be enabled and
363 disabled by writing to the "cgroup.subtree_control" file::
365 # echo "+cpu +memory -io" > cgroup.subtree_control
367 Only controllers which are listed in "cgroup.controllers" can be
368 enabled. When multiple operations are specified as above, either they
369 all succeed or fail. If multiple operations on the same controller
370 are specified, the last one is effective.
372 Enabling a controller in a cgroup indicates that the distribution of
373 the target resource across its immediate children will be controlled.
374 Consider the following sub-hierarchy. The enabled controllers are
375 listed in parentheses::
377 A(cpu,memory) - B(memory) - C()
380 As A has "cpu" and "memory" enabled, A will control the distribution
381 of CPU cycles and memory to its children, in this case, B. As B has
382 "memory" enabled but not "CPU", C and D will compete freely on CPU
383 cycles but their division of memory available to B will be controlled.
385 As a controller regulates the distribution of the target resource to
386 the cgroup's children, enabling it creates the controller's interface
387 files in the child cgroups. In the above example, enabling "cpu" on B
388 would create the "cpu." prefixed controller interface files in C and
389 D. Likewise, disabling "memory" from B would remove the "memory."
390 prefixed controller interface files from C and D. This means that the
391 controller interface files - anything which doesn't start with
392 "cgroup." are owned by the parent rather than the cgroup itself.
398 Resources are distributed top-down and a cgroup can further distribute
399 a resource only if the resource has been distributed to it from the
400 parent. This means that all non-root "cgroup.subtree_control" files
401 can only contain controllers which are enabled in the parent's
402 "cgroup.subtree_control" file. A controller can be enabled only if
403 the parent has the controller enabled and a controller can't be
404 disabled if one or more children have it enabled.
407 No Internal Process Constraint
408 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
410 Non-root cgroups can distribute domain resources to their children
411 only when they don't have any processes of their own. In other words,
412 only domain cgroups which don't contain any processes can have domain
413 controllers enabled in their "cgroup.subtree_control" files.
415 This guarantees that, when a domain controller is looking at the part
416 of the hierarchy which has it enabled, processes are always only on
417 the leaves. This rules out situations where child cgroups compete
418 against internal processes of the parent.
420 The root cgroup is exempt from this restriction. Root contains
421 processes and anonymous resource consumption which can't be associated
422 with any other cgroups and requires special treatment from most
423 controllers. How resource consumption in the root cgroup is governed
424 is up to each controller.
426 Note that the restriction doesn't get in the way if there is no
427 enabled controller in the cgroup's "cgroup.subtree_control". This is
428 important as otherwise it wouldn't be possible to create children of a
429 populated cgroup. To control resource distribution of a cgroup, the
430 cgroup must create children and transfer all its processes to the
431 children before enabling controllers in its "cgroup.subtree_control"
441 A cgroup can be delegated in two ways. First, to a less privileged
442 user by granting write access of the directory and its "cgroup.procs",
443 "cgroup.threads" and "cgroup.subtree_control" files to the user.
444 Second, if the "nsdelegate" mount option is set, automatically to a
445 cgroup namespace on namespace creation.
447 Because the resource control interface files in a given directory
448 control the distribution of the parent's resources, the delegatee
449 shouldn't be allowed to write to them. For the first method, this is
450 achieved by not granting access to these files. For the second, the
451 kernel rejects writes to all files other than "cgroup.procs" and
452 "cgroup.subtree_control" on a namespace root from inside the
455 The end results are equivalent for both delegation types. Once
456 delegated, the user can build sub-hierarchy under the directory,
457 organize processes inside it as it sees fit and further distribute the
458 resources it received from the parent. The limits and other settings
459 of all resource controllers are hierarchical and regardless of what
460 happens in the delegated sub-hierarchy, nothing can escape the
461 resource restrictions imposed by the parent.
463 Currently, cgroup doesn't impose any restrictions on the number of
464 cgroups in or nesting depth of a delegated sub-hierarchy; however,
465 this may be limited explicitly in the future.
468 Delegation Containment
469 ~~~~~~~~~~~~~~~~~~~~~~
471 A delegated sub-hierarchy is contained in the sense that processes
472 can't be moved into or out of the sub-hierarchy by the delegatee.
474 For delegations to a less privileged user, this is achieved by
475 requiring the following conditions for a process with a non-root euid
476 to migrate a target process into a cgroup by writing its PID to the
479 - The writer must have write access to the "cgroup.procs" file.
481 - The writer must have write access to the "cgroup.procs" file of the
482 common ancestor of the source and destination cgroups.
484 The above two constraints ensure that while a delegatee may migrate
485 processes around freely in the delegated sub-hierarchy it can't pull
486 in from or push out to outside the sub-hierarchy.
488 For an example, let's assume cgroups C0 and C1 have been delegated to
489 user U0 who created C00, C01 under C0 and C10 under C1 as follows and
490 all processes under C0 and C1 belong to U0::
492 ~~~~~~~~~~~~~ - C0 - C00
495 ~~~~~~~~~~~~~ - C1 - C10
497 Let's also say U0 wants to write the PID of a process which is
498 currently in C10 into "C00/cgroup.procs". U0 has write access to the
499 file; however, the common ancestor of the source cgroup C10 and the
500 destination cgroup C00 is above the points of delegation and U0 would
501 not have write access to its "cgroup.procs" files and thus the write
502 will be denied with -EACCES.
504 For delegations to namespaces, containment is achieved by requiring
505 that both the source and destination cgroups are reachable from the
506 namespace of the process which is attempting the migration. If either
507 is not reachable, the migration is rejected with -ENOENT.
513 Organize Once and Control
514 ~~~~~~~~~~~~~~~~~~~~~~~~~
516 Migrating a process across cgroups is a relatively expensive operation
517 and stateful resources such as memory are not moved together with the
518 process. This is an explicit design decision as there often exist
519 inherent trade-offs between migration and various hot paths in terms
520 of synchronization cost.
522 As such, migrating processes across cgroups frequently as a means to
523 apply different resource restrictions is discouraged. A workload
524 should be assigned to a cgroup according to the system's logical and
525 resource structure once on start-up. Dynamic adjustments to resource
526 distribution can be made by changing controller configuration through
530 Avoid Name Collisions
531 ~~~~~~~~~~~~~~~~~~~~~
533 Interface files for a cgroup and its children cgroups occupy the same
534 directory and it is possible to create children cgroups which collide
535 with interface files.
537 All cgroup core interface files are prefixed with "cgroup." and each
538 controller's interface files are prefixed with the controller name and
539 a dot. A controller's name is composed of lower case alphabets and
540 '_'s but never begins with an '_' so it can be used as the prefix
541 character for collision avoidance. Also, interface file names won't
542 start or end with terms which are often used in categorizing workloads
543 such as job, service, slice, unit or workload.
545 cgroup doesn't do anything to prevent name collisions and it's the
546 user's responsibility to avoid them.
549 Resource Distribution Models
550 ============================
552 cgroup controllers implement several resource distribution schemes
553 depending on the resource type and expected use cases. This section
554 describes major schemes in use along with their expected behaviors.
560 A parent's resource is distributed by adding up the weights of all
561 active children and giving each the fraction matching the ratio of its
562 weight against the sum. As only children which can make use of the
563 resource at the moment participate in the distribution, this is
564 work-conserving. Due to the dynamic nature, this model is usually
565 used for stateless resources.
567 All weights are in the range [1, 10000] with the default at 100. This
568 allows symmetric multiplicative biases in both directions at fine
569 enough granularity while staying in the intuitive range.
571 As long as the weight is in range, all configuration combinations are
572 valid and there is no reason to reject configuration changes or
575 "cpu.weight" proportionally distributes CPU cycles to active children
576 and is an example of this type.
582 A child can only consume upto the configured amount of the resource.
583 Limits can be over-committed - the sum of the limits of children can
584 exceed the amount of resource available to the parent.
586 Limits are in the range [0, max] and defaults to "max", which is noop.
588 As limits can be over-committed, all configuration combinations are
589 valid and there is no reason to reject configuration changes or
592 "io.max" limits the maximum BPS and/or IOPS that a cgroup can consume
593 on an IO device and is an example of this type.
599 A cgroup is protected to be allocated upto the configured amount of
600 the resource if the usages of all its ancestors are under their
601 protected levels. Protections can be hard guarantees or best effort
602 soft boundaries. Protections can also be over-committed in which case
603 only upto the amount available to the parent is protected among
606 Protections are in the range [0, max] and defaults to 0, which is
609 As protections can be over-committed, all configuration combinations
610 are valid and there is no reason to reject configuration changes or
613 "memory.low" implements best-effort memory protection and is an
614 example of this type.
620 A cgroup is exclusively allocated a certain amount of a finite
621 resource. Allocations can't be over-committed - the sum of the
622 allocations of children can not exceed the amount of resource
623 available to the parent.
625 Allocations are in the range [0, max] and defaults to 0, which is no
628 As allocations can't be over-committed, some configuration
629 combinations are invalid and should be rejected. Also, if the
630 resource is mandatory for execution of processes, process migrations
633 "cpu.rt.max" hard-allocates realtime slices and is an example of this
643 All interface files should be in one of the following formats whenever
646 New-line separated values
647 (when only one value can be written at once)
653 Space separated values
654 (when read-only or multiple values can be written at once)
666 KEY0 SUB_KEY0=VAL00 SUB_KEY1=VAL01...
667 KEY1 SUB_KEY0=VAL10 SUB_KEY1=VAL11...
670 For a writable file, the format for writing should generally match
671 reading; however, controllers may allow omitting later fields or
672 implement restricted shortcuts for most common use cases.
674 For both flat and nested keyed files, only the values for a single key
675 can be written at a time. For nested keyed files, the sub key pairs
676 may be specified in any order and not all pairs have to be specified.
682 - Settings for a single feature should be contained in a single file.
684 - The root cgroup should be exempt from resource control and thus
685 shouldn't have resource control interface files. Also,
686 informational files on the root cgroup which end up showing global
687 information available elsewhere shouldn't exist.
689 - If a controller implements weight based resource distribution, its
690 interface file should be named "weight" and have the range [1,
691 10000] with 100 as the default. The values are chosen to allow
692 enough and symmetric bias in both directions while keeping it
693 intuitive (the default is 100%).
695 - If a controller implements an absolute resource guarantee and/or
696 limit, the interface files should be named "min" and "max"
697 respectively. If a controller implements best effort resource
698 guarantee and/or limit, the interface files should be named "low"
699 and "high" respectively.
701 In the above four control files, the special token "max" should be
702 used to represent upward infinity for both reading and writing.
704 - If a setting has a configurable default value and keyed specific
705 overrides, the default entry should be keyed with "default" and
706 appear as the first entry in the file.
708 The default value can be updated by writing either "default $VAL" or
711 When writing to update a specific override, "default" can be used as
712 the value to indicate removal of the override. Override entries
713 with "default" as the value must not appear when read.
715 For example, a setting which is keyed by major:minor device numbers
716 with integer values may look like the following::
718 # cat cgroup-example-interface-file
722 The default value can be updated by::
724 # echo 125 > cgroup-example-interface-file
728 # echo "default 125" > cgroup-example-interface-file
730 An override can be set by::
732 # echo "8:16 170" > cgroup-example-interface-file
736 # echo "8:0 default" > cgroup-example-interface-file
737 # cat cgroup-example-interface-file
741 - For events which are not very high frequency, an interface file
742 "events" should be created which lists event key value pairs.
743 Whenever a notifiable event happens, file modified event should be
744 generated on the file.
750 All cgroup core files are prefixed with "cgroup."
754 A read-write single value file which exists on non-root
757 When read, it indicates the current type of the cgroup, which
758 can be one of the following values.
760 - "domain" : A normal valid domain cgroup.
762 - "domain threaded" : A threaded domain cgroup which is
763 serving as the root of a threaded subtree.
765 - "domain invalid" : A cgroup which is in an invalid state.
766 It can't be populated or have controllers enabled. It may
767 be allowed to become a threaded cgroup.
769 - "threaded" : A threaded cgroup which is a member of a
772 A cgroup can be turned into a threaded cgroup by writing
773 "threaded" to this file.
776 A read-write new-line separated values file which exists on
779 When read, it lists the PIDs of all processes which belong to
780 the cgroup one-per-line. The PIDs are not ordered and the
781 same PID may show up more than once if the process got moved
782 to another cgroup and then back or the PID got recycled while
785 A PID can be written to migrate the process associated with
786 the PID to the cgroup. The writer should match all of the
787 following conditions.
789 - It must have write access to the "cgroup.procs" file.
791 - It must have write access to the "cgroup.procs" file of the
792 common ancestor of the source and destination cgroups.
794 When delegating a sub-hierarchy, write access to this file
795 should be granted along with the containing directory.
797 In a threaded cgroup, reading this file fails with EOPNOTSUPP
798 as all the processes belong to the thread root. Writing is
799 supported and moves every thread of the process to the cgroup.
802 A read-write new-line separated values file which exists on
805 When read, it lists the TIDs of all threads which belong to
806 the cgroup one-per-line. The TIDs are not ordered and the
807 same TID may show up more than once if the thread got moved to
808 another cgroup and then back or the TID got recycled while
811 A TID can be written to migrate the thread associated with the
812 TID to the cgroup. The writer should match all of the
813 following conditions.
815 - It must have write access to the "cgroup.threads" file.
817 - The cgroup that the thread is currently in must be in the
818 same resource domain as the destination cgroup.
820 - It must have write access to the "cgroup.procs" file of the
821 common ancestor of the source and destination cgroups.
823 When delegating a sub-hierarchy, write access to this file
824 should be granted along with the containing directory.
827 A read-only space separated values file which exists on all
830 It shows space separated list of all controllers available to
831 the cgroup. The controllers are not ordered.
833 cgroup.subtree_control
834 A read-write space separated values file which exists on all
835 cgroups. Starts out empty.
837 When read, it shows space separated list of the controllers
838 which are enabled to control resource distribution from the
839 cgroup to its children.
841 Space separated list of controllers prefixed with '+' or '-'
842 can be written to enable or disable controllers. A controller
843 name prefixed with '+' enables the controller and '-'
844 disables. If a controller appears more than once on the list,
845 the last one is effective. When multiple enable and disable
846 operations are specified, either all succeed or all fail.
849 A read-only flat-keyed file which exists on non-root cgroups.
850 The following entries are defined. Unless specified
851 otherwise, a value change in this file generates a file
855 1 if the cgroup or its descendants contains any live
856 processes; otherwise, 0.
858 cgroup.max.descendants
859 A read-write single value files. The default is "max".
861 Maximum allowed number of descent cgroups.
862 If the actual number of descendants is equal or larger,
863 an attempt to create a new cgroup in the hierarchy will fail.
866 A read-write single value files. The default is "max".
868 Maximum allowed descent depth below the current cgroup.
869 If the actual descent depth is equal or larger,
870 an attempt to create a new child cgroup will fail.
873 A read-only flat-keyed file with the following entries:
876 Total number of visible descendant cgroups.
879 Total number of dying descendant cgroups. A cgroup becomes
880 dying after being deleted by a user. The cgroup will remain
881 in dying state for some time undefined time (which can depend
882 on system load) before being completely destroyed.
884 A process can't enter a dying cgroup under any circumstances,
885 a dying cgroup can't revive.
887 A dying cgroup can consume system resources not exceeding
888 limits, which were active at the moment of cgroup deletion.
897 The "cpu" controllers regulates distribution of CPU cycles. This
898 controller implements weight and absolute bandwidth limit models for
899 normal scheduling policy and absolute bandwidth allocation model for
900 realtime scheduling policy.
902 WARNING: cgroup2 doesn't yet support control of realtime processes and
903 the cpu controller can only be enabled when all RT processes are in
904 the root cgroup. Be aware that system management software may already
905 have placed RT processes into nonroot cgroups during the system boot
906 process, and these processes may need to be moved to the root cgroup
907 before the cpu controller can be enabled.
913 All time durations are in microseconds.
916 A read-only flat-keyed file which exists on non-root cgroups.
917 This file exists whether the controller is enabled or not.
919 It always reports the following three stats:
925 and the following three when the controller is enabled:
932 A read-write single value file which exists on non-root
933 cgroups. The default is "100".
935 The weight in the range [1, 10000].
938 A read-write single value file which exists on non-root
939 cgroups. The default is "0".
941 The nice value is in the range [-20, 19].
943 This interface file is an alternative interface for
944 "cpu.weight" and allows reading and setting weight using the
945 same values used by nice(2). Because the range is smaller and
946 granularity is coarser for the nice values, the read value is
947 the closest approximation of the current weight.
950 A read-write two value file which exists on non-root cgroups.
951 The default is "max 100000".
953 The maximum bandwidth limit. It's in the following format::
957 which indicates that the group may consume upto $MAX in each
958 $PERIOD duration. "max" for $MAX indicates no limit. If only
959 one number is written, $MAX is updated.
965 The "memory" controller regulates distribution of memory. Memory is
966 stateful and implements both limit and protection models. Due to the
967 intertwining between memory usage and reclaim pressure and the
968 stateful nature of memory, the distribution model is relatively
971 While not completely water-tight, all major memory usages by a given
972 cgroup are tracked so that the total memory consumption can be
973 accounted and controlled to a reasonable extent. Currently, the
974 following types of memory usages are tracked.
976 - Userland memory - page cache and anonymous memory.
978 - Kernel data structures such as dentries and inodes.
980 - TCP socket buffers.
982 The above list may expand in the future for better coverage.
985 Memory Interface Files
986 ~~~~~~~~~~~~~~~~~~~~~~
988 All memory amounts are in bytes. If a value which is not aligned to
989 PAGE_SIZE is written, the value may be rounded up to the closest
990 PAGE_SIZE multiple when read back.
993 A read-only single value file which exists on non-root
996 The total amount of memory currently being used by the cgroup
1000 A read-write single value file which exists on non-root
1001 cgroups. The default is "0".
1003 Best-effort memory protection. If the memory usages of a
1004 cgroup and all its ancestors are below their low boundaries,
1005 the cgroup's memory won't be reclaimed unless memory can be
1006 reclaimed from unprotected cgroups.
1008 Putting more memory than generally available under this
1009 protection is discouraged.
1012 A read-write single value file which exists on non-root
1013 cgroups. The default is "max".
1015 Memory usage throttle limit. This is the main mechanism to
1016 control memory usage of a cgroup. If a cgroup's usage goes
1017 over the high boundary, the processes of the cgroup are
1018 throttled and put under heavy reclaim pressure.
1020 Going over the high limit never invokes the OOM killer and
1021 under extreme conditions the limit may be breached.
1024 A read-write single value file which exists on non-root
1025 cgroups. The default is "max".
1027 Memory usage hard limit. This is the final protection
1028 mechanism. If a cgroup's memory usage reaches this limit and
1029 can't be reduced, the OOM killer is invoked in the cgroup.
1030 Under certain circumstances, the usage may go over the limit
1033 This is the ultimate protection mechanism. As long as the
1034 high limit is used and monitored properly, this limit's
1035 utility is limited to providing the final safety net.
1038 A read-only flat-keyed file which exists on non-root cgroups.
1039 The following entries are defined. Unless specified
1040 otherwise, a value change in this file generates a file
1044 The number of times the cgroup is reclaimed due to
1045 high memory pressure even though its usage is under
1046 the low boundary. This usually indicates that the low
1047 boundary is over-committed.
1050 The number of times processes of the cgroup are
1051 throttled and routed to perform direct memory reclaim
1052 because the high memory boundary was exceeded. For a
1053 cgroup whose memory usage is capped by the high limit
1054 rather than global memory pressure, this event's
1055 occurrences are expected.
1058 The number of times the cgroup's memory usage was
1059 about to go over the max boundary. If direct reclaim
1060 fails to bring it down, the cgroup goes to OOM state.
1063 The number of time the cgroup's memory usage was
1064 reached the limit and allocation was about to fail.
1066 Depending on context result could be invocation of OOM
1067 killer and retrying allocation or failing alloction.
1069 Failed allocation in its turn could be returned into
1070 userspace as -ENOMEM or siletly ignored in cases like
1071 disk readahead. For now OOM in memory cgroup kills
1072 tasks iff shortage has happened inside page fault.
1075 The number of processes belonging to this cgroup
1076 killed by any kind of OOM killer.
1079 A read-only flat-keyed file which exists on non-root cgroups.
1081 This breaks down the cgroup's memory footprint into different
1082 types of memory, type-specific details, and other information
1083 on the state and past events of the memory management system.
1085 All memory amounts are in bytes.
1087 The entries are ordered to be human readable, and new entries
1088 can show up in the middle. Don't rely on items remaining in a
1089 fixed position; use the keys to look up specific values!
1092 Amount of memory used in anonymous mappings such as
1093 brk(), sbrk(), and mmap(MAP_ANONYMOUS)
1096 Amount of memory used to cache filesystem data,
1097 including tmpfs and shared memory.
1100 Amount of memory allocated to kernel stacks.
1103 Amount of memory used for storing in-kernel data
1107 Amount of memory used in network transmission buffers
1110 Amount of cached filesystem data that is swap-backed,
1111 such as tmpfs, shm segments, shared anonymous mmap()s
1114 Amount of cached filesystem data mapped with mmap()
1117 Amount of cached filesystem data that was modified but
1118 not yet written back to disk
1121 Amount of cached filesystem data that was modified and
1122 is currently being written back to disk
1124 inactive_anon, active_anon, inactive_file, active_file, unevictable
1125 Amount of memory, swap-backed and filesystem-backed,
1126 on the internal memory management lists used by the
1127 page reclaim algorithm
1130 Part of "slab" that might be reclaimed, such as
1131 dentries and inodes.
1134 Part of "slab" that cannot be reclaimed on memory
1138 Total number of page faults incurred
1141 Number of major page faults incurred
1145 Number of refaults of previously evicted pages
1149 Number of refaulted pages that were immediately activated
1151 workingset_nodereclaim
1153 Number of times a shadow node has been reclaimed
1157 Amount of scanned pages (in an active LRU list)
1161 Amount of scanned pages (in an inactive LRU list)
1165 Amount of reclaimed pages
1169 Amount of pages moved to the active LRU list
1173 Amount of pages moved to the inactive LRU lis
1177 Amount of pages postponed to be freed under memory pressure
1181 Amount of reclaimed lazyfree pages
1184 A read-only single value file which exists on non-root
1187 The total amount of swap currently being used by the cgroup
1188 and its descendants.
1191 A read-write single value file which exists on non-root
1192 cgroups. The default is "max".
1194 Swap usage hard limit. If a cgroup's swap usage reaches this
1195 limit, anonymous meomry of the cgroup will not be swapped out.
1201 "memory.high" is the main mechanism to control memory usage.
1202 Over-committing on high limit (sum of high limits > available memory)
1203 and letting global memory pressure to distribute memory according to
1204 usage is a viable strategy.
1206 Because breach of the high limit doesn't trigger the OOM killer but
1207 throttles the offending cgroup, a management agent has ample
1208 opportunities to monitor and take appropriate actions such as granting
1209 more memory or terminating the workload.
1211 Determining whether a cgroup has enough memory is not trivial as
1212 memory usage doesn't indicate whether the workload can benefit from
1213 more memory. For example, a workload which writes data received from
1214 network to a file can use all available memory but can also operate as
1215 performant with a small amount of memory. A measure of memory
1216 pressure - how much the workload is being impacted due to lack of
1217 memory - is necessary to determine whether a workload needs more
1218 memory; unfortunately, memory pressure monitoring mechanism isn't
1225 A memory area is charged to the cgroup which instantiated it and stays
1226 charged to the cgroup until the area is released. Migrating a process
1227 to a different cgroup doesn't move the memory usages that it
1228 instantiated while in the previous cgroup to the new cgroup.
1230 A memory area may be used by processes belonging to different cgroups.
1231 To which cgroup the area will be charged is in-deterministic; however,
1232 over time, the memory area is likely to end up in a cgroup which has
1233 enough memory allowance to avoid high reclaim pressure.
1235 If a cgroup sweeps a considerable amount of memory which is expected
1236 to be accessed repeatedly by other cgroups, it may make sense to use
1237 POSIX_FADV_DONTNEED to relinquish the ownership of memory areas
1238 belonging to the affected files to ensure correct memory ownership.
1244 The "io" controller regulates the distribution of IO resources. This
1245 controller implements both weight based and absolute bandwidth or IOPS
1246 limit distribution; however, weight based distribution is available
1247 only if cfq-iosched is in use and neither scheme is available for
1255 A read-only nested-keyed file which exists on non-root
1258 Lines are keyed by $MAJ:$MIN device numbers and not ordered.
1259 The following nested keys are defined.
1261 ====== ===================
1263 wbytes Bytes written
1264 rios Number of read IOs
1265 wios Number of write IOs
1266 ====== ===================
1268 An example read output follows:
1270 8:16 rbytes=1459200 wbytes=314773504 rios=192 wios=353
1271 8:0 rbytes=90430464 wbytes=299008000 rios=8950 wios=1252
1274 A read-write flat-keyed file which exists on non-root cgroups.
1275 The default is "default 100".
1277 The first line is the default weight applied to devices
1278 without specific override. The rest are overrides keyed by
1279 $MAJ:$MIN device numbers and not ordered. The weights are in
1280 the range [1, 10000] and specifies the relative amount IO time
1281 the cgroup can use in relation to its siblings.
1283 The default weight can be updated by writing either "default
1284 $WEIGHT" or simply "$WEIGHT". Overrides can be set by writing
1285 "$MAJ:$MIN $WEIGHT" and unset by writing "$MAJ:$MIN default".
1287 An example read output follows::
1294 A read-write nested-keyed file which exists on non-root
1297 BPS and IOPS based IO limit. Lines are keyed by $MAJ:$MIN
1298 device numbers and not ordered. The following nested keys are
1301 ===== ==================================
1302 rbps Max read bytes per second
1303 wbps Max write bytes per second
1304 riops Max read IO operations per second
1305 wiops Max write IO operations per second
1306 ===== ==================================
1308 When writing, any number of nested key-value pairs can be
1309 specified in any order. "max" can be specified as the value
1310 to remove a specific limit. If the same key is specified
1311 multiple times, the outcome is undefined.
1313 BPS and IOPS are measured in each IO direction and IOs are
1314 delayed if limit is reached. Temporary bursts are allowed.
1316 Setting read limit at 2M BPS and write at 120 IOPS for 8:16::
1318 echo "8:16 rbps=2097152 wiops=120" > io.max
1320 Reading returns the following::
1322 8:16 rbps=2097152 wbps=max riops=max wiops=120
1324 Write IOPS limit can be removed by writing the following::
1326 echo "8:16 wiops=max" > io.max
1328 Reading now returns the following::
1330 8:16 rbps=2097152 wbps=max riops=max wiops=max
1336 Page cache is dirtied through buffered writes and shared mmaps and
1337 written asynchronously to the backing filesystem by the writeback
1338 mechanism. Writeback sits between the memory and IO domains and
1339 regulates the proportion of dirty memory by balancing dirtying and
1342 The io controller, in conjunction with the memory controller,
1343 implements control of page cache writeback IOs. The memory controller
1344 defines the memory domain that dirty memory ratio is calculated and
1345 maintained for and the io controller defines the io domain which
1346 writes out dirty pages for the memory domain. Both system-wide and
1347 per-cgroup dirty memory states are examined and the more restrictive
1348 of the two is enforced.
1350 cgroup writeback requires explicit support from the underlying
1351 filesystem. Currently, cgroup writeback is implemented on ext2, ext4
1352 and btrfs. On other filesystems, all writeback IOs are attributed to
1355 There are inherent differences in memory and writeback management
1356 which affects how cgroup ownership is tracked. Memory is tracked per
1357 page while writeback per inode. For the purpose of writeback, an
1358 inode is assigned to a cgroup and all IO requests to write dirty pages
1359 from the inode are attributed to that cgroup.
1361 As cgroup ownership for memory is tracked per page, there can be pages
1362 which are associated with different cgroups than the one the inode is
1363 associated with. These are called foreign pages. The writeback
1364 constantly keeps track of foreign pages and, if a particular foreign
1365 cgroup becomes the majority over a certain period of time, switches
1366 the ownership of the inode to that cgroup.
1368 While this model is enough for most use cases where a given inode is
1369 mostly dirtied by a single cgroup even when the main writing cgroup
1370 changes over time, use cases where multiple cgroups write to a single
1371 inode simultaneously are not supported well. In such circumstances, a
1372 significant portion of IOs are likely to be attributed incorrectly.
1373 As memory controller assigns page ownership on the first use and
1374 doesn't update it until the page is released, even if writeback
1375 strictly follows page ownership, multiple cgroups dirtying overlapping
1376 areas wouldn't work as expected. It's recommended to avoid such usage
1379 The sysctl knobs which affect writeback behavior are applied to cgroup
1380 writeback as follows.
1382 vm.dirty_background_ratio, vm.dirty_ratio
1383 These ratios apply the same to cgroup writeback with the
1384 amount of available memory capped by limits imposed by the
1385 memory controller and system-wide clean memory.
1387 vm.dirty_background_bytes, vm.dirty_bytes
1388 For cgroup writeback, this is calculated into ratio against
1389 total available memory and applied the same way as
1390 vm.dirty[_background]_ratio.
1396 The process number controller is used to allow a cgroup to stop any
1397 new tasks from being fork()'d or clone()'d after a specified limit is
1400 The number of tasks in a cgroup can be exhausted in ways which other
1401 controllers cannot prevent, thus warranting its own controller. For
1402 example, a fork bomb is likely to exhaust the number of tasks before
1403 hitting memory restrictions.
1405 Note that PIDs used in this controller refer to TIDs, process IDs as
1413 A read-write single value file which exists on non-root
1414 cgroups. The default is "max".
1416 Hard limit of number of processes.
1419 A read-only single value file which exists on all cgroups.
1421 The number of processes currently in the cgroup and its
1424 Organisational operations are not blocked by cgroup policies, so it is
1425 possible to have pids.current > pids.max. This can be done by either
1426 setting the limit to be smaller than pids.current, or attaching enough
1427 processes to the cgroup such that pids.current is larger than
1428 pids.max. However, it is not possible to violate a cgroup PID policy
1429 through fork() or clone(). These will return -EAGAIN if the creation
1430 of a new process would cause a cgroup policy to be violated.
1436 Device controller manages access to device files. It includes both
1437 creation of new device files (using mknod), and access to the
1438 existing device files.
1440 Cgroup v2 device controller has no interface files and is implemented
1441 on top of cgroup BPF. To control access to device files, a user may
1442 create bpf programs of the BPF_CGROUP_DEVICE type and attach them
1443 to cgroups. On an attempt to access a device file, corresponding
1444 BPF programs will be executed, and depending on the return value
1445 the attempt will succeed or fail with -EPERM.
1447 A BPF_CGROUP_DEVICE program takes a pointer to the bpf_cgroup_dev_ctx
1448 structure, which describes the device access attempt: access type
1449 (mknod/read/write) and device (type, major and minor numbers).
1450 If the program returns 0, the attempt fails with -EPERM, otherwise
1453 An example of BPF_CGROUP_DEVICE program may be found in the kernel
1454 source tree in the tools/testing/selftests/bpf/dev_cgroup.c file.
1460 The "rdma" controller regulates the distribution and accounting of
1463 RDMA Interface Files
1464 ~~~~~~~~~~~~~~~~~~~~
1467 A readwrite nested-keyed file that exists for all the cgroups
1468 except root that describes current configured resource limit
1469 for a RDMA/IB device.
1471 Lines are keyed by device name and are not ordered.
1472 Each line contains space separated resource name and its configured
1473 limit that can be distributed.
1475 The following nested keys are defined.
1477 ========== =============================
1478 hca_handle Maximum number of HCA Handles
1479 hca_object Maximum number of HCA Objects
1480 ========== =============================
1482 An example for mlx4 and ocrdma device follows::
1484 mlx4_0 hca_handle=2 hca_object=2000
1485 ocrdma1 hca_handle=3 hca_object=max
1488 A read-only file that describes current resource usage.
1489 It exists for all the cgroup except root.
1491 An example for mlx4 and ocrdma device follows::
1493 mlx4_0 hca_handle=1 hca_object=20
1494 ocrdma1 hca_handle=1 hca_object=23
1503 perf_event controller, if not mounted on a legacy hierarchy, is
1504 automatically enabled on the v2 hierarchy so that perf events can
1505 always be filtered by cgroup v2 path. The controller can still be
1506 moved to a legacy hierarchy after v2 hierarchy is populated.
1515 cgroup namespace provides a mechanism to virtualize the view of the
1516 "/proc/$PID/cgroup" file and cgroup mounts. The CLONE_NEWCGROUP clone
1517 flag can be used with clone(2) and unshare(2) to create a new cgroup
1518 namespace. The process running inside the cgroup namespace will have
1519 its "/proc/$PID/cgroup" output restricted to cgroupns root. The
1520 cgroupns root is the cgroup of the process at the time of creation of
1521 the cgroup namespace.
1523 Without cgroup namespace, the "/proc/$PID/cgroup" file shows the
1524 complete path of the cgroup of a process. In a container setup where
1525 a set of cgroups and namespaces are intended to isolate processes the
1526 "/proc/$PID/cgroup" file may leak potential system level information
1527 to the isolated processes. For Example::
1529 # cat /proc/self/cgroup
1530 0::/batchjobs/container_id1
1532 The path '/batchjobs/container_id1' can be considered as system-data
1533 and undesirable to expose to the isolated processes. cgroup namespace
1534 can be used to restrict visibility of this path. For example, before
1535 creating a cgroup namespace, one would see::
1537 # ls -l /proc/self/ns/cgroup
1538 lrwxrwxrwx 1 root root 0 2014-07-15 10:37 /proc/self/ns/cgroup -> cgroup:[4026531835]
1539 # cat /proc/self/cgroup
1540 0::/batchjobs/container_id1
1542 After unsharing a new namespace, the view changes::
1544 # ls -l /proc/self/ns/cgroup
1545 lrwxrwxrwx 1 root root 0 2014-07-15 10:35 /proc/self/ns/cgroup -> cgroup:[4026532183]
1546 # cat /proc/self/cgroup
1549 When some thread from a multi-threaded process unshares its cgroup
1550 namespace, the new cgroupns gets applied to the entire process (all
1551 the threads). This is natural for the v2 hierarchy; however, for the
1552 legacy hierarchies, this may be unexpected.
1554 A cgroup namespace is alive as long as there are processes inside or
1555 mounts pinning it. When the last usage goes away, the cgroup
1556 namespace is destroyed. The cgroupns root and the actual cgroups
1563 The 'cgroupns root' for a cgroup namespace is the cgroup in which the
1564 process calling unshare(2) is running. For example, if a process in
1565 /batchjobs/container_id1 cgroup calls unshare, cgroup
1566 /batchjobs/container_id1 becomes the cgroupns root. For the
1567 init_cgroup_ns, this is the real root ('/') cgroup.
1569 The cgroupns root cgroup does not change even if the namespace creator
1570 process later moves to a different cgroup::
1572 # ~/unshare -c # unshare cgroupns in some cgroup
1573 # cat /proc/self/cgroup
1576 # echo 0 > sub_cgrp_1/cgroup.procs
1577 # cat /proc/self/cgroup
1580 Each process gets its namespace-specific view of "/proc/$PID/cgroup"
1582 Processes running inside the cgroup namespace will be able to see
1583 cgroup paths (in /proc/self/cgroup) only inside their root cgroup.
1584 From within an unshared cgroupns::
1588 # echo 7353 > sub_cgrp_1/cgroup.procs
1589 # cat /proc/7353/cgroup
1592 From the initial cgroup namespace, the real cgroup path will be
1595 $ cat /proc/7353/cgroup
1596 0::/batchjobs/container_id1/sub_cgrp_1
1598 From a sibling cgroup namespace (that is, a namespace rooted at a
1599 different cgroup), the cgroup path relative to its own cgroup
1600 namespace root will be shown. For instance, if PID 7353's cgroup
1601 namespace root is at '/batchjobs/container_id2', then it will see::
1603 # cat /proc/7353/cgroup
1604 0::/../container_id2/sub_cgrp_1
1606 Note that the relative path always starts with '/' to indicate that
1607 its relative to the cgroup namespace root of the caller.
1610 Migration and setns(2)
1611 ----------------------
1613 Processes inside a cgroup namespace can move into and out of the
1614 namespace root if they have proper access to external cgroups. For
1615 example, from inside a namespace with cgroupns root at
1616 /batchjobs/container_id1, and assuming that the global hierarchy is
1617 still accessible inside cgroupns::
1619 # cat /proc/7353/cgroup
1621 # echo 7353 > batchjobs/container_id2/cgroup.procs
1622 # cat /proc/7353/cgroup
1623 0::/../container_id2
1625 Note that this kind of setup is not encouraged. A task inside cgroup
1626 namespace should only be exposed to its own cgroupns hierarchy.
1628 setns(2) to another cgroup namespace is allowed when:
1630 (a) the process has CAP_SYS_ADMIN against its current user namespace
1631 (b) the process has CAP_SYS_ADMIN against the target cgroup
1634 No implicit cgroup changes happen with attaching to another cgroup
1635 namespace. It is expected that the someone moves the attaching
1636 process under the target cgroup namespace root.
1639 Interaction with Other Namespaces
1640 ---------------------------------
1642 Namespace specific cgroup hierarchy can be mounted by a process
1643 running inside a non-init cgroup namespace::
1645 # mount -t cgroup2 none $MOUNT_POINT
1647 This will mount the unified cgroup hierarchy with cgroupns root as the
1648 filesystem root. The process needs CAP_SYS_ADMIN against its user and
1651 The virtualization of /proc/self/cgroup file combined with restricting
1652 the view of cgroup hierarchy by namespace-private cgroupfs mount
1653 provides a properly isolated cgroup view inside the container.
1656 Information on Kernel Programming
1657 =================================
1659 This section contains kernel programming information in the areas
1660 where interacting with cgroup is necessary. cgroup core and
1661 controllers are not covered.
1664 Filesystem Support for Writeback
1665 --------------------------------
1667 A filesystem can support cgroup writeback by updating
1668 address_space_operations->writepage[s]() to annotate bio's using the
1669 following two functions.
1671 wbc_init_bio(@wbc, @bio)
1672 Should be called for each bio carrying writeback data and
1673 associates the bio with the inode's owner cgroup. Can be
1674 called anytime between bio allocation and submission.
1676 wbc_account_io(@wbc, @page, @bytes)
1677 Should be called for each data segment being written out.
1678 While this function doesn't care exactly when it's called
1679 during the writeback session, it's the easiest and most
1680 natural to call it as data segments are added to a bio.
1682 With writeback bio's annotated, cgroup support can be enabled per
1683 super_block by setting SB_I_CGROUPWB in ->s_iflags. This allows for
1684 selective disabling of cgroup writeback support which is helpful when
1685 certain filesystem features, e.g. journaled data mode, are
1688 wbc_init_bio() binds the specified bio to its cgroup. Depending on
1689 the configuration, the bio may be executed at a lower priority and if
1690 the writeback session is holding shared resources, e.g. a journal
1691 entry, may lead to priority inversion. There is no one easy solution
1692 for the problem. Filesystems can try to work around specific problem
1693 cases by skipping wbc_init_bio() or using bio_associate_blkcg()
1697 Deprecated v1 Core Features
1698 ===========================
1700 - Multiple hierarchies including named ones are not supported.
1702 - All v1 mount options are not supported.
1704 - The "tasks" file is removed and "cgroup.procs" is not sorted.
1706 - "cgroup.clone_children" is removed.
1708 - /proc/cgroups is meaningless for v2. Use "cgroup.controllers" file
1709 at the root instead.
1712 Issues with v1 and Rationales for v2
1713 ====================================
1715 Multiple Hierarchies
1716 --------------------
1718 cgroup v1 allowed an arbitrary number of hierarchies and each
1719 hierarchy could host any number of controllers. While this seemed to
1720 provide a high level of flexibility, it wasn't useful in practice.
1722 For example, as there is only one instance of each controller, utility
1723 type controllers such as freezer which can be useful in all
1724 hierarchies could only be used in one. The issue is exacerbated by
1725 the fact that controllers couldn't be moved to another hierarchy once
1726 hierarchies were populated. Another issue was that all controllers
1727 bound to a hierarchy were forced to have exactly the same view of the
1728 hierarchy. It wasn't possible to vary the granularity depending on
1729 the specific controller.
1731 In practice, these issues heavily limited which controllers could be
1732 put on the same hierarchy and most configurations resorted to putting
1733 each controller on its own hierarchy. Only closely related ones, such
1734 as the cpu and cpuacct controllers, made sense to be put on the same
1735 hierarchy. This often meant that userland ended up managing multiple
1736 similar hierarchies repeating the same steps on each hierarchy
1737 whenever a hierarchy management operation was necessary.
1739 Furthermore, support for multiple hierarchies came at a steep cost.
1740 It greatly complicated cgroup core implementation but more importantly
1741 the support for multiple hierarchies restricted how cgroup could be
1742 used in general and what controllers was able to do.
1744 There was no limit on how many hierarchies there might be, which meant
1745 that a thread's cgroup membership couldn't be described in finite
1746 length. The key might contain any number of entries and was unlimited
1747 in length, which made it highly awkward to manipulate and led to
1748 addition of controllers which existed only to identify membership,
1749 which in turn exacerbated the original problem of proliferating number
1752 Also, as a controller couldn't have any expectation regarding the
1753 topologies of hierarchies other controllers might be on, each
1754 controller had to assume that all other controllers were attached to
1755 completely orthogonal hierarchies. This made it impossible, or at
1756 least very cumbersome, for controllers to cooperate with each other.
1758 In most use cases, putting controllers on hierarchies which are
1759 completely orthogonal to each other isn't necessary. What usually is
1760 called for is the ability to have differing levels of granularity
1761 depending on the specific controller. In other words, hierarchy may
1762 be collapsed from leaf towards root when viewed from specific
1763 controllers. For example, a given configuration might not care about
1764 how memory is distributed beyond a certain level while still wanting
1765 to control how CPU cycles are distributed.
1771 cgroup v1 allowed threads of a process to belong to different cgroups.
1772 This didn't make sense for some controllers and those controllers
1773 ended up implementing different ways to ignore such situations but
1774 much more importantly it blurred the line between API exposed to
1775 individual applications and system management interface.
1777 Generally, in-process knowledge is available only to the process
1778 itself; thus, unlike service-level organization of processes,
1779 categorizing threads of a process requires active participation from
1780 the application which owns the target process.
1782 cgroup v1 had an ambiguously defined delegation model which got abused
1783 in combination with thread granularity. cgroups were delegated to
1784 individual applications so that they can create and manage their own
1785 sub-hierarchies and control resource distributions along them. This
1786 effectively raised cgroup to the status of a syscall-like API exposed
1789 First of all, cgroup has a fundamentally inadequate interface to be
1790 exposed this way. For a process to access its own knobs, it has to
1791 extract the path on the target hierarchy from /proc/self/cgroup,
1792 construct the path by appending the name of the knob to the path, open
1793 and then read and/or write to it. This is not only extremely clunky
1794 and unusual but also inherently racy. There is no conventional way to
1795 define transaction across the required steps and nothing can guarantee
1796 that the process would actually be operating on its own sub-hierarchy.
1798 cgroup controllers implemented a number of knobs which would never be
1799 accepted as public APIs because they were just adding control knobs to
1800 system-management pseudo filesystem. cgroup ended up with interface
1801 knobs which were not properly abstracted or refined and directly
1802 revealed kernel internal details. These knobs got exposed to
1803 individual applications through the ill-defined delegation mechanism
1804 effectively abusing cgroup as a shortcut to implementing public APIs
1805 without going through the required scrutiny.
1807 This was painful for both userland and kernel. Userland ended up with
1808 misbehaving and poorly abstracted interfaces and kernel exposing and
1809 locked into constructs inadvertently.
1812 Competition Between Inner Nodes and Threads
1813 -------------------------------------------
1815 cgroup v1 allowed threads to be in any cgroups which created an
1816 interesting problem where threads belonging to a parent cgroup and its
1817 children cgroups competed for resources. This was nasty as two
1818 different types of entities competed and there was no obvious way to
1819 settle it. Different controllers did different things.
1821 The cpu controller considered threads and cgroups as equivalents and
1822 mapped nice levels to cgroup weights. This worked for some cases but
1823 fell flat when children wanted to be allocated specific ratios of CPU
1824 cycles and the number of internal threads fluctuated - the ratios
1825 constantly changed as the number of competing entities fluctuated.
1826 There also were other issues. The mapping from nice level to weight
1827 wasn't obvious or universal, and there were various other knobs which
1828 simply weren't available for threads.
1830 The io controller implicitly created a hidden leaf node for each
1831 cgroup to host the threads. The hidden leaf had its own copies of all
1832 the knobs with ``leaf_`` prefixed. While this allowed equivalent
1833 control over internal threads, it was with serious drawbacks. It
1834 always added an extra layer of nesting which wouldn't be necessary
1835 otherwise, made the interface messy and significantly complicated the
1838 The memory controller didn't have a way to control what happened
1839 between internal tasks and child cgroups and the behavior was not
1840 clearly defined. There were attempts to add ad-hoc behaviors and
1841 knobs to tailor the behavior to specific workloads which would have
1842 led to problems extremely difficult to resolve in the long term.
1844 Multiple controllers struggled with internal tasks and came up with
1845 different ways to deal with it; unfortunately, all the approaches were
1846 severely flawed and, furthermore, the widely different behaviors
1847 made cgroup as a whole highly inconsistent.
1849 This clearly is a problem which needs to be addressed from cgroup core
1853 Other Interface Issues
1854 ----------------------
1856 cgroup v1 grew without oversight and developed a large number of
1857 idiosyncrasies and inconsistencies. One issue on the cgroup core side
1858 was how an empty cgroup was notified - a userland helper binary was
1859 forked and executed for each event. The event delivery wasn't
1860 recursive or delegatable. The limitations of the mechanism also led
1861 to in-kernel event delivery filtering mechanism further complicating
1864 Controller interfaces were problematic too. An extreme example is
1865 controllers completely ignoring hierarchical organization and treating
1866 all cgroups as if they were all located directly under the root
1867 cgroup. Some controllers exposed a large amount of inconsistent
1868 implementation details to userland.
1870 There also was no consistency across controllers. When a new cgroup
1871 was created, some controllers defaulted to not imposing extra
1872 restrictions while others disallowed any resource usage until
1873 explicitly configured. Configuration knobs for the same type of
1874 control used widely differing naming schemes and formats. Statistics
1875 and information knobs were named arbitrarily and used different
1876 formats and units even in the same controller.
1878 cgroup v2 establishes common conventions where appropriate and updates
1879 controllers so that they expose minimal and consistent interfaces.
1882 Controller Issues and Remedies
1883 ------------------------------
1888 The original lower boundary, the soft limit, is defined as a limit
1889 that is per default unset. As a result, the set of cgroups that
1890 global reclaim prefers is opt-in, rather than opt-out. The costs for
1891 optimizing these mostly negative lookups are so high that the
1892 implementation, despite its enormous size, does not even provide the
1893 basic desirable behavior. First off, the soft limit has no
1894 hierarchical meaning. All configured groups are organized in a global
1895 rbtree and treated like equal peers, regardless where they are located
1896 in the hierarchy. This makes subtree delegation impossible. Second,
1897 the soft limit reclaim pass is so aggressive that it not just
1898 introduces high allocation latencies into the system, but also impacts
1899 system performance due to overreclaim, to the point where the feature
1900 becomes self-defeating.
1902 The memory.low boundary on the other hand is a top-down allocated
1903 reserve. A cgroup enjoys reclaim protection when it and all its
1904 ancestors are below their low boundaries, which makes delegation of
1905 subtrees possible. Secondly, new cgroups have no reserve per default
1906 and in the common case most cgroups are eligible for the preferred
1907 reclaim pass. This allows the new low boundary to be efficiently
1908 implemented with just a minor addition to the generic reclaim code,
1909 without the need for out-of-band data structures and reclaim passes.
1910 Because the generic reclaim code considers all cgroups except for the
1911 ones running low in the preferred first reclaim pass, overreclaim of
1912 individual groups is eliminated as well, resulting in much better
1913 overall workload performance.
1915 The original high boundary, the hard limit, is defined as a strict
1916 limit that can not budge, even if the OOM killer has to be called.
1917 But this generally goes against the goal of making the most out of the
1918 available memory. The memory consumption of workloads varies during
1919 runtime, and that requires users to overcommit. But doing that with a
1920 strict upper limit requires either a fairly accurate prediction of the
1921 working set size or adding slack to the limit. Since working set size
1922 estimation is hard and error prone, and getting it wrong results in
1923 OOM kills, most users tend to err on the side of a looser limit and
1924 end up wasting precious resources.
1926 The memory.high boundary on the other hand can be set much more
1927 conservatively. When hit, it throttles allocations by forcing them
1928 into direct reclaim to work off the excess, but it never invokes the
1929 OOM killer. As a result, a high boundary that is chosen too
1930 aggressively will not terminate the processes, but instead it will
1931 lead to gradual performance degradation. The user can monitor this
1932 and make corrections until the minimal memory footprint that still
1933 gives acceptable performance is found.
1935 In extreme cases, with many concurrent allocations and a complete
1936 breakdown of reclaim progress within the group, the high boundary can
1937 be exceeded. But even then it's mostly better to satisfy the
1938 allocation from the slack available in other groups or the rest of the
1939 system than killing the group. Otherwise, memory.max is there to
1940 limit this type of spillover and ultimately contain buggy or even
1941 malicious applications.
1943 Setting the original memory.limit_in_bytes below the current usage was
1944 subject to a race condition, where concurrent charges could cause the
1945 limit setting to fail. memory.max on the other hand will first set the
1946 limit to prevent new charges, and then reclaim and OOM kill until the
1947 new limit is met - or the task writing to memory.max is killed.
1949 The combined memory+swap accounting and limiting is replaced by real
1950 control over swap space.
1952 The main argument for a combined memory+swap facility in the original
1953 cgroup design was that global or parental pressure would always be
1954 able to swap all anonymous memory of a child group, regardless of the
1955 child's own (possibly untrusted) configuration. However, untrusted
1956 groups can sabotage swapping by other means - such as referencing its
1957 anonymous memory in a tight loop - and an admin can not assume full
1958 swappability when overcommitting untrusted jobs.
1960 For trusted jobs, on the other hand, a combined counter is not an
1961 intuitive userspace interface, and it flies in the face of the idea
1962 that cgroup controllers should account and limit specific physical
1963 resources. Swap space is a resource like all others in the system,
1964 and that's why unified hierarchy allows distributing it separately.