Alex Tsariounov <alext@novell.com>
@@ -476,6 +750,7 @@ the cset command: cset(1), cset-shield(1), cset-set(1), and cse
more details. Additionally, the cset command has online help.
1. Introduction
In the Linux kernel, the cpuset facility provides a mechanism for creating @@ -491,27 +766,28 @@ end for the Linux cpusets functionality. Working with cpusets directly can be confusing and slightly complex. The cset tool hides that complexity behind an easy-to-use command line interface.
There are two distinct use cases for cset: the basic shielding use case and
-the "advanced" case of using raw set and proc subcommands. The basic
-shielding function is accessed with the shield subcommand and described in
-the next section. Using the raw set and proc subcommands allows one to
+the "advanced" case of using raw set and proc subcommands. The basic
+shielding function is accessed with the shield subcommand and described in
+the next section. Using the raw set and proc subcommands allows one to
set up arbitrarily complex cpusets and is described in the later sections.
Note that in general, one either uses the shield subcommand or a -combination of the set and proc subcommands. One rarely, if ever, uses +
Note that in general, one either uses the shield subcommand or a
+combination of the set and proc subcommands. One rarely, if ever, uses
all of these subcommands together. Doing so will likely become too
-confusing. Additionally, the shield subcommand sets up its required cpusets
+confusing. Additionally, the shield subcommand sets up its required cpusets
with exclusively marked CPUs. This can interfere with your cpuset
strategy. If you find that you need more functionality for your strategy than
-shield provides, go ahead and transition to using set and proc
-exclusively. It is straightforward to implement what shield does with a few
-extra set and proc subcommands.
1.1. Obtaining Online Help
+shield provides, go ahead and transition to using set and proc
+exclusively. It is straightforward to implement what shield does with a few
+extra set and proc subcommands.1.1. Obtaining Online Help
- For a full list of cset subcommands
-
- # cset help +
# cset help - @@ -519,7 +795,7 @@ For in-depth help on individual subcommands
-
- # cset help <subcommand> +
# cset help <subcommand> - @@ -527,16 +803,19 @@ For options of individual subcommands
-
- # cset <subcommand> (-h | --help) +
# cset <subcommand> (-h | --help)
2. The Basic Shielding Model
Although any set up of cpusets can really be described as "shielding," there is one prevalent shielding model in use that is so common that cset has a -subcommand that is dedicated to its use. This subcommand is called shield.
shield.The concept behind this model is the use of three cpusets. The root cpuset which is always present in all configurations and contains all CPUs. The system cpuset which contains CPUs which are used for system tasks. These @@ -545,7 +824,7 @@ system. And finally, the user cpuset which contains CPUs which are use "important" tasks. The user cpuset is the shield. Only those tasks that are somehow important, usually tasks whose performance determines the overall rating for the machine, are run in the user cpuset.
The shield subcommand manages all of these cpusets and lets you define the +
The shield subcommand manages all of these cpusets and lets you define the
CPUs and Memory Nodes that are in the shielded and unshielded sets. The
subcommand automatically moves all movable tasks on the system into the
unshielded cpuset on shield activation, and back into the root cpuset on
@@ -553,17 +832,18 @@ shield tear down. The subcommand then lets you move tasks into and out of the
shield. Additionally, you can move special tasks (kernel threads) which
normally run in the root cpuset into the unshielded set so that your
shield will have even less disturbance.
The shield subcommand abstracts the management of these cpusets away from +
The shield subcommand abstracts the management of these cpusets away from
you and provides options that drive how the shield is set up, which tasks are
to be shielded and which tasks are not, and status of the shield. In fact,
you need not be bothered with the naming of the required cpusets or even where
-the cpuset filesystem is mounted. Cset and the shield subcommand takes
+the cpuset filesystem is mounted. Cset and the shield subcommand takes
care of all that.
If you find yourself needing to define more cpusets for your application, then
it is likely that this simple shielding is not a rich enough model for you.
-In this case, you should transition to using the set and proc subcommands
+In this case, you should transition to using the set and proc subcommands
described in a later section.
2.1. A Simple Shielding Example
+2.1. A Simple Shielding Example
Assume that we have a 4-way machine that is not NUMA. This means there are 4 CPUs at our disposal and there is only one Memory Node available. On such machines, we do not need to specify any memory node parameters to cset, it @@ -583,22 +863,23 @@ tasks that are running on the system.
2.1.1. Setup and Teardown of the Shield
To set up a shield of 3 CPUs with 1 CPU left for low priority system processing, issue the following command.
[zuul:cpuset-trunk]# cset shield -c 1-3 ++cset: "user" cpuset of CPUSPEC(1-3) with 0 tasks running[zuul:cpuset-trunk]# cset shield -c 1-3 cset: --> activating shielding: cset: moving 176 tasks from root into system cpuset... [==================================================]% cset: "system" cpuset of CPUSPEC(0) with 176 tasks running -cset: "user" cpuset of CPUSPEC(1-3) with 0 tasks running
This command does a number of things. First, a user cpuset is created with
-what’s called a CPUSPEC (CPU specification) from the -c/--cpu option. This
+what’s called a CPUSPEC (CPU specification) from the -c/--cpu option. This
CPUSPEC specifies to use CPUs 1 through 3 inclusively. Next, the command
-creates a system cpuset with a CPUSPEC that is the inverse of the -c
+creates a system cpuset with a CPUSPEC that is the inverse of the -c
option for the current machine. On this machine that cpuset will only contain
the first CPU, CPU0. Next, all userspace processes running in the root
cpuset are transfered to the system cpuset. This makes all those processes
@@ -607,16 +888,16 @@ through 3 and they are now idling.
Note that the command did not move the kernel threads that are running in the root cpuset to the system cpuset. This is because you may want these kernel threads to use all available CPUs. If you do not, the you can use the --k/--kthread option as described below.
-k/--kthread option as described below.The shield setup command above outputs the information of which cpusets were created and how many tasks are running on each. If you want to see the current status of the shield again, issue this command:
[zuul:cpuset-trunk]# cset shield ++cset: "user" cpuset of CPUSPEC(1-3) with 0 tasks running[zuul:cpuset-trunk]# cset shield cset: --> shielding system active with cset: "system" cpuset of CPUSPEC(0) with 176 tasks running -cset: "user" cpuset of CPUSPEC(1-3) with 0 tasks running
Which shows us that the shield is set up and that 176 tasks are running in the system cpuset—the "unshielded" cpuset.
[zuul:cpuset-trunk]# cset shield -k on ++cset: done[zuul:cpuset-trunk]# cset shield -k on cset: --> activating kthread shielding cset: kthread shield activated, moving 70 tasks into system cpuset... [==================================================]% -cset: done
You can see that this moved an additional 70 tasks to the unshielded system
-cpuset. Note that the -k/--kthread on parameter can be given at the shield
+cpuset. Note that the -k/--kthread on parameter can be given at the shield
creation time as well and you do not need to perform these two steps
separately if you know that you will want kernel thread shielding as well.
Executing cset shield again shows us the current state of the shield.
[zuul:cpuset-trunk]# cset shield ++cset: "user" cpuset of CPUSPEC(1-3) with 0 tasks running[zuul:cpuset-trunk]# cset shield cset: --> shielding system active with cset: "system" cpuset of CPUSPEC(0) with 246 tasks running -cset: "user" cpuset of CPUSPEC(1-3) with 0 tasks running
You can get a detailed listing of what is running in the shield by specifying
-either -s/--shield or -u/--unshield to the shield subcommand and using
+either -s/--shield or -u/--unshield to the shield subcommand and using
the verbose flag. You will get output similar to the following.
[zuul:cpuset-trunk]# cset shield --unshield -v ++ root 32653 25222 Roth python ./cset shield --unshield -v[zuul:cpuset-trunk]# cset shield --unshield -v cset: "system" cpuset of CPUSPEC(0) with 251 tasks running USER PID PPID SPPr TASK NAME -------- ----- ----- ---- --------- @@ -670,7 +951,7 @@ cset: "system" cpuset of CPUSPEC(0) with 251 tasks running root 84 2 Sf50 [IRQ-9] ... alext 31796 31789 Soth less - root 32653 25222 Roth python ./cset shield --unshield -v
Note that I abbreviated the listing; we do have 251 tasks running in the
system set. The output is self-explanatory; however, the "SPPr" field may
@@ -679,24 +960,26 @@ can see that the initial two tasks are Stopped and running in timeshare
priority, marked as "oth" (for "other"). The [IRQ-9] task is also stopped,
but marked at real time FIFO policy with a priority of 50. The last task in
the listing is the cset command itself and is marked as running. Also note
-that adding a second -v/--verbose option will not restrict the output to
+that adding a second -v/--verbose option will not restrict the output to
fit into an 80 character screen.
Tear down of the shield, stopping the shield in other words, is done with the
--r/--reset option to the shield subcommand. When this command is issued,
+-r/--reset option to the shield subcommand. When this command is issued,
both the system and user cpusets are deleted and any tasks that are
running in both of those cpusets are moved to the root cpuset. Once so
moved, all tasks will have access to all resources on the system. For
example:
[zuul:cpuset-trunk]# cset shield --reset ++cset: done[zuul:cpuset-trunk]# cset shield --reset cset: --> deactivating/reseting shielding cset: moving 0 tasks from "/user" user set to root set... cset: moving 250 tasks from "/system" system set to root set... [==================================================]% cset: deleting "/user" and "/system" sets -cset: done
2.1.2. Moving Interesting Tasks Into and Out of the Shield
Now that we have a shield running, the objective is to run our "important" processes in that shield. These processes can be anything, but usually they @@ -714,14 +997,15 @@ Move an already running task into the shield
Execing a Process into the Shield
-Running a new process in the shield can be done with the -e/--exec option -to the shield subcommand. This is the simplest way to get a task to run in +
Running a new process in the shield can be done with the -e/--exec option
+to the shield subcommand. This is the simplest way to get a task to run in
the shield. For this example, let’s exec a new bash shell into the shield
with the following commands.
[zuul:cpuset-trunk]# cset shield -s ++cset: done[zuul:cpuset-trunk]# cset shield -s cset: "user" cpuset of CPUSPEC(1-3) with 0 tasks running cset: done @@ -739,12 +1023,12 @@ cset: "user" cpuset of CPUSPEC(1-3) with 2 tasks running [zuul:cpuset-trunk]# cset shield -s cset: "user" cpuset of CPUSPEC(1-3) with 0 tasks running -cset: done
The first command above lists the status of the shield. We see that the shield is defined as CPUs 1 through 3 inclusive and currently there are no tasks running in it.
The second command execs the bash shell into the shield with the -e +
The second command execs the bash shell into the shield with the -e
option. The last message of cset lists the PID of the new process.
cset follows the tradition of separating the tool options from the
-command to be execed options with a double dash (--). This is not shown in
+command to be execed options with a double dash (--). This is not shown in
this simple example, but if the command you want to exec also takes options,
-separate them with the double dash like so: # cset shield -e mycommand -- -v
-The -v will be passed to mycommand, and not to cset. |
+separate them with the double dash like so:
The next command lists the status of the shield again. You will note that @@ -778,11 +1062,11 @@ and see that once again, it does not contain any tasks.
You may have noticed in the output above that both the new shell and the status command are running as the root user. This is because cset needs to run as root and so all it’s children will also run as root. If you need to -run a process under a different user and or group, you may use the --user -and --group options for exec as follows.
--user
+and --group options for exec as follows.[zuul:cpuset-trunk]# cset shield --user=alext --group=users -e bash ++ alext 14241 14212 Roth python ./cset shield -s -v[zuul:cpuset-trunk]# cset shield --user=alext --group=users -e bash cset: --> last message, executed args into cpuset "/user", new pid is: 14212 alext@zuul> cset shield -s -v @@ -790,21 +1074,23 @@ cset: "user" cpuset of CPUSPEC(1-3) with 2 tasks running USER PID PPID SPPr TASK NAME -------- ----- ----- ---- --------- alext 14212 8583 Soth bash - alext 14241 14212 Roth python ./cset shield -s -v
Moving a Running Task into and out of the Shield
While execing a process into the shield is undoubtably useful, most of the
time, you’ll want to move already running tasks into and out of the shield.
The cset shield subcommand includes two options for doing this:
--s/--shield and -u/--unshield. These options require what’s called a
-PIDSPEC (process specification) to also be specified with the -p/--pid
+-s/--shield and -u/--unshield. These options require what’s called a
+PIDSPEC (process specification) to also be specified with the -p/--pid
option. The PIDSPEC defines which tasks get operated on. The PIDSPEC can be
a single process ID, a list of process IDs separated by commas, and a list of
process ID ranges separated by dashes, groups of which are separated by
commas. For example:
-
---shield --pid 1234
+
--shield --pid 1234 -
@@ -812,7 +1098,7 @@ This PIDSPEC argument specifies that PID 1234 be shielded.
-
---shield --pid 1234,42,1934,15000,15001,15002
+
--shield --pid 1234,42,1934,15000,15001,15002 -
@@ -821,7 +1107,7 @@ shield.
-
---unshield -p 5000,5100,6010-7000,9232
+
--unshield -p 5000,5100,6010-7000,9232 -
@@ -845,8 +1131,8 @@ range.
Use of the appropriate PIDSPEC can thus be handy to move tasks and groups of
tasks into and out of the shield. Additionally, there is one more option that
-can help with multi-threaded processes, and that is the --threads flag. If
-this flag is present in a shield or unshield command with a PIDSPEC and if
+can help with multi-threaded processes, and that is the --threads flag. If
+this flag is present in a shield or unshield command with a PIDSPEC and if
any of the task IDs in the PIDSPEC belong to a thread in a process container,
then all the sibling threads in that process container will get shielded or
unshielded as well. This flag provides an easy mechanism to shield/unshield
@@ -855,7 +1141,7 @@ all threads of a process by simply specifying one thread in that process.
[zuul:cpuset-trunk]# echo $$ ++cset: done[zuul:cpuset-trunk]# echo $$ 22018 [zuul:cpuset-trunk]# cset shield -s -p 22010-22020 @@ -876,76 +1162,83 @@ cset: done [zuul:cpuset-trunk]# cset shield -s cset: "user" cpuset of CPUSPEC(1-3) with 0 tasks running -cset: done
|
Note
|
-Ordinarily, the shield option will shield a PIDSPEC only if it is -currently running in the system set—the unshielded set. The unshield + | Ordinarily, the shield option will shield a PIDSPEC only if it is
+currently running in the system set—the unshielded set. The unshield
option will unshield a PIDSPEC only if it is currently running in the user
-set—the shielded set. If you want to shield/unshield a process that
-happens to be running in the root set (not common), then use the --force
+set—the shielded set. If you want to shield/unshield a process that
+happens to be running in the root set (not common), then use the --force
option for these commands. |