doc:fix poststart doc by ningmingxiao · Pull Request #1259 · opencontainers/runtime-spec

ningmingxiao · 2024-07-13T13:09:17Z

I find poststart is in runc create stage

kolyshkin

This change is obviously wrong, or perhaps I'm missing something here?

Looking into runc sources, I see that poststart is run right after we start init, but before we tell it to continue. This is kind of what the current doc says.

ningmingxiao · 2024-07-13T18:35:39Z

This change is obviously wrong, or perhaps I'm missing something here?

Looking into runc sources, I see that poststart is run right after we start init, but before we tell it to continue. This is kind of what the current doc says.

what does "user-specified process" means ?

{
    "ociVersion": "1.0.2-dev",
    "process": {
            "terminal": false,
            "user": {
                    "uid": 0,
                    "gid": 0
            },
            "args": [
                    "sh"
            ],
....

does it refer to this run custom process ? In runc start

utam0k · 2024-07-14T04:12:21Z

what does "user-specified process" means ?

It means a container process. I've double-checked runc's code, and I found a poststart hook is kicked after a container process started.

https://github.com/opencontainers/runc/blob/3778ae603c706494fd1e2c2faf83b406e38d687d/libcontainer/container_linux.go#L350-L352

https://github.com/opencontainers/runc/blob/3778ae603c706494fd1e2c2faf83b406e38d687d/libcontainer/container_linux.go#L362-L367

utam0k · 2024-07-14T04:13:03Z

Feel free to re-open if there is anything you want to discuss. Thanks for your contribution 🙏

ningmingxiao · 2024-07-14T05:22:54Z

config.json

{
    "ociVersion": "1.0.2-dev",
    "process": {
            "terminal": false,
            "user": {
                    "uid": 0,
                    "gid": 0
            },
            "args": [
                    "sh"
            ],
            "env": [
                    "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                    "TERM=xterm"
            ],
            "cwd": "/",
            "capabilities": {
                    "bounding": [
                            "CAP_AUDIT_WRITE",
                            "CAP_KILL",
                            "CAP_NET_BIND_SERVICE"
                    ],
                    "effective": [
                            "CAP_AUDIT_WRITE",
                            "CAP_KILL",
                            "CAP_NET_BIND_SERVICE"
                    ],
                    "permitted": [
                            "CAP_AUDIT_WRITE",
                            "CAP_KILL",
                            "CAP_NET_BIND_SERVICE"
                    ],
                    "ambient": [
                            "CAP_AUDIT_WRITE",
                            "CAP_KILL",
                            "CAP_NET_BIND_SERVICE"
                    ]
            },
            "rlimits": [
                    {
                            "type": "RLIMIT_NOFILE",
                            "hard": 1024,
                            "soft": 1024
                    }
            ],
            "noNewPrivileges": true
    },
    "root": {
            "path": "rootfs",
            "readonly": false
    },
    "hostname": "runc",
    "mounts": [
            {
                    "destination": "/proc",
                    "type": "proc",
                    "source": "proc"
            },
            {
                    "destination": "/dev",
                    "type": "tmpfs",
                    "source": "tmpfs",
                    "options": [
                            "nosuid",
                            "strictatime",
                            "mode=755",
                            "size=65536k"
                    ]
            },
            {
                    "destination": "/dev/pts",
                    "type": "devpts",
                    "source": "devpts",
                    "options": [
                            "nosuid",
                            "noexec",
                            "newinstance",
                            "ptmxmode=0666",
                            "mode=0620"
                    ]
            },
            {
                    "destination": "/dev/shm",
                    "type": "tmpfs",
                    "source": "shm",
                    "options": [
                            "nosuid",
                            "noexec",
                            "nodev",
                            "mode=1777",
                            "size=65536k"
                    ]
            },
            {
                    "destination": "/dev/mqueue",
                    "type": "mqueue",
                    "source": "mqueue",
                    "options": [
                            "nosuid",
                            "noexec",
                            "nodev"
                    ]
            },
            {
                    "destination": "/sys",
                    "type": "sysfs",
                    "source": "sysfs",
                    "options": [
                            "nosuid",
                            "noexec",
                            "nodev",
                            "ro"
                    ]
            },
            {
                    "destination": "/sys/fs/cgroup",
                    "type": "cgroup",
                    "source": "cgroup",
                    "options": [
                            "nosuid",
                            "noexec",
                            "nodev",
                            "relatime",
                            "ro"
                    ]
            }
    ],
    "hooks": {
            "Poststart": [
                {
                    "path": "/usr/bin/sleep",
                    "args":  ["/usr/bin/sleep","10"]
                }
            ]
    },
    "linux": {
            "resources": {
                    "devices": [
                            {
                                    "allow": false,
                                    "access": "rwm"
                            }
                    ]
            },
            "uidMappings": [
                    {
                            "containerID": 0,
                            "hostID": 1000,
                            "size": 1
                    }
            ],
            "gidMappings": [
                    {
                            "containerID": 0,
                            "hostID": 1000,
                            "size": 1
                    }
            ],
            "namespaces": [
                    {
                            "type": "pid"
                    },
                    {
                            "type": "network"
                    },
                    {
                            "type": "ipc"
                    },
                    {
                            "type": "uts"
                    },
                    {
                            "type": "mount"
                    }
            ],
            "maskedPaths": [
                    "/proc/acpi",
                    "/proc/asound",
                    "/proc/kcore",
                    "/proc/keys",
                    "/proc/latency_stats",
                    "/proc/timer_list",
                    "/proc/timer_stats",
                    "/proc/sched_debug",
                    "/sys/firmware",
                    "/proc/scsi"
            ],
            "readonlyPaths": [
                    "/proc/bus",
                    "/proc/fs",
                    "/proc/irq",
                    "/proc/sys",
                    "/proc/sysrq-trigger"
            ]
    }
}

I set

            "Poststart": [
                {
                    "path": "/usr/bin/sleep",
                    "args":  ["/usr/bin/sleep","10"]
                }

[root@localhost mycontainer]# time runc create test0001

real    0m10.199s
user    0m0.010s
sys     0m0.035s

that means user-specified process doesn't run. (user-specified process will run in runc start stage). @utam0k
does user-specified process means following code ?

            "args": [
                    "sh"
            ],

cyphar · 2024-07-15T02:18:36Z

@utam0k process.start doesn't actually start the process, that's actually the implementation of runc create. runc start calls container.Exec. (Yes this is confusing, this happened because when we added runc create we changed the semantics of runc start and it seems we forgot to rename the internal functions properly.)

I think this is a runc bug.

@kolyshkin

Looking into runc sources, I see that poststart is run right after we start init, but before we tell it to continue. This is kind of what the current doc says.

That is what we are doing, but that's not what the spec says should happen AFAICS. The spec says that we should run poststart after the user-specified process is started (so during runc start after we signal the exec FIFO to start, not runc create as we are currently doing).

cyphar

If runc's behaviour doesn't match the spec then we need to fix runc.

utam0k · 2024-07-15T11:23:10Z

@utam0k process.start doesn't actually start the process, that's actually the implementation of runc create. runc start calls container.Exec. (Yes this is confusing, this happened because when we added runc create we changed the semantics of runc start and it seems we forgot to rename the internal functions properly.)

@cyphar @ningmingxiao I revised the runc's code and I found my misunderstanding. Thanks for pointing out it to me 🙏

If runc's behaviour doesn't match the spec then we need to fix runc.

+1

utam0k · 2024-07-15T11:23:26Z

I wonder if we should check other runtimes ...

ningmingxiao · 2024-07-15T11:41:23Z

I wonder if we should check other runtimes ...

crun have same problem

utam0k · 2024-07-15T11:48:14Z

I wonder if we should check other runtimes ...

crun have same problem

Thanks!

cc: @giuseppe

utam0k · 2024-07-15T11:51:10Z

I wonder if we should check other runtimes ...

crun have same problem

@ningmingxiao
youki doesn't have the problem, doesn't it?

$ time sudo ./youki create -b postStart-bug test
sudo ./youki create -b postStart-bug test  0.00s user 0.01s system 17% cpu 0.047 total

https://github.com/containers/youki/blob/9ec0cb2a7716615f259cfe9726cd9ca71ba507d6/crates/libcontainer/src/container/container_start.rs#L71-L78

utam0k · 2024-07-15T11:55:18Z

I wonder if we should check other runtimes ...

crun have same problem

@ningmingxiao
I couldn't reproduce it with crun. How did you check it?

$ time sudo crun create -b postStart-bug test
sudo crun create -b postStart-bug test  0.00s user 0.01s system 23% cpu 0.035 total

$ crun -v
crun version 1.15
commit: e6eacaf4034e84185fd8780ac9262bbf57082278
rundir: /run/user/1000/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL

ningmingxiao · 2024-07-15T12:05:19Z

{
    "ociVersion": "1.0.2-dev",
    "process": {
            "terminal": false,
            "user": {
                    "uid": 0,
                    "gid": 0
            },
            "args": [
                    "sh"
            ],
            "env": [
                    "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                    "TERM=xterm"
            ],
            "cwd": "/",
            "capabilities": {
                    "bounding": [
                            "CAP_AUDIT_WRITE",
                            "CAP_KILL",
                            "CAP_NET_BIND_SERVICE"
                    ],
                    "effective": [
                            "CAP_AUDIT_WRITE",
                            "CAP_KILL",
                            "CAP_NET_BIND_SERVICE"
                    ],
                    "permitted": [
                            "CAP_AUDIT_WRITE",
                            "CAP_KILL",
                            "CAP_NET_BIND_SERVICE"
                    ],
                    "ambient": [
                            "CAP_AUDIT_WRITE",
                            "CAP_KILL",
                            "CAP_NET_BIND_SERVICE"
                    ]
            },
            "rlimits": [
                    {
                            "type": "RLIMIT_NOFILE",
                            "hard": 1024,
                            "soft": 1024
                    }
            ],
            "noNewPrivileges": true
    },
    "root": {
            "path": "rootfs",
            "readonly": false
    },
    "hostname": "runc",
    "mounts": [
            {
                    "destination": "/proc",
                    "type": "proc",
                    "source": "proc"
            },
            {
                    "destination": "/dev",
                    "type": "tmpfs",
                    "source": "tmpfs",
                    "options": [
                            "nosuid",
                            "strictatime",
                            "mode=755",
                            "size=65536k"
                    ]
            },
            {
                    "destination": "/dev/pts",
                    "type": "devpts",
                    "source": "devpts",
                    "options": [
                            "nosuid",
                            "noexec",
                            "newinstance",
                            "ptmxmode=0666",
                            "mode=0620"
                    ]
            },
            {
                    "destination": "/dev/shm",
                    "type": "tmpfs",
                    "source": "shm",
                    "options": [
                            "nosuid",
                            "noexec",
                            "nodev",
                            "mode=1777",
                            "size=65536k"
                    ]
            },
            {
                    "destination": "/dev/mqueue",
                    "type": "mqueue",
                    "source": "mqueue",
                    "options": [
                            "nosuid",
                            "noexec",
                            "nodev"
                    ]
            },
            {
                    "destination": "/sys",
                    "type": "sysfs",
                    "source": "sysfs",
                    "options": [
                            "nosuid",
                            "noexec",
                            "nodev",
                            "ro"
                    ]
            },
            {
                    "destination": "/sys/fs/cgroup",
                    "type": "cgroup",
                    "source": "cgroup",
                    "options": [
                            "nosuid",
                            "noexec",
                            "nodev",
                            "relatime",
                            "ro"
                    ]
            }
    ],
    "hooks": {
            "Poststart": [
                {
                    "path": "/usr/bin/sleep",
                    "args":  ["/usr/bin/sleep","10"]
                }
            ]
    },
    "linux": {
            "resources": {
                    "devices": [
                            {
                                    "allow": false,
                                    "access": "rwm"
                            }
                    ]
            },
            "uidMappings": [
                    {
                            "containerID": 0,
                            "hostID": 1000,
                            "size": 1
                    }
            ],
            "gidMappings": [
                    {
                            "containerID": 0,
                            "hostID": 1000,
                            "size": 1
                    }
            ],
            "namespaces": [
                    {
                            "type": "pid"
                    },
                    {
                            "type": "network"
                    },
                    {
                            "type": "ipc"
                    },
                    {
                            "type": "uts"
                    },
                    {
                            "type": "mount"
                    }
            ],
            "maskedPaths": [
                    "/proc/acpi",
                    "/proc/asound",
                    "/proc/kcore",
                    "/proc/keys",
                    "/proc/latency_stats",
                    "/proc/timer_list",
                    "/proc/timer_stats",
                    "/proc/sched_debug",
                    "/sys/firmware",
                    "/proc/scsi"
            ],
            "readonlyPaths": [
                    "/proc/bus",
                    "/proc/fs",
                    "/proc/irq",
                    "/proc/sys",
                    "/proc/sysrq-trigger"
            ]
    }
}

I recheck crun it is ok.

lifubang · 2024-07-16T15:21:52Z

opencontainers/runc#4348 (comment)

Could you please help to check this in crun and youki? Thanks.

utam0k · 2024-07-19T11:11:06Z

opencontainers/runc#4348 (comment)

Could you please help to check this in crun and youki? Thanks.

As far as my investigation, the latest crun and youki couldn't reproduce this issue.

doc:fix poststart doc

9f229d1

ningmingxiao requested review from AkihiroSuda, crosbymichael, cyphar, dqminh, giuseppe, hqhq, kolyshkin, mrunalp, thaJeztah, tianon and utam0k as code owners July 13, 2024 13:09

kolyshkin requested changes Jul 13, 2024

View reviewed changes

utam0k closed this Jul 14, 2024

cyphar mentioned this pull request Jul 15, 2024

poststart run before user-specified process? #1260

Closed

cyphar reopened this Jul 15, 2024

cyphar requested changes Jul 15, 2024

View reviewed changes

cyphar mentioned this pull request Jul 15, 2024

runc's poststart behaviour doesn't match the runtime-spec opencontainers/runc#4347

Open

Conversation

ningmingxiao commented Jul 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kolyshkin left a comment

Choose a reason for hiding this comment

Uh oh!

ningmingxiao commented Jul 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

utam0k commented Jul 14, 2024

Uh oh!

utam0k commented Jul 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ningmingxiao commented Jul 14, 2024 • edited by utam0k Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cyphar commented Jul 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cyphar left a comment

Choose a reason for hiding this comment

Uh oh!

utam0k commented Jul 15, 2024

Uh oh!

utam0k commented Jul 15, 2024

Uh oh!

ningmingxiao commented Jul 15, 2024

Uh oh!

utam0k commented Jul 15, 2024

Uh oh!

utam0k commented Jul 15, 2024

Uh oh!

utam0k commented Jul 15, 2024

Uh oh!

ningmingxiao commented Jul 15, 2024

Uh oh!

lifubang commented Jul 16, 2024

Uh oh!

utam0k commented Jul 19, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ningmingxiao commented Jul 13, 2024 •

edited

Loading

ningmingxiao commented Jul 13, 2024 •

edited

Loading

utam0k commented Jul 14, 2024 •

edited

Loading

ningmingxiao commented Jul 14, 2024 •

edited by utam0k

Loading

cyphar commented Jul 15, 2024 •

edited

Loading