Why does Cuckoo need CAP_SYS_ADMIN?

[OpenSourcerer 1363]

In reference to the new AirVPN Suite alpha with Cuckoo bundled:

Why do you need the CAP_SYS_ADMIN capability for cuckoo? Nothing in the description of that capability points at something cuckoo would need, so I'm very reluctant to do in the airvpn-suite-beta-bin AUR package what your install script does.

Edited ... by OpenSourcerer
Beautify intro sentence following the split

[Staff 9768]

On 9/18/2023 at 12:30 PM, OpenSourcerer said:

Erm… why do you need the CAP_SYS_ADMIN capability for cuckoo?

Hello!

Any user in any Linux distribution can run a process with CAP_SYS_ADMIN capability. The installation script makes sure that "airvpn" has that ability too, just in case some distribution erroneously has not set the ability to airvpn. You can check for example with:

sudo capsh --print --user=<any existing user>

Quote

Nothing in the description of that capability points at something cuckoo would need,

As you might have read, traffic splitting in this implementation relies on namespaces, so CAP_SYS_ADMIN is strictly necessary for setns() - immediately after, setuid is disabled. On top of that cuckoo drops all setuid privileges and sets those of the user running it. Verify this trivially as a cross/double-check during the tests.
 

Quote

I'm very reluctant to do in the airvpn-suite-beta-bin AUR package what your install script does.

We can't understand your point in this case. Anyway, we would be reluctant too but for a very different reason, i.e. is it appropriate to prepare an AUR package with "beta" in its name for a software which is not beta but alpha?

Kind regards
 

[OpenSourcerer 1363]

6 hours ago, Staff said:

Any user in any Linux distribution has the CAP_SYS_ADMIN capability

Tested on Ubuntu 20 and Armbian 23.08 (so assuming Debian does the same), SLES 12 and 15 and RHEL 7 and 8 (and my Arch machines, of course), with neither of these I had this cap by default. I must ask about the source of this info.

$ capsh --current
Current: =
Current IAB:
$ capsh --has-p=cap_sys_admin
cap[cap_sys_admin] not permitted

.

6 hours ago, Staff said:

As you might have read, traffic splitting in this implementation relies on namespaces, so CAP_SYS_ADMIN is strictly necessary for setns() - immediately after, setuid is disabled. On top of that cuckoo drops all setuid privileges and sets those of the user running it. Verify this trivially as a cross/double-check during the tests.

You can use namespaces of other processes using nsenter, though, can't you? The only difference, if I understood correctly, is that the caller itself is not associated with the target namespace (rather forks, then execs), while with setns() caller and children are. And it doesn't seem to need that cap. Here's where my understanding ends, though.

Interestingly, my research turned up that this cap and the lack of documentation surrounding it, especially for kernel devs coding a new kernel feature, is a constant source of grievances. LWN once suggested to rename this CAP_AS_GOOD_AS_ROOT. It's grants so many permissions now with no alternatives. Rather entertaining.
 

6 hours ago, Staff said:

We can't understand your point in this case. Anyway, we would be reluctant too but for a very different reason, i.e. is it appropriate to prepare an AUR package with "beta" in its name for a software which is not beta but alpha?

Heh. I once had only two packages tracking stable, one using git sources with ABS and the other using binaries. But then one of your users complained that I was allegedly tracking a very old version, despite new ones being announced and published in the last months at that time – they were pre-release versions, though. Looking at other packages and the packaging guidelines I saw that generally in AUR people separate releases (tracking tags) from pre-releases (tracking main or dev branches). But this is something I cannot follow in AirVPN's case: You don't publish the source for pre-releases, so a -git package base makes no sense.

So, I was made aware of the interest in tracking your pre-built pre-release binaries as published in Announcements. And I wrote in the comments that for those interested in all binary publications, pre-release or not, I will be creating a special package base which will track the newest version made public by you, which turned out to be airvpn-suite-beta-bin, for lack of a better -git naming. Yes, the name really is bad in many ways, and I need ideas for a better name, but I've not seen any positive examples of handling this specific situation in the AUR. The best convention, as written, is using -git.
I wrote it many times, I will write it again: Your approach to versioning in git is really odd and nonstandard. It simply boggles the mind why you refrain from using tags for your releases and squash all changes into a jumbo commit with the version as comment.

Also, yes, the cuckoo part of the suite is alpha – but the rest of the bundled software is not. You can build airvpn-suite-beta-bin and still find yourself in a situation where Bluetit will launch and Goldcrest can be used to manage it.
 

6 hours ago, Staff said:

Verify this trivially as a cross/double-check during the tests.

So with the above written, I would gladly do that, but as things stand I'd essentially execute closed-source software with a rather haunting kernel capability LWN suggested to name "CAP_AS_GOOD_AS_ROOT"… yes, Bluetit runs as root. Yes, Hummingbird even refuses showing the help without root. Yes, people run this daily. Yes, many people execute them on an obvious base of trust.
But the code for what it'd do as root is there, isn't it? What does Cuckoo do as as good as root?…

[Staff 9768]

17 hours ago, OpenSourcerer said:

Tested on Ubuntu 20 and Armbian 23.08 (so assuming Debian does the same), SLES 12 and 15 and RHEL 7 and 8 (and my Arch machines, of course), with neither of these I had this cap by default. I must ask about the source of this info.

$ capsh --current
Current: =
Current IAB:
$ capsh --has-p=cap_sys_admin
cap[cap_sys_admin] not permitted

.
 

Hello!
That output is correct, and it does not imply what you assume, but only that the program you have just launched runs in an ambient which does not have the specified vector raised. For your verification see our previous command, or just verify for each user the capability. Example (as root):

capsh --user=<username> --has-p=CAP_SYS_ADMIN ; echo $?

It will exit with status 1 if the ambient vector has not that capability, it will exit with status 0 if it has.
Please note that the whole new Suite would work anyway, in all the distributions we tested, if the installer doesn't edit /etc/security/capability.conf but we deem that this is a nice feature anyway, as it might be useful in some obscure distribution, and it adds clarity.

 

17 hours ago, OpenSourcerer said:

You can use namespaces of other processes using nsenter, though, can't you?

We can't: nsenter links the process you run to some existing PID, making it a child of some already existing process in the namespace, so nsenter has different usage for quite different purposes.

 

Quote

Interestingly, my research turned up that this cap and the lack of documentation surrounding it, especially for kernel devs coding a new kernel feature, is a constant source of grievances. LWN once suggested to rename this CAP_AS_GOOD_AS_ROOT. It's grants so many permissions now with no alternatives. Rather entertaining.

Yes, that obsolete paper anyway confirms how good this implementation is. Our use case is exactly one of the few perfectly proper, correct and needed "usages" of CAP_SYS_ADMIN without doubts. Additionally, all the worries of the original writer have been properly addressed, as explained (we paste here to readers' comfort again):

Quote

Immediately after, setuid is disabled. On top of that cuckoo drops all setuid privileges and sets those of the user running it. Verify this trivially as a cross/double-check during the tests.

 addressing the issues raised by Eklektix and Kerrisk and others. To clarify: verify with ps.

Kind regards




 

[OpenSourcerer 1363]

6 hours ago, Staff said:

Hello!
That output is correct, and it does not imply what you assume, but only that the program you have just launched runs in an ambient which does not have the specified vector raised. For your verification see our previous command, or just verify for each user the capability. Example (as root):

capsh --user=<username> --has-p=CAP_SYS_ADMIN ; echo $?

It will exit with status 1 if the ambient vector has not that capability, it will exit with status 0 if it has.
Please note that the whole new Suite would work anyway, in all the distributions we tested, if the installer doesn't edit /etc/security/capability.conf but we deem that this is a nice feature anyway, as it might be useful in some obscure distribution, and it adds clarity.

 

But this doesn't make sense at all! I mean, look, an easier example.

$ whoami
gigan3rd
$ id gigan3rd
uid=1000(gigan3rd) gid=1000(gigan3rd) groups=1000(gigan3rd),…
$ sudo capsh --cap-uid=1000 --has-p=CAP_NET_BIND_SERVICE; echo $?
0

By your definition, 0 = I have it. And yet:

$ nc -vl 0.0.0.0 30
Permission denied
$ nc -vl 0.0.0.0 2019
Listening on 0.0.0.0 2019
^C

Is it the difference between effective, permitted and inherited caps? I don't get how I can't open a privileged port in userspace when I supposedly have that cap, but Bluetit can set up a namespace in the background just fine (is it because Bluetit runs as root, perchance?).

---

No matter. Tested cuckoo without /etc/security.conf on Arch, found that it works. Will rollout without it.

[Staff 9768]

@OpenSourcerer

Hello!

There is some confusion on a few Linux concepts and architectural design in your last message which would require some longer explanation or a course-like series of articles. We're afraid that this thread could go off rails and on a long question/answer/question/answer "ping pong" which might be detrimental to the original purpose: community testing and bug reporting. Please feel free to ask your questions on some other forum, for example in "Off Topic" community forum and we'll do our best to explain, or maybe someone from the community will explain even better. We want to leave this thread (remember we're in "News and announcement") aimed at AirVPN Suite 2 preview version(s) community testing and bug reporting, thank you in advance for your understanding. 😉

Kind regards
 

[fsy 27]

14 hours ago, OpenSourcerer said:

But this doesn't make sense at all! I mean, look, an easier example.

$ whoami
gigan3rd
$ id gigan3rd
uid=1000(gigan3rd) gid=1000(gigan3rd) groups=1000(gigan3rd),…
$ sudo capsh --cap-uid=1000 --has-p=CAP_NET_BIND_SERVICE; echo $?
0

By your definition, 0 = I have it. And yet:

$ nc -vl 0.0.0.0 30
Permission denied
$ nc -vl 0.0.0.0 2019
Listening on 0.0.0.0 2019
^C

Is it the difference between effective, permitted and inherited caps? I don't get how I can't open a privileged port in userspace when I supposedly have that cap, but Bluetit can set up a namespace in the background just fine (is it because Bluetit runs as root, perchance?).

 

It makes a lot of sense. Your nc (check the actual binary, not the symlink) does not have the permission to do that because it does not have the CAP_NET_BIND_SERVICE bit set.
Sexy Staff asked to keep this thread on topic so you can split and put up this message where you want to. but this is too huge and embarrassing to ignore it... I have become accustomed to the exotic fauna of wise Win-like-dunces around many Linux distributions and I know that an immediate clarification can deter other dunces adding up to the commotion with wacky arguments, so here it is! 😂
Also to save staff from keeping a class on capabilities and kernel 😅, read here the 1st answer https://unix.stackexchange.com/questions/389879/how-to-set-capabilities-with-setcap-command and then check the man and the docs.
Set the bit and see that the user you mentioned in your example can run netcat listening to any port. Over and out.
 

[revsplus 7]

Hi,
good to know, interesting method. Simple and effective: so if I got it, the answer in short to the OP question "Why does Cuckoo need CAP_SYS_ADMIN?" under a design perspective is that developers don't want that cuckoo runs with root privileges, OTOH it must be able to force running anything in a network namespace through setns() system call, and must be launched only by airvpn group users who typically have well bound privileges. Is my interpretation correct? I'm more a BSD guy but always curious about tortuous Linux twists of old Unix features.
All the best!
 

[OpenSourcerer 1363]

2 hours ago, revsplus said:

I'm more a BSD guy but always curious about tortuous Linux twists of old Unix features.

How is this handled on BSD?

[revsplus 7]

16 hours ago, OpenSourcerer said:

How is this handled on BSD?

For compartmentalization or for sandboxing I would go straight to Jails, solving some problems with current approach of simple & single network namespace Suite designers are having with browsers. Jails are very powerful yet easy to configure and maintain and quite light. Just to say, you can run jails inside a jail, you can have vnet, you can run pf and bhyve (the FreeBSD powerful type 2 hypervisor) inside a jail, full ZFS support and much more.
For the fine grained "capabilities", if the old MAC modules and trustedBSD are not enough (they offer something like 400 different, specific privileges), then I would say capsicum, which "hybridizes a true capability-system model with a Unix design and POSIX API" in contrast to Linux "capabilities" which don't. In reality they sound more like privileges oddly renamed as "capabilities" in one of the Linux twists of names and features (here I will save you from other Linux horrors because such horrors are caused by distributions jungle and not by the kernel itself). capsicum compares to Linux seccomp as well but it is definitely superior as you might see.

A more comprehensive and I would say pleasant overview of Unix sandboxing techniques was illustrated in 2017 in this (in my opinion) good looking article. https://freebsdfoundation.org/wp-content/uploads/2017/10/A-Comparison-of-Unix-Sandboxing-Techniques.pdf

In the above article you can find a couple of paragraphs emblematic to understand the choice by Suite designers and probably instrumental to the OP (your) question:

Quote

For all of these reasons, seccomp-bpf alone is insufficient to truly sandbox arbitrary application code. A complete application sandbox must also use the Linux clone(2) system call to sequester a process within a new IPC namespace (to cut off access to the host's global System V IPC name-space), network namespace (interfaces, routing, firewall, /proc and /sys/class/net, etc.), mount namespace (similar to chroot(2)), and PID namespace (to cut off inappropriate uses of kill(2)). Creating such namespaces requires the CAP_SYS_ADMIN privilege, which is effectively equivalent to superuser privilege on Linux. Thus, creating an effective application sandbox on Linux requires running programs as root or creating setuid binaries.

I guess that's all clear now. In other words: if Suite designers want to keep traffic splitting as simple as requiring a single network namespace (I guess (?) to avoid the serious problem caused by Mullvad's cgroups based solution), with or without true process isolation / boosted virtualization they anyway need CAP_SYS_ADMIN for cuckoo tool. Here you can understand the need for precautions too, crucially and obviously: forbidding root running, dropping setuid privileges, setting the privileges of the user running cuckoo itself.

All the best!
-- revsplus

 

[flat4 79]

You guys are in deep space, however this is very interesting to read.

[OpenSourcerer 1363]

3 hours ago, flat4 said:

You guys are in deep space, however this is very interesting to read.

I may be, but I am sure stuggling to breathe.