Finding people's use of /usr/bin/Python with the Linux audit framework

Our Ubuntu systems have had a /usr/bin/python that was Python 2 for more or less as long as we've had Ubuntu systems, which by now is more than fifteen years. Over that time, our users have written a certain amount of Python 2 programs that use '#!/usr/bin/python' to get Python 2, because that's been the standard way to do it for a relatively long time. However, Python 2 is going away on Ubuntu since it has on Debian, and as part of that we're probably going to stop having a /usr/bin/python in our future 24.04 LTS servers. It would be nice to find out which of our users are still using '/usr/bin/python' so that we can contact them in advance and get them either to move their programs to Python 3 or at the very least start using '#!/usr/bin/python2'. One way to do this is to use the Linux kernel's audit framework. Or, really, two ways, the broad general way and the narrow specific way. Unfortunately neither of these are ideal.

The ideal option we would like is an audit rule for 'if /usr/bin/python is being used as the first argument to execve()', or equivalently 'if the name /usr/bin/python is being accessed in order to execute it'. Unfortunately, as far as I can tell you can't write either of these potential audit rules, although it may appear that you can.

The narrow specific way is to set a file audit on '/usr/bin/python' for read access, and then post-process the result to narrow it down to suitable system calls. For example:

-w /usr/bin/python -p r -k bin-python-exec

When you run a program that has a '#!/usr/bin/python', it will result in an audit log line like:

type=SYSCALL msg=audit(1672884109.008:233812): arch=c000003e syscall=89 success=yes exit=7 a0=560a65335980 a1=7fffe74c59e0 a2=1000 a3=560a632ba4e0 items=1 ppid=183379 pid=184601 auid=915 uid=915 gid=1010 euid=915 suid=915 fsuid=915 egid=1010 sgid=1010 fsgid=1010 tty=pts1 ses=6837 comm="fred" exe="/usr/bin/python2.7" subj=unconfined key="bin-python-exec"

Syscall 89 is (64-bit x86) readlink() (per this table), which in this case is being done inside the kernel as part of an execve() system call. The 'exe=' being Python 2.7 means that this can't be a readlink() call being done by some other program (ls, for example); however, we can't tell this from someone running Python 2.7 themselves and doing 'os.readlink("/usr/bin/python")'. This last case is probably sufficiently uncommon that you can not worry about it, and just contact the person in question (obtained from the uid= value) to let them know.

(One drawback of this narrow specific way is that you may not be able to tell people very much about what program of theirs is still using '/usr/bin/python'. The comm= value tells you what it's more or less called, but you don't have the specific path, although often you can dig it out by decoding the associated 'type=PROCTITLE' audit line for this audit record.)

Using '-p x' to trigger this on 'execute' in the audit rule doesn't work, because as far as the audit framework is concerned the symbolic link here is not being executed, it's being read (this is the same trap as I ran into when I worked out how to use the audit framework to find 32-bit programs).

The other approach, the broad general way, is to start by auditing execve(), possibly limited to execve() of /usr/bin/python2.7. I'm using a filter key option as good general practice, but we're going to see that it's not actually important:

-a always,exit -F arch=b64 -S execve -F path=/usr/bin/python2.7 -k bin-python-exec
-a always,exit -F arch=b32 -S execve -F path=/usr/bin/python2.7 -k bin-python-exec

(You can leave out the second line if you don't have to worry about 32-bit x86 programs.)

This will get you a set of audit records every time Python 2 gets executed, either directly or via some symlink. Starting from these, you want to pick out the records where "/usr/bin/python" is the initial argument to execve. The relevant lines from these records will look like this:

type=SYSCALL msg=audit(1672886976.075:237851): arch=c000003e syscall=59 success=yes exit=0 a0=55fa1621dd90 a1=55fa1621ddf0 a2=55fa16220e40 a3=8 items=3 ppid=183379 pid=189501 auid=915 uid=915 gid=1010 euid=915 suid=915 fsuid=915 egid=1010 sgid=1010 fsgid=1010 tty=pts1 ses=6837 comm="fred" exe="/usr/bin/python2.7" subj=unconfined key="bin-python-exec"
type=EXECVE msg=audit(1672886976.075:237851): argc=2 a0="/usr/bin/python" a1="/tmp/fred"

The 'type=EXECVE' record's 'a0=' value tells you that execve() was called on the /usr/bin/python symlink, instead of eg /usr/bin/python2. To get the user ID of the person doing this, you need to look back to the corresponding 'type=SYSCALL' record for the execve, which has a matching msg= value. Unfortunately as far as I know the audit system can't directly match type=EXECVE records for you. The second argument of the EXECVE record will generally tell you what Python program is being run, which you can pass on to the user involved.

The advantage of the broad general way is that you may already be tracing execve() system calls for general system auditing purposes. If you are, you can exploit your existing auditing logs by just searching for the relevant EXECVE lines.

Because Linux's audit framework is quite old by now, it's everywhere and all of the programs and components work. However, these days it's probably not the best tool for this sort of narrowly scoped question. Instead, I suspect that something using eBPF tracing is a better approach these days, even though various aspects of the eBPF tools are still works in progress, even on relatively recent Linux distributions.

(I'm still a little bit grumpy that both Ubuntu 22.04 LTS and Fedora 36 broke bits of bpftrace for a while, and I believe 22.04 LTS still hasn't fixed them. We're better than we were in 2020, but still not great, and then there's problems with kernel lockdown mode in some environments.)

from Hacker News https://ift.tt/KhrFBe8