Welcome to LWN.net
The following subscription-only content has been made available to you by an LWN subscriber. Thousands of subscribers depend on LWN for the best news from the Linux and free software communities. If you enjoy this article, please consider subscribing to LWN. Thank you for visiting LWN.net!
While there are several rootkits that target Linux, they have so far not fully embraced the open-source ethos typical of Linux software. Luckily, Matheus Alves has been working to remedy this lack by creating an open-source rootkit called Singularity for Linux systems. Users who feel their computers are too secure can install the Singularity kernel module in order to allow remote code execution, disable security features, and hide files and processes from normal administrative tools. Despite its many features, Singularity is not currently known to be in use in the wild — instead, it provides security researchers with a testbed to investigate new detection and evasion techniques.
Alves is quite emphatic about the research nature of Singularity, saying that
its main purpose is to help drive security research forward by demonstrating
what is currently possible. He
calls for anyone using the software to "be a
researcher, not a criminal
", and to test it only on systems where they have
explicit permission to test. If one did wish to use Singularity for nefarious
purposes, however, the code is MIT licensed and freely available — using it in
that way would only be a crime, not an instance of copyright infringement.
The whole problem of how to obtain root permissions on a system and go about installing a kernel module is out of scope for Singularity; its focus is on how to maintain an undetected presence in the kernel once things have already been compromised. In order to do this, Singularity goes to a lot of trouble to present the illusion that the system hasn't been modified at all. It uses the kernel's existing Ftrace mechanism to hook into the functions that handle many system calls and change their responses to hide any sign of its presence.
Using Ftrace offers several advantages to the rootkit; most importantly, it means that the rootkit doesn't need to change the CPU trap-handling vector for system calls, which was one of the ways that some rootkits have been identified historically. It also avoids having to patch the kernel's functions directly — kernel functions already have hooks for Ftrace, so the rootkit doesn't need to perform its own ad-hoc modifications to the kernel's machine code, which might be detected. The Ftrace mechanism can be disabled at run time, of course — so Singularity helpfully enables it automatically and blocks any attempts to turn it off.
Singularity is concerned with hiding four classes of things: its own presence, the existence of attacker-controlled processes, network communication with those processes, and the files that those processes use. Hiding its own presence is actually fairly straightforward: when the kernel module is loaded, it resets the kernel's taint marker and removes itself from the list of active kernel modules. This also means that Singularity cannot be unloaded, since it doesn't appear in the normal interfaces that are used for unloading kernel modules. It also blocks the loading of subsequent kernel modules (although they will appear to load — they'll just silently fail). Consequently, Alves recommends experimenting with Singularity in a virtual machine.
Hiding processes, on the other hand, is more complicated. The mechanism that Singularity uses starts by identifying and remembering which processes are supposed to be hidden. Singularity uses a single 32-entry array of process IDs to track attacker-controlled processes; this is because a more sophisticated data structure would introduce more opportunities for the rootkit to be caught, either by adding additional memory allocations that could be noticed, or by introducing delays whenever one of its hooked functions needs to check the list of hidden process IDs.
Singularity supports two ways to add processes to the list: by sending an unused signal, or by setting a special environment variable and launching a new process. To implement the former, it hooks the kill() system call to detect an unused signal (number 59, by default), quashes the signal, adds the target process to its internal list, and gives the process root permissions in the global namespace. This means that attacker-controlled processes can be added from inside containers, and automatically escape the container using their new root privileges. To handle the environment variable, the execve() system call is hooked in a similar way.
Once a process is in the list, attempts to send signal 0 (to check whether the process exists) are also intercepted, as are other system calls that could refer to the process, such as getpgid(), sched_getaffinity(), and others. The total number of processes on the system, as reported by sysinfo() is also decremented to keep things consistent. The process's files in /proc are hidden by Singularity's file-hiding code. That code is probably the trickiest part of the whole rootkit. The basic idea is to filter out hidden directory entries such that the filesystem appears to remain in a consistent state, but filesystem code is difficult to get right at the best of times.
When a program calls getdents(), the kernel fills the provided buffer with directory entries as normal. Then, Singularity's hook copies the buffer back from user memory, removes the hidden entries, puts the modified buffer back in user memory, and changes the return value of the system call to reflect the smaller number of directory entries returned. This slightly complicated process is because the kernel doesn't provide a good place for Singularity to inject a hook before the directory entries are written to user memory the first time. So, one potential way to identify the rootkit is to have another thread race with the attempt to read directory entries, trying to spot any that were removed.
Changing the number of returned directory entries alone would make the system appear to be in an inconsistent state, however. Directories in Linux filesystems are supposed to track the number of references to them; this includes the ".." references inside child directories. So, when hiding a directory, Singularity also needs to intercept calls to stat() in order to adjust the number of visible links to its parent directory.
Direct access to hidden directories, in the form of openat() and related system calls, is also made to fail. readlink() poses a special challenge because it resolves symbolic links without actually opening them; it has to be handled separately. In addition to the procfs files of hidden processes, Singularity also hides any directories matching a set of user-supplied patterns. By default, it hides things named "singularity", but the project's documentation suggests changing this in the build configuration, since otherwise detecting the rootkit becomes straightforward.
Despite this sophisticated file-hiding machinery, Singularity doesn't help against forensic examinations of a hard disk from another computer. If it isn't installed in the running kernel, it can't hide anything. Therefore, the documentation also recommends putting as many hidden files as possible onto temporary filesystems stored in RAM, so that they don't show up after the system is rebooted.
Another problem for the rootkit is files that contain traces of its presence, but that would raise eyebrows if they disappeared entirely. This includes things like the system log, but also files in procfs like kallsyms or enabled_functions that expose which kernel functions have had Ftrace probes attached. For those files, Singularity doesn't hide them at the filesystem level, but it does filter calls to read() to hide incriminating information.
Deciding which log lines are incriminating isn't a completely solved problem, though. Right now, Singularity relies on matching a set of known strings. This is another place where users will have to customize the build to avoid simple detection methods.
Even once an attacker's processes can hide themselves and their files, it is still usually desirable to communicate information back to a command-and-control server. Singularity will work to hide network connections using a specific TCP port (8081, by default), and hide packets sent to and from that port from packet captures. It supports both IPv4 and IPv6. Hiding the connections from tools like netstat uses the same filesystem-hiding code as before. Hiding things from packet captures requires hooking into the kernel's packet-receiving code.
On the other hand, this is another place where Singularity can't control the observations of uncompromised computers: if one is running a network tap on another computer, the packets to and from Singularity's hidden port will be totally visible.
Singularity only supports x86 and x86_64, but it does support both 64-bit and 32-bit system call interfaces. This is important, because otherwise a 32-bit application running on top of a 64-bit kernel could potentially see different results, which would be suspicious. To avoid this, Singularity inserts all of the aforementioned Ftrace hooks twice, once on the 32-bit system call and once on the 64-bit system call. A generic wrapper function converts from the 32-bit calling convention to the 64-bit calling convention before forwarding to the actual implementation of the hook.
Singularity has been tested on a variety of 6.x kernels, including some versions shipped by Ubuntu, CentOS Stream, Debian, and Fedora. Since the tool primarily uses the Ftrace interface, it should be supported on most kernels — although since it interfaces with internal details of the kernel, there is always the chance that an update will break things.
The tool also comes bundled with a set of utility scripts for cleaning up evidence that it was installed in the first place. These include a script that mimics normal log-rotation behavior, except that it silently truncates the logs to hinder analysis; a script that securely shreds a source-code checkout in case the module was compiled locally; and a script that automatically configures the rootkit's module to be loaded on boot.
Overall, Singularity is remarkably sneaky. If someone didn't know what to look for, they would probably have trouble identifying that anything was amiss. The rootkit's biggest tell is probably the way that it prevents Ftrace from being disabled; if one writes "0" to /proc/sys/kernel/ftrace_enabled and the content of the file remains "1", that's a pretty clear sign that something is going on.
Readers interested in fixing that limitation are welcome to submit a pull request to the project; Alves is interested in receiving bug fixes, suggestions for new evasion techniques, and reports of working detection methods. The code itself is simple and modular, so it is relatively easy to adapt Singularity for one's own purposes. Perhaps having such a vivid demonstration of what is possible to do with a rootkit will inspire new, better detection or prevention methods.