Detecting and Removing Rootkits

What the hell is a Rootkit?

( Definition from ). A rootkit is a collection of tools (programs) that a hacker uses to mask intrusion and obtain administrator-level access to a computer or computer network. The intruder installs a rootkit on a computer after first obtaining user-level access, either by exploiting a known vulnerability or cracking a password. The rootkit then collects userids and passwords to other machines on the network, thus giving the hacker root or privileged access.

A rootkit may consist of utilities that also: monitor traffic and keystrokes; create a "backdoor" into the system for the hacker's use; alter log files; attack other machines on the network; and alter existing system tools to circumvent detection.

If you think/believe your system is Rootkit'ed, read whole of this text before doing anything. If your system is already possessed, an extra 20 minutes will not change much. We know that the text is a bit long but we believe it is worth the effort; not because the text is a good one, but because the Rootkits are a very serious threat and you have to be very careful while removing them.

The procedures described below works for Mandrake distros. We have no experience with other distros at all. The procedures rely on the rpm tool.

Rootkits are nuisance and can be very serious threats to Linux boxes. Basically they exploit a kernel weakness (/dev/kmem exploit) to run incognito on a Linux machine, usually sniffing the network interfaces and recording critical information such as user IDs and passwords.

On a Rootkit infected machine, you will not be able to see any foreign processes by a "ps ax" command nor see an intruder using a "w" or "who" command. You shouldn't even think about finding anything useful in your log files.

What happens when your Linux box is Rootkit Infected

Checking your system against Rootkits

The Google literature mentions a chkrootkit utility. What the chkrootkit utility basically does is it compares the number of running processes reported by the ps command and the process maintenance directories under the /proc directory.

We prefer using a manual method which does almost exactly the same, but gives the admin's eyes a chance to view the running processes and decide whether a suspicious process is running. In the end, intruders need not use a predefined set of program names for their malicious code. Our technique to see the actual list of running processes is :

cd  /proc
for  i  in  `seq 1 33000`

test  -d  $i    &&   ls  -l  $i/exe

You must be root to issue this shell command(s). This shell script
On a rootkit'ed computer, you will get a listing which looks like (we have clipped the original list):

# for i in `seq 1 33000`; do test -d $i && ls -l $i/exe; done

lrwxrwxrwx    1 root     root            0 Jan  4 12:48 1/exe -> /sbin/initsk (deleted)
ls: cannot read symbolic link 2/exe: No such file or directory
lrwxrwxrwx    1 root     root            0 Jan  4 12:48 2/exe
ls: cannot read symbolic link 3/exe: No such file or directory
lrwxrwxrwx    1 root     root            0 Jan  4 12:48 3/exe
ls: cannot read symbolic link 4/exe: No such file or directory
. . .
. . .
. . .
lrwxrwxrwx    1 root     root            0 Jan  4 12:48 18580/exe -> /sbin/syslogd
lrwxrwxrwx    1 root     root            0 Jan  4 12:48 18588/exe -> /sbin/klogd
lrwxrwxrwx    1 root     root            0 Jan  4 12:48 20543/exe -> /var/tmp/.pink-sk/sniffsk
lrwxrwxrwx    1 root     root            0 Jan  4 12:48 20982/exe -> /usr/sbin/httpd2
lrwxrwxrwx    1 cayfer   cayfer          0 Jan  4 12:48 22232/exe -> /usr/bin/ssh

lrwxrwxrwx    1 cayfer   cayfer          0 Jan  4 12:48 22232/exe -> /usr/sbin/imapd

Now, you have to study this long list carefully for any "weird" processes that shouldn't be there. A very typical sign of infection is the string "(deleted)" on the line which belongs to the init process (or rather the fake one). If there is the mark "(deleted)" next to some init process, it means a tampered init program is running rather than the real one. On an infected machine, a tampered init starts during boot, makes necessary kernel modifications in the RAM to hide the tools the intruder will soon run, then forks the real init process.

Somewhere down the /proc listing, it is very likely that you will see processes with names like ".pink-sk", ".sniff-sk", ".pink-sniff" etc.etc. It is important that you do not rely on these names only (pink-sk is just an example); study the list thoroughly. We have seen malicious code running as "/bin/bin", "/usr/libexec/xyz".

If you find a weird named process, you have to be creative in finding ways to check whether they are part of the operating system or an application or some malicious code. A good way is to query the RPM (RedHat Packet Manager) databases for each process you suspect, to see whether they are part of a package; and then to see whether they are tampered with.

As an example, you can query the RPM database to see whether /usr/sbin/imapd (see above list, last line) is part of a distribution package (you should type the whole file path):

# rpm  -qf  /usr/sbin/imapd

In the above example, "rpm -qf" reports that the process running as  /usr/sbin/imapd is part of the imap-2002d-6mdk package.

If you want to check whether /usr/sbin/imapd is a modified copy (potentially malicious code), you can run the command

# rpm  -V  imap-2002d-6mdk
S.5....T c /etc/xinetd.d/ipop3

The report produced by "rpm -V" needs to be interpreted carefully.

The first part of the report line ( namely "S.5....T c" ) indicates that
Well, we were querying "/usr/sbin/imapd" but it seems that things have rather changed for "/etc/xinetd.d/imapd".

The catch is that both /usr/bin/imapd and /etc/xinetd.d/ipop3 files belong to the same package and only /etc/xinet.d/pop3 has been altered so far. For this specific example, changes are not critical; the altered file is a configuration file and probably the alterations were made by the sysadmin as part of fine tuning or whatever. Since /usr/sbin/imapd is not reported as a "modified" file, we know that it is safe to have it running on our system.

The rpm verification report flags that you might see are:

S : file Size has been changed
M : Mode has changed (such as permissions and file type) 
5 : MD5 sum has changed
U : Ownership has changed
G : Group ownership has changed
T : Timestamp has changed
c : Configuration file

(Please see "man rpm" for a few more flags.) Ownership, mode and checksum changes
for binaries are extremely ominous!

You have to go through the process list carefully, paying special attention to the paths of programs and not ony to names. If you see any program file name(s) starting with a dot, you have probably found it!

If you find malicious code running on your system

The first thing you need to do is to isolate the program file and its log file(s). The problem is that, although you can see the program file in the /proc listing, you will not be able to see it (them) on the disk or remove it (them). The modified kernel code in the RAM will hide these from other programs. You will have to reboot your system in single user mode, avoiding the modified init program to have access to these files.

The technique is fairly simple, but we must warn you that we have seen rootkits that remove whole directories of open files  during shutdown.   We strongly recommend that you do not "restart" or "shutdown" or "init 0" your system. Probably the malicious "init" program is removing directories from your system. Instead, do "Alt-Ctrl-Shift-SysRq-U" (unmount disks), then "Alt-Ctrl-Shift-SysRq-S" (sync) and finally "Alt-Ctrl-Shift-SysRq-B" (boot) to reboot. You will have to exit any X desktop manager you might be using and return to console mode. (Another good reason to set you system booting up to console mode rather than to a GUI logon screen). If these keys are not enabled on your Linux, just enter a couple of "sync" commands and turn the power off. Believe me, the risk is less... You might as well make backup copies of critical directories like /etc and /var and the homes before resetting or cycling the power.

While your system is restarting, hit the escape key or the Tab key (depending on whether your LILO is a gui one or a text screen one) and type in

linux  ro  init=/bin/bash

to the "boot:" prompt

The command causes the root partition to be mounted read only and the /bin/bash loaded into the memory instead of  init.

Wait till you get the root prompt. Once you get the root prompt, mount the /proc file system by typing

mount  /proc

If this is succesful (we do not know what to do if this step fails) remount the root file system as read-write using the command

mount -wo remount /

After that, start the devfsd daemon manually by typing

/sbin/devfsd  /dev

will handle /dev links to filesystems on your system.

Now you can either mount all filesystems mentioned in /etc/fstab with
mount  -a

or mount only the file systems you need with commands like

mount  -t  ext3   /dev/hda5    /usr                      etc. etc.

Your system is now partially up without the malicious init running; therefore the rootkit program files should now be visible to command like "ls", "rm", "find" etc. For the sample /proc listing above, you will need to mount the /var directory to see what is in /var/tmp/.pink-sk.

We strongly recommend that you examine each file in the directory where the rootkit is installed. For the binaries, you can use the

strings binary_file

command to get an idea of what other directories and files the malicious code might be using.

It is also very important to study the rootkit's log files to see what "user name-password" pairs the instruder has got hold of.

Reclaiming your system

Well, this is the most risky and tedious part. It is not possible to know exactly what damage the instruder has made to your system. The best and safest way is to install the whole system from scratch. Although it is relatively easy to build a whole opsys from scratch, the data/site specific applications will force you to think twice before formatting your disk.

  Our prefecerence is to give a try recovering the system. The procedure is:
  1. Checking /etc/passwd and /etc/shadow for any alien accounts.
  2. Changing the root password.
  3. Annulling passwords of EVERY single user account.
  4. Removing the rootkit's init, and restoring a clean init program.
  5. Scanning the whole disk space for SUID programs; especially those owned by root.
  6. Scanning for modified binary files (our experience indicates that the /sbin/init and Open SSL server and client binaries are almost always modified, plus some additional SUID programs are planted in less likely or brand new directories like /dev, /bin/libexec. SUID programs are files which when executed, runs with the privileges of the file's owner, regardless of the identity of the user who starts the program. A good example is the "passwd" command. Every user on the system is allowed to change their own passwords. Changing a password means modifying the /etc/shadow (sometimes /etc/passwd) file; which is normally not writable by anyone else but root. The solution is making the '/bin/passwd' file a SUID file and owned by root. When user "jack" runs the SUID password command, the program will run with the privileges of the owner of the /bin/passwd file, namely "root", rather than jack's privileges. Rootkits place some SUID files here and there owned by root, therefore even if you change the root password, they can run the program with root privileges if they can login to your system with a plain account. That's why you should annull each and every user password.
  7. Recovering modified binaries from distros or by reinstalling them from their tar balls.
First things first; change your root password, change your own password and manually edit /etc/shadow to annull all other user passwords. To annull an account's password, replace the password field on the /etc/shadow file with asterixes. Take your time to study the /etc/passwd file to see if there are any weird accounts, especially with weird home directories.

Rootkits usually keep a copy of the real init program in the /sbin directory. Afterall, they need this to fork into after their fake init does its work. So far, we have always found the clean init file as /sbin/init-sk. You might want to make sure that such a file is the real init by issuing a strings command to the binary and/or comparing it to a clean init file on another computer.

Scanning the system for SUID programs is easy; only if you can trust your "find" program. So, the next trivial step is to make sure the "find" command is not tampered with.

rpm  -qf  /bin/find

will report the distro package which contains the "find" command. For Mandrake systems, it is something like  "findutils-4.1.20-1mdk".

To re-install the find program file(s), you have to issue a command like

rpm  -Uvh  --replacepkgs  /your-distro-media/Mandrake/RPMS/findutils-4.1.20-1mdk

Having a clean find command, now you can issue the

find  /  -perm  +4000

command to find all SUID programs. You can use "find  /  -perm  +4000  -a  -user root"  to find SUID files owned by root (the really dangerous ones).

We are at a point where experience pays... The list of SUID programs can be quite long. Most of the SUID files you will see are files that really must be SUID. Knowing the real ones from malicious ones requires some experience and insight. Be paranoid! Double check whether a file should be SUID! Read the manual pages, refer to Google, ask more experienced sysadmins, compare with clean systems.

A safer and faster way of checking whether a file should be SUID or not, is comparing its attributes with the original in the distro package.

Suppose you want to check whether "/usr/bin/chage" should really be a SUID program file. All you need to do is compare "/usr/bin/chage" with the one in the original distribution. To do this, first you need to find out what distro package /usr/bin/chage belongs to:

# rpm -qf /usr/bin/chage

Once you know that the RPM package containing "/usr/bin/chage" is  "shadow-utils-4.0.3-5mdk", you need to run the command

# rpm -V shadow-utils-4.0.3-5mdk
..5....T c /etc/login.defs

to see any of the members of the package have been modified. The above example indicates that the "/etc/login.defs" member of the package has been modified. Since this is a config file, you might decide that your "chage" command is intact and really should be a SUID file. If it were the other way around, rpm would report a mode/size change for "usr/bin/chage".
If you come across a file for which you cannot be sure whether it should be a SUID file and be kept on the system at all, change its owner and group to "nobody" and make a note of this change. If later something fails, you can reconsider restoring the attributes of this file. To change the owner of a file, you can use a command like

chown  nobody:nobody  /usr/bin/somefile

Now it is time to scan the disk(s) for modified binary files...

We strongly recommend that you install the "init", "reboot" and OpenSSH server and client programs without even bothering testing.

First find the packages containing these programs and then re-install them:

# rpm -qf /sbin/init          

# rpm  -qf /sbin/reboot

# rpm  -qf  /usr/bin/ssh


# rpm  -qf  /usr/sbin/sshd

# rpm  -Uvh  --replacepkgs  /your-distro-media/Mandrake/RPMS/SysVinit-2.85-1mdk
# rpm  -Uvh  --replacepkgs  /your-distro-media/Mandrake/RPMS/openssh-clients-3.6.lp2-8mdk
# rpm  -Uvh  --replacepkgs  /your-distro-media/Mandrake/RPMS/openssh-server-3.6.lp2-8mdk

To check ALL your installed packages against the RPM database for integrity, you can use the

# rpm  -Va 

command. This command will compare all the system programs and applications against their RPM checksums and report any discrepancies. We strongly recommend that you re-install all packages which have altered binaries. Altered configuration files and shell scripts are not much of a risk but you might want to examine any altered scripts and/or configuration files before making a decision though.

If your system is continously attacked

We are fully aware that the procedures we have described are not complete or 100% corrective at all. You should not be shocked if you are rootkit'ed again. We can suggest a simple Perl program to help you catch the insisting intruder by continuously checking the /proc directory and stopping network services when an infection is detected. This will at least save the system log files.

while (1) {
  @out = `cd /proc; for i in \`seq 1 33000\`; do test -d \$i \&\& ls -l \$i/exe; done`;

  foreach $_ (@out) {
        # trap for "pink"rootkit
        if ( (/init/ && /deleted/i) || /pink/ || /-sk/ ) {
  # test every 15 seconds
  sleep 15;


sub stop_net {

       # notify sysadm with an email

       open(MAIL, "|/usr/sbin/sendmail -t");
       print MAIL "From: <user\\n";
       print MAIL "Subject: We are stoned!\n";
       print MAIL "To: admin\\n\n";
       print MAIL qq~
             We are rootkit'ted

       close MAIL;
       # let the mail to be delivered, 3 seconds should be enough
       sleep 3;
       # stop the network services
       system("/etc/rc.d/init.d/network stop");

If you run the following code in the background, it will detect a new rootkit and bring down the network interface immediately ( actually in at most 13 seconds) to prevent any further damage.

Was this document useful?

If you find this document useful, especially if you make use of it to recover a Rootkit'ed system, please drop us an email to "cayfer at bilkent edu tr" and/or "komur at bilkent edu tr".

Yavuz S. Komur & Can Ugur Ayfer
Jan 2004