Detecting and Removing Rootkits
What the hell is a Rootkit?
(
Definition
from SearchSecurity.com ). A
rootkit is a collection of tools
(programs) that a hacker
uses to mask intrusion and obtain administrator-level access to a
computer or computer network. The intruder installs a rootkit on a
computer after first obtaining user-level access, either by exploiting
a known vulnerability or cracking a password. The rootkit then collects
userids and passwords to other machines on the network, thus giving the
hacker
root or privileged access.
A rootkit may consist of utilities that
also: monitor traffic and
keystrokes; create a "backdoor" into the system for the hacker's use;
alter log files; attack other machines on the network; and alter
existing system tools to circumvent detection.
Important:
If you think/believe
your system is Rootkit'ed, read whole of this text before doing
anything. If your system is already possessed, an extra 20 minutes will
not
change much. We know that the text is a bit long but we believe it is
worth the effort; not because the text is a good one, but because the
Rootkits are a very serious threat and you have to be very careful
while removing them.
The procedures described below works for Mandrake distros. We have no
experience with other distros at all. The procedures rely on the rpm tool.
Rootkits are nuisance and can be very serious threats to Linux boxes.
Basically they exploit a kernel weakness (/dev/kmem exploit) to run
incognito on a Linux machine, usually sniffing the network interfaces
and recording critical information such as user IDs and passwords.
On a Rootkit infected machine, you will not be able to see any foreign
processes by a "ps ax" command
nor see an intruder using a "w"
or
"who" command. You shouldn't
even think about finding anything useful in your log files.
What happens when your Linux
box
is Rootkit Infected
- First things first, you will not be aware of the infection unless
you check for it.
- A frequently observed effect is that suddenly you will not be
able to make ssh connections to your box.
- A sniffer will be running hidden from your "ps" command and
logging userIDs and passwords it comes across.
- The program and log files of the sniffer will also be hidden from
"ls", "find" commands.
- Your computer will have suid programs hidden in various
directories.
- You will have some "new" accounts on your machine, possibly
with userID=0 (root privileges).
- Some Linux commands will be replaced with tampered ones which
help the rootkit hide its own files and processes.
- Any log files that you might want to see in order to trace the
intrusion will be removed.
- As a result, your computer will be totally in the hands of the
intruder.
Checking your system against
Rootkits
The Google literature mentions a chkrootkit
utility. What the chkrootkit
utility basically does is it compares the number of running processes
reported by the ps command and
the process maintenance directories under the /proc directory.
We prefer using a manual method which does almost exactly the same, but
gives
the admin's eyes a chance to view the running processes and decide
whether a suspicious process is running. In the end, intruders need not
use a predefined set of program names for their malicious code. Our
technique to see the actual
list of running processes is :
cd /proc
for i in `seq 1 33000`
do
test
-d $i && ls
-l $i/exe
done
You must be root to issue this shell command(s). This shell script
- generates a list of integers from 1 to 33000 (you actually need
integers from 1 to 32768, but 33000 is easier to type),
- for each integer in this list, checks whether a directory with
that integer as the name exists (the kernel manages a directory for
each running process using the PID number as the directory name),
- if a directory with name "$i"
exists, lists the contents of the
subdirectory "exe" in that
directory.
On a rootkit'ed computer, you will get a listing which looks like (we
have clipped the
original list):
# for
i in `seq 1 33000`; do test -d $i && ls -l $i/exe; done
lrwxrwxrwx 1 root
root
0 Jan 4 12:48 1/exe -> /sbin/initsk
(deleted)
ls: cannot read symbolic link 2/exe: No such file or directory
lrwxrwxrwx 1 root
root
0 Jan 4 12:48 2/exe
ls: cannot read symbolic link 3/exe: No such file or directory
lrwxrwxrwx 1 root
root
0 Jan 4 12:48 3/exe
ls: cannot read symbolic link 4/exe: No such file or directory
. . .
. . .
. . .
lrwxrwxrwx 1 root
root
0 Jan 4 12:48 18580/exe -> /sbin/syslogd
lrwxrwxrwx 1 root
root
0 Jan 4 12:48 18588/exe -> /sbin/klogd
lrwxrwxrwx 1 root
root
0 Jan 4 12:48 20543/exe -> /var/tmp/.pink-sk/sniffsk
lrwxrwxrwx 1 root
root
0 Jan 4 12:48 20982/exe -> /usr/sbin/httpd2
lrwxrwxrwx 1 cayfer
cayfer 0
Jan 4 12:48 22232/exe -> /usr/bin/ssh
lrwxrwxrwx 1 cayfer
cayfer 0
Jan 4 12:48 22232/exe -> /usr/sbin/imapd
Now, you have to study this long list carefully for any "weird"
processes that shouldn't be there. A very typical sign of infection is
the string "(deleted)" on the
line which belongs to the
init process (or rather the
fake one). If there is the mark "(deleted)"
next to some init process, it
means a
tampered init program is
running rather than the real one. On an
infected machine, a tampered init
starts during boot, makes necessary kernel modifications in the RAM to
hide the tools the intruder will soon run, then forks the real init
process.
Somewhere down the /proc
listing, it is very likely that you will see processes with names like ".pink-sk", ".sniff-sk", ".pink-sniff"
etc.etc. It is important that you do not rely on these names only (pink-sk is just an example); study
the list thoroughly. We have seen malicious code running as "/bin/bin", "/usr/libexec/xyz".
If you find a weird named process, you have to be creative in finding
ways to check whether they are part of the operating system or an
application or some malicious code. A good way is to query the RPM
(RedHat Packet Manager)
databases for each process you suspect, to see whether they are part of
a package; and then to see
whether they are tampered with.
As an example, you can query the RPM database to see whether
/usr/sbin/imapd (see above
list, last line) is part of a
distribution package (you should type the whole file path):
#
rpm -qf /usr/sbin/imapd
imap-2002d-6mdk
In the above example, "rpm -qf"
reports that the process running as /usr/sbin/imapd is part of the imap-2002d-6mdk package.
If you want to check whether /usr/sbin/imapd
is a modified copy (potentially malicious code), you can run the command
#
rpm -V imap-2002d-6mdk
S.5....T
c /etc/xinetd.d/ipop3
The report produced by "rpm -V"
needs to be interpreted carefully.
The first part of the report line ( namely "S.5....T
c" ) indicates that
- the file /etc/xinetd.d/ipop3
of the package is different in size than the original in the distro
package (the "S" flag),
- its MD5 checksum is different (the "5"
flag) and
- the time stamp has changed (the "T"
flag).
Well, we were querying "/usr/sbin/imapd"
but it seems that
things have rather changed for "/etc/xinetd.d/imapd".
The catch is that both /usr/bin/imapd
and /etc/xinetd.d/ipop3 files
belong to the same package and only /etc/xinet.d/pop3
has been altered so far. For
this specific example, changes are not critical; the altered file is a
configuration file and probably
the alterations were made by the
sysadmin as part of fine tuning or whatever. Since /usr/sbin/imapd is not reported as a
"modified" file, we know that it is safe to have it running on our
system.
The rpm verification report flags that you might see are:
S
: file
Size has been changed
M
: Mode has
changed (such as permissions and file type)
5
: MD5 sum has changed
U
: Ownership has changed
G
: Group ownership has changed
T
: Timestamp has changed
c
: Configuration file
(Please see "man
rpm" for a few more flags.) Ownership, mode and checksum changes for binaries are
extremely ominous!
You have to go through the process list carefully, paying special
attention to the paths of programs and not ony to names. If you see any
program file name(s) starting with a dot, you have probably found it!
If you find malicious code
running on your system
The first thing you need to do is to isolate the program file and its
log file(s). The problem is that, although you can see the program file
in the /proc listing, you will
not be able to see it (them) on the disk or remove it (them). The
modified kernel code in the RAM will hide these from other programs.
You will have
to reboot your system in
single user mode, avoiding the modified init program to have access to these
files.
The technique is fairly simple, but
we must warn you that we have seen rootkits that remove whole
directories
of
open files during shutdown. We strongly recommend
that you do not "restart" or "shutdown" or "init 0" your system. Probably
the malicious "init" program
is removing directories from your system.
Instead, do "Alt-Ctrl-Shift-SysRq-U"
(unmount disks), then
"Alt-Ctrl-Shift-SysRq-S" (sync)
and finally "Alt-Ctrl-Shift-SysRq-B"
(boot) to reboot. You will have to exit any X desktop manager you might
be using and return to console mode. (Another good reason to set you
system booting up to console mode rather than to a GUI logon screen).
If these keys are not enabled on your Linux, just enter a couple of
"sync" commands and turn the
power off. Believe me,
the risk is less... You might as well make backup copies of critical
directories like /etc and /var and the homes before resetting
or cycling the power.
While your system is restarting, hit the escape key or the Tab key
(depending on whether your LILO is a gui one or a text screen one) and
type in
linux ro init=/bin/bash
to the "boot:" prompt
The command causes the root partition to be mounted read only and the /bin/bash loaded into the memory
instead of init.
Wait till you get the root prompt. Once you get the root prompt, mount
the /proc file system by
typing
mount /proc
If this is succesful (we do not know what to do if this step
fails) remount the root file system as read-write using the command
mount
-wo remount /
After that, start the devfsd daemon
manually by typing
/sbin/devfsd
/dev
devfsd will handle /dev
links to filesystems on your system.
Now you can either mount all filesystems mentioned in /etc/fstab
with
mount
-a
or mount only the file systems you need with commands like
mount
-t ext3 /dev/hda5 /usr
etc.
etc.
Your system is now partially up without
the malicious init
running;
therefore the rootkit program files should now be visible to command
like "ls", "rm", "find" etc. For the sample
/proc listing above, you will
need to mount the /var
directory to see
what is in /var/tmp/.pink-sk.
We strongly recommend that you examine each file in the directory where
the rootkit is installed. For the binaries, you can use the
strings
binary_file
command to get an idea of what other directories and files the
malicious code might be using.
It is also very important to
study the rootkit's log files to see what "user name-password" pairs
the instruder has got hold of.
Reclaiming
your system
Well, this is the most risky
and tedious part. It is not possible to know exactly what damage the
instruder has made to your system. The best and safest way is to
install the whole system from scratch. Although it is relatively easy
to build a whole opsys from scratch, the data/site specific
applications will force you to think twice before formatting your disk.
Our prefecerence is to give a try recovering the system. The procedure
is:
- Checking /etc/passwd and /etc/shadow for any alien accounts.
- Changing the root password.
- Annulling passwords of EVERY single user account.
- Removing the rootkit's init,
and restoring a clean init
program.
- Scanning the whole disk space for SUID programs; especially those
owned by root.
- Scanning for modified binary files (our experience indicates that
the /sbin/init and Open SSL
server and client binaries are almost always modified, plus some
additional SUID programs are planted
in less likely or brand new directories like /dev, /bin/libexec. SUID programs are
files which when executed, runs with the privileges of the file's
owner,
regardless of the identity of the user who starts the program. A good
example is the "passwd"
command. Every user on the system is allowed to change their own
passwords. Changing a password means modifying the /etc/shadow (sometimes /etc/passwd) file; which is normally
not writable by anyone else but root. The solution is making the
'/bin/passwd' file a SUID file
and owned by root. When user "jack" runs
the SUID password command, the
program will run with the privileges of the owner of the /bin/passwd file, namely "root",
rather than jack's privileges. Rootkits place some SUID files here and
there owned by root, therefore even if you change the root password,
they can run the program with root privileges if they can login to your
system with a plain account. That's why you should annull each and
every user password.
- Recovering modified binaries from distros or by reinstalling them
from their tar balls.
First things first; change your root password, change your own password
and manually edit /etc/shadow
to annull all other user passwords. To annull an account's password,
replace the password field on the /etc/shadow
file with asterixes. Take your time to study the /etc/passwd file to see if there are
any weird accounts, especially with weird home directories.
Rootkits usually keep a copy of the real init program in the /sbin directory. Afterall, they need
this to fork into after their fake init
does its work. So far, we have always found the clean init file as /sbin/init-sk. You might want to
make sure that such a file is the real init by issuing a strings
command to the binary and/or comparing it to a clean init file on another computer.
Scanning the system for SUID programs is easy; only if you can trust
your "find" program. So, the
next trivial step is to make sure the
"find" command is not tampered
with.
rpm
-qf /bin/find
will report the distro package which contains the "find" command. For Mandrake
systems, it is something like "findutils-4.1.20-1mdk".
To re-install the find program
file(s), you have to issue a command like
rpm
-Uvh --replacepkgs
/your-distro-media/Mandrake/RPMS/findutils-4.1.20-1mdk
Having a clean find command,
now you can issue the
find
/ -perm +4000
command to find all SUID programs. You can use "find / -perm +4000
-a -user root" to find SUID files owned by
root (the really dangerous ones).
We are at a point where experience
pays... The list of SUID programs can be quite long. Most of the SUID
files you will see are files that really must be SUID. Knowing the real
ones from malicious ones requires some experience and insight. Be
paranoid! Double check whether a file should be SUID! Read the manual
pages, refer to Google, ask more experienced sysadmins, compare with
clean systems.
A safer and faster way of checking whether a file should be SUID
or not, is comparing its attributes with the original in the distro
package.
Suppose you want to check whether "/usr/bin/chage"
should really be a
SUID program file. All you need to do is compare "/usr/bin/chage" with
the one in the original distribution. To do this, first you need to
find
out what distro package /usr/bin/chage
belongs to:
#
rpm -qf /usr/bin/chage
shadow-utils-4.0.3-5mdk
Once you know that the RPM package containing "/usr/bin/chage" is "shadow-utils-4.0.3-5mdk", you need
to run the command
#
rpm -V shadow-utils-4.0.3-5mdk
..5....T c /etc/login.defs
to see any of the members of the package have been modified. The above
example indicates that the "/etc/login.defs"
member of the package has been modified. Since this is a config file,
you might decide that your "chage"
command is intact and really should be a SUID file. If it were the
other way around, rpm would report a mode/size change for "usr/bin/chage".
If you come across a file for which you cannot be sure whether it
should be a SUID file and be kept on the system at all, change its
owner and group to "nobody" and make a note of this change. If later
something fails, you can reconsider restoring the attributes of this
file. To change the owner of a file, you can use a command
like
chown
nobody:nobody /usr/bin/somefile
Now it is time to scan the disk(s) for modified binary files...
We strongly recommend that you install the "init", "reboot" and OpenSSH server and client programs without even
bothering testing.
First find the packages containing these programs and
then re-install them:
#
rpm -qf /sbin/init
SysVinit-2.85-1mdk
#
rpm -qf /sbin/reboot
SysVinit-2.85-1mdk
# rpm -qf /usr/bin/ssh
openssh-clients-3.6.1p2-8mdk
#
rpm -qf /usr/sbin/sshd
openssh-server-3.6.1p2-8mdk
#
rpm -Uvh --replacepkgs
/your-distro-media/Mandrake/RPMS/SysVinit-2.85-1mdk
# rpm -Uvh --replacepkgs
/your-distro-media/Mandrake/RPMS/openssh-clients-3.6.lp2-8mdk
# rpm -Uvh --replacepkgs
/your-distro-media/Mandrake/RPMS/openssh-server-3.6.lp2-8mdk
To check ALL your installed packages against the RPM database for
integrity, you can use the
#
rpm -Va
command. This command will compare all the system programs and
applications against their RPM checksums and report any discrepancies.
We strongly recommend that you re-install all packages which have
altered binaries. Altered configuration files and shell scripts are not
much of a risk but you might want to examine any altered scripts and/or
configuration files before making a decision though.
If your system is continously attacked
We are fully aware that the
procedures we have described are not
complete or 100% corrective at all. You should not be shocked if you
are
rootkit'ed again. We can suggest a simple Perl program to help you
catch
the insisting intruder by continuously checking the /proc directory and
stopping network services when an infection is detected. This will at
least save the
system log files.
#!/usr/bin/perl
#
while (1) {
@out = `cd /proc; for i in
\`seq 1 33000\`; do test -d \$i \&\& ls -l \$i/exe; done`;
foreach $_ (@out) {
# trap for "pink"rootkit
if ( (/init/ && /deleted/i) || /pink/ || /-sk/ ) {
stop_net();
}
}
# test every 15 seconds
sleep 15;
}
exit;
sub stop_net {
# notify sysadm with an email
open(MAIL, "|/usr/sbin/sendmail -t");
print MAIL "From: <user\@domain.edu\n";
print MAIL "Subject: We are stoned!\n";
print MAIL "To: admin\@domain.edu\n\n";
print MAIL qq~
We are rootkit'ted
~;
close MAIL;
# let the mail to be delivered, 3
seconds should be enough
sleep 3;
#
# stop the network services
system("/etc/rc.d/init.d/network stop");
exit;
}
If you run the following code in the background, it will detect a new
rootkit and bring down the network interface
immediately ( actually in at most 13 seconds) to prevent any further
damage.
Was
this document useful?
If
you find this document useful, especially if you make use of it to
recover a Rootkit'ed system, please drop us an email to "cayfer at bilkent edu tr" and/or "komur at bilkent edu tr".
Yavuz S. Komur & Can Ugur Ayfer
Jan 2004