How would you determine if your computer or server has been compromised? If you suspect this, this article certainly applies to you. Several commands that every Linux user or incident response expert should know are also detailed here. Some of these commands also work for Mac and Windows, so it is useful to learn these tips that you may want to use.
Contents
- Stay calm and don't panic if you've been attacked
- Display a list of the last logged-in users 2.1 w or who 2.2 Using the last command 2.3 Last command of a user
- Check the current connections of your computer or server 3.1 Netstat 3.2 lsof 3.3 strace 3.4 Using ps 3.5 Check running processes with TOP 3.6 Check running processes with HTOP 3.7 Check SSH connections 3.8 Open ports
- Kernel structures
- System files that have changed recently
- Common attack points
- System activity log review – logs 7.1 File filtering – Grep and zgrep 7.2 Row count in a file using WC
- Crontab 7.1 View software-specific cronjobs 7.2 Listing user cron jobs when using systemd timers
- Conclusion
Stay calm and don't panic if you've been attacked
If you have found something, do not delete anything or make any changes yet. The first step is to begin classifying the evidence; do not access any files with cat or strings — catalogue the files and save them. Once you start deleting things, you can already investigate how deeply they have penetrated. Stay calm, investigate and investigate.
Look for traces of the attacker on the server — you may find an IP address or a strange file. Have fun; think like a detective looking for evidence at a crime scene. The game is about finding out as much as you can about the attacker. Once all the evidence has been collected, you can safely delete the malicious files.
Of course, if there is a lot of money involved on the server and it is urgent, the best option is to hire an ethical hacker or cybersecurity expert to help you. If you have a business, don't take risks and ask for help — there are laws and insurance policies that protect you only if the evidence is collected properly.
"What hackers do is think like attackers. They think maliciously about how to alter the functioning of software and help companies and individuals protect themselves through education. Because of that, they have a strong desire to share this information with others and to explain it to people whose only motivation is the desire to learn."
Display a list of the last logged-in users
W or who
The first thing you should look for is who is currently connected to your computer. It is not uncommon to find the attacker connected to the server at the same time as you.
Using the last command
This command lets you display a list of the last connected users. The history of this command goes back to the beginning of the computer or server's configuration. (You can also immediately determine how long this Linux distribution has been running.)
last
last -h
Usage:
last [options] [<username>...] [<tty>...]
Show a listing of last logged in users.
Options:
-<number> how many lines to show
-a, --hostlast display hostnames in the last column
-d, --dns translate the IP number back into a hostname
-f, --file <file> use a specific file instead of /var/log/wtmp
-F, --fulltimes print full login and logout times and dates
-i, --ip display IP numbers in numbers-and-dots notation
-n, --limit <number> how many lines to show
-R, --nohostname don't display the hostname field
-s, --since <time> display the lines since the specified time
-t, --until <time> display the lines until the specified time
-p, --present <time> display who were present at the specified time
-w, --fullnames display full user and domain names
-x, --system display system shutdown entries and run level changes
--time-format <format> show timestamps in the specified <format>:
notime|short|full|iso
-h, --help display this help
-V, --version display version
Last command of a user
In case an attacker has access to your computer or server, it is very likely that they sent a command via SSH. The following command will display the last 100 lines of commands for the current user. The 100 can be changed to any value you wish. You can use cat instead of tail to read all the logs.
tail-n 200 ~ / .bash_history | more
cat ~ / .bash_history | more
Of course, you can also use an editor such as vim or nano and save the output for later analysis. I also recommend checking the .bash_history of other users on the machine, which are located at /home/USER, and that you also investigate the root user.
sudo vim /home/USER_YOU_WANT_TO_VIEW/.bash_history
In the following example I show some suspicious commands found on a compromised server:
Some of these commands were not entered by the users — it is also not normal for someone to spin up an HTTP server, which possibly meant that unknown files were introduced into the system. And other .sh bash files were possibly set up to establish persistence on the machine.
Check the current connections of your computer or server
Netstat
netstat --help
usage: netstat [-vWeenNcCF] [<Af>] -r netstat {-V|--version|-h|--help}
netstat [-vWnNcaeol] [<Socket> ...]
netstat { [-vWeenNac] -i | [-cnNe] -M | -s [-6tuw] }
-r, --route display routing table
-i, --interfaces display interface table
-g, --groups display multicast group memberships
-s, --statistics display networking statistics (like SNMP)
-M, --masquerade display masqueraded connections
-v, --verbose be verbose
-W, --wide don't truncate IP addresses
-n, --numeric don't resolve names
--numeric-hosts don't resolve host names
--numeric-ports don't resolve port names
--numeric-users don't resolve user names
-N, --symbolic resolve hardware names
-e, --extend display other/more information
-p, --programs display PID/Program name for sockets
-o, --timers display timers
-c, --continuous continuous listing
-l, --listening display listening server sockets
-a, --all display all sockets (default: connected)
-F, --fib display Forwarding Information Base (default)
-C, --cache display routing cache instead of FIB
-Z, --context display SELinux security context for sockets
<Socket>={-t|--tcp} {-u|--udp} {-U|--udplite} {-S|--sctp} {-w|--raw}
{-x|--unix} --ax25 --ipx --netrom
<AF>=Use '-6|-4' or '-A <af>' or '--<af>'; default: inet
List of possible address families (which support routing):
inet (DARPA Internet) inet6 (IPv6) ax25 (AMPR AX.25)
netrom (AMPR NET/ROM) ipx (Novell IPX) ddp (Appletalk DDP)
x25 (CCITT X.25)
If you want more information about netstat, you can use the man page (manual)
man netstat
Typically an attacker will install a program that does nothing except listen on its assigned network port for certain instructions. Therefore, you should look for any process that is waiting for a connection (LISTEN) or has an open connection (ESTABLISHED). If you do not recognize these processes, use "strace" or "lsof" (an example below) to try to see what they are doing.
This command will show you 2 parts: the first is "Active Internet connections (w/o servers)" and the second is "Active UNIX domain sockets (w/o servers)"
Check both carefully, because if you have a malicious script running somewhere, it could be sending spam or trying to pivot (jump to another server) to an adjacent server — this is even more common than you might think, especially on adjacent networks.
netstat | more
You can also use netstat on WINDOWS. To do so, open cmd and use the same command.
netstat | more
Below I show an example of how to use the sudo netstat -atnp command. This command tells netstat to get all currently active sockets (-a), list TCP sockets (-t) along with their IPs (-n) and process numbers (-p). The first image shows nothing open; the second image shows when I opened some tabs in Chrome.
This command is very useful in many situations — for example, in the past it was possible to see the addresses of people you were talking to in applications like Telegram or WhatsApp. Although nowadays this vulnerability has already been patched.
sudo netstat -atnp | grep ESTA
When this command is entered it returns a descending list of IPs connected to your machine. It is normal to see up to 50 connections per IP. However, if you see IPs with more than 100 connections, you should sit down and analyze that traffic.
netstat -ntu|awk '{print $5}'|cut -d: -f1 -s|sort|uniq -c|sort -nk1 -r
lsof
The lsof command will list all network processes. I recommend using it with the -i option to display open sockets.
lsof -i
strace
A tool for debugging and troubleshooting. It captures and records all system calls made by a process and the signals received by it.
If strace is not pre-installed on your Linux system, run the appropriate command below for your distribution to install it.
Debian / Ubuntu
sudo apt install strace
RHEL / CentOS
yum install strace
Fedora 22+
dnf install strace
After installing strace, we can use this command to debug and display whichever process we want to inspect.
strace -d -p <PID Number>
Using ps
The PS (process status) command is one of the most widely used commands in Linux. It is generally used to get more detailed information about a specific process or all processes — for example, to find out whether a particular process is running or not, who is running which process on the system, which process is consuming more memory or CPU, how long a process has been running, etc.
ps aux
a = show processes for all users u = display the user/owner of the process x = also show processes not attached to a terminal
u = display the user/owner of the process
x = also show processes not attached to a terminal
Check running processes with TOP
The top command is a quick way to see which processes are consuming resources. Top comes pre-installed on every Linux distribution, is interactive, and you can navigate through the list of processes, kill a process, etc.
To navigate you can use the arrow keys and the page up/down keys. To exit, you can press "q".
top
Check running processes with HTOP
htop is an interactive process viewer and system resource monitor. It is an enhanced version of the top command.
It has an amazing visual interface that you can also interact with. When you first launch htop, you will receive a colorful interface showing a list of all processes running on the system. These are normally sorted by the amount of CPU usage, ordered from highest to lowest. It also shows the state of CPU, physical memory, and swap usage.
Kill a process without exiting htop: press F9 or k
To kill a process, select the process to be removed from the list and press F9 or k, which will display the "Send signal" menu listing all the available signals you can send to the command.
Check SSH connections
Check the SSH logs to see if someone is trying to gain access to the machine. You can check the server access log (SSH) with the following command, which will show you the last 100 lines of all attempts to log into the server via SSH — both failed and authorized — with important information such as the username that was attempting to gain access.
If you are using a Debian distribution:
tail -n 300 /var/log/auth.log
tail -n 300 /var/log/auth.log | grep sshd
If you are using a CentOS/RedHat-based distribution:
tail -n 300 /var/log/secure
tail -n 300 /var/log/secure | grep 'sshd'
Open ports
What ports do you have open? You can see this very well with nmap. A simple nmap scan will serve for an initial overview.
nmap localhost
This tool will be covered in more depth on another occasion.
Kernel structures
The proc filesystem is a pseudo-filesystem that provides an interface to kernel data structures. It is typically mounted at /proc.
sudo ls /proc/*/exe -la
System files that have changed recently
With this command, you can see what has happened recently. The "-2" means 2 days — it shows all files modified in the last 2 days.
sudo find /etc/var -mtime -2
Common attack points
These are the most common places where an attacker may leave traces of scripts or other artifacts:
ls /tmp -la
ls /dev/shm -la
ls /var/tmp -la
It is worth noting that at this point it is good to become an expert at using the ls command, which displays a list of files in the directory. It is recommended to use it with the -a option to show hidden files and -l to display files in long format.
ls -la
It is important that during an investigation you also analyze the /home/USER and /root/ directories with the ls -la command to look for files that may be hidden. In the following example I show some suspicious files found on a compromised server:
It is important to note that many attackers hide malicious scripts in folders such as .ICE-unix, or create hidden directories that start with "." — therefore it is important that during an investigation you look for any suspicious files or folders and collect the evidence appropriately.
System activity log review – logs
Another very important point, once the previous items have been investigated, is to always review the logs. It is important to know that on a server handling heavy traffic there will likely be a large number of system records.
The files are stored as text and can be found in the /var/log subdirectory. There are Linux logs for everything: system, kernel, package managers, boot processes, Xorg, Apache, MySQL. Most directories can be grouped into one of four categories:
- Application logs
- Event logs
- Service logs
- System logs
The most critical server monitoring logs where you can start looking are the following:
/var/log/syslog (Debian) o /var/log/messages (CentOS/RedHat): mensajes generales del sistema
/var/log/auth.log (Debian) o /var/log/secure (CentOS/RedHat): mensajes de autenticación de usuarios
/var/log/boot.log: información de arranque del sistema operativo
/var/log/maillog or var/log/mail.log: servicios de email, postfix o smtpd
/var/log/kern: mensajes de advertencia del kernel
/var/log/dmesg: mensajes de controladores
/var/log/faillog: errores de inicio de sesión; aquí puede encontrar intentos de fuerza bruta o de comprometer el sistema
/var/log/cron: mensajes sobre cron-jobs, estas son tareas automatizadas
/var/log/yum.log (CentOS/RedHat): paquetes instalados mediante el comando yum
/var/log/httpd/: contiene los error_log y access_log del sistema
error_log: errores de memoria o sistema del servicio httpd
access_log: Contiene todas las peticiones recibidas por HTTP
/var/log/mysqld.log (CentOS/RedHat/Fedora) or /var/log/mysql.log(Debian): información sobre el servicio MySQL
File filtering – Grep and zgrep
I recommend that whenever you go to search logs you use the grep command, which searches files by name and pattern. We will primarily use it to filter key points of the investigation such as the following:
- The period in which the incident occurred, whether a specific date or time
- IP address
- Username (the user can also be a system user found in /etc/passwd)
- Investigation keywords, such as a specific script, command, service, etc.
With this data, we will proceed to use the grep command appended to whichever file we want to apply it to, or by navigating to the root directory and using the wildcard *, which allows us to search across all files in the current directory and subdirectories.
grep -r "May 10/|10/May/" archivo_log
ls -la | grep palabra_clave
strings archivo | grep palabra_clave
grep IP_a_buscar secure*
Row count in a file using WC
Sometimes attackers can cover their tracks by deleting lines from logs, so we can use the wc (word count) command to count the words in certain important logs such as /var/log/httpd. This way we will notice if there is a low line count in a log from a production server and know that it was indeed compromised.
cat archivo_log | wc -l
The -l option lets us count the number of lines in that file, and just like this option, this command has many more, such as byte or character counters.
As we can observe, on the date of March 18th in this example a very low line count was found, which can indicate that a manual deletion of important system records took place. To arrive at this conclusion, a hypothesis was of course formed that an incident occurred on the server on this date — therefore I recommend always asking the right questions and having a complete picture of what you might be looking for in order to conduct a more detailed investigation.
Crontab
Another approach is to check cronjobs. You may be able to spot a malicious script or application here.
"crontab is a list of commands that run at a specified time."
A cronjob may be running from the /etc/crontab file. To view it, run the following command:
less /etc/crontab
View software-specific cronjobs
cd /etc/cron.d/
ls -l
cat nombredelarchivo
Listing user cron jobs when using systemd timers
Systemd comes with a cron system called systemd.timer. This is another option you can use on your Linux distribution. Use the systemctl command as follows to list cron jobs on Linux.
systemctl list-timers
Conclusion
In this article, I show useful commands and tools that can help you determine whether your Linux computer or server has been compromised. You can also use tools such as Wireshark, Snort, and Splunk, among others, to accomplish this. I would like to conclude with the following.
- This article was written for educational purposes
- The information will only be used to expand knowledge and not to carry out malicious or harmful attacks
- It is important that in the event of an attack or threat you contact an expert in the field who can advise you and provide you with the tools to resolve your issues
My goal is to educate people, especially in Costa Rica, so that they can learn about this amazing world of Cybersecurity — and help more companies move forward.
Leave a Comment
Comments are reviewed before publishing.