Detect Linux Kernel Rootkits Cheatsheet

Rootkits represent one of the most insidious forms of malware, allowing attackers to gain privileged access to a system and often hiding themselves within the kernel.

Detecting Linux kernel rootkits is a critical task for maintaining system security, as these rootkits can subvert standard monitoring tools and remain undetected for long periods. This guide provides a comprehensive approach to identifying and mitigating kernel rootkits on Linux systems.

1. Initial Investigation

Before diving into advanced methods, it’s essential to perform an initial assessment of the system to identify any irregularities.

Check System Integrity

Use rootkit detection tools like chkrootkit or rkhunter:

chkrootkit
rkhunter --check

Inspect Boot Logs

Search the kernel logs for rootkit-related entries:

dmesg | grep -i rootkit
journalctl -k | grep -i rootkit

Look for Suspicious Modules

Identify unusual kernel modules loaded into the system:

lsmod | grep -i suspicious

2. Monitoring Kernel Modules

Kernel modules are a prime target for rootkits since they allow direct interaction with the kernel. Monitoring these modules can help identify anomalies.

List Loaded Modules

Display all currently loaded modules:

lsmod

Verify Module Details

Check the source and authenticity of specific modules:

modinfo [module_name]

Cross-Verify Modules

Compare the output of /proc/modules with lsmod to detect discrepancies:

cat /proc/modules

3. File System Inspection

Rootkits often hide files and directories. Use the following commands to uncover hidden entities.

Search for Hidden Files and Directories

Look for suspicious files:

find / -type d -name "." 2>/dev/null find / -type f -name "." 2>/dev/null

Identify Rootkit Signature Files

Scan for files commonly associated with rootkits, such as .ko files:

locate *.ko

4. Process Analysis

Some rootkits hide processes from standard tools. Advanced process analysis can help reveal hidden activities.

Check for Missing Processes

Detect processes that do not appear in ps:

ps -ef | grep -vE "$(ls /proc | grep -E '^[0-9]+$' | tr '\n' '|')" | grep -v '^$'

Use Alternate Tools

Employ tools like htop or lsof to monitor processes:

htop
lsof

5. Network Activity Monitoring

Many rootkits enable malicious network activities. Monitoring network connections can help identify such behavior.

Monitor Open Ports

List all open ports:

netstat -tulnp
ss -tulnp

Inspect Active Connections

Identify unusual network activity:

lsof -i -n -P

Capture Suspicious Packets

Use tools like tcpdump to analyze network traffic:

tcpdump -i [interface] port [port_number]

6. Kernel Syscall Analysis

Rootkits often hijack system calls to conceal their activities. Analyzing syscall behavior can help detect tampering.

Check System Call Table

Locate the system call table in memory:

cat /proc/kallsyms | grep sys_call_table

Detect Hijacked Syscalls

Use tools like kprobes or ebpf to monitor syscall anomalies.

7. Memory and Kernel Analysis

Analyzing kernel memory directly can uncover deeply hidden rootkits.

Dump Kernel Memory

Extract a snapshot of the kernel memory:

dd if=/dev/kmem of=/root/kernel_dump bs=1M

Analyze Memory Dumps

Use tools like Volatility or Rekall to inspect memory for inconsistencies.

8. Integrity Verification

Verifying the integrity of the kernel and system files can reveal unauthorized changes.

Compare Kernel Binary

Verify the kernel binary against a trusted checksum:

sha256sum /boot/vmlinuz-$(uname -r)

Verify System Files

Check for modified system files:

rpm -Va # For RPM-based systems
debsums # For Debian-based systems

9. Advanced Tools for Rootkit Detection

Rootkit Scanners

• chkrootkit: Detects known rootkits and suspicious activity.
• rkhunter: Scans for backdoors, rootkits, and vulnerabilities.
• OSSEC: A robust HIDS for detecting rootkits and malware.

Live Kernel Analysis Tools

• KernScope: Monitors kernel activities for rootkit-like behavior.
• LiME: Extracts live memory for analysis.
• kpatch: Enables live patching and monitoring of kernel behavior.

10. Mitigation Strategies

Remove Suspicious Modules

Unload suspicious modules:

rmmod [module_name]

Restart in Rescue Mode

Boot into a live CD/USB and conduct system repairs.

Rebuild the Kernel

Rebuilding and reinstalling the kernel ensures a clean environment.

11. Post-Detection Steps

Check User Accounts

Inspect user accounts for unauthorized entries:

cat /etc/passwd
awk -F: '($3 == 0) {print}' /etc/passwd

Monitor Logins

Analyze login records:

lastlog

Inspect System Logs

Search for suspicious activity:

less /var/log/secure
grep -i "failed" /var/log/auth.log

Best Practices for Prevention

Enable SELinux or AppArmor: Enhance kernel-level security.

• Regular Updates: Keep the kernel and packages updated.
• Kernel Hardening: Use tools like Grsecurity or KSPP.
• Restrict Root Access: Implement multi-factor authentication.
• Continuous Monitoring: Deploy tools for real-time anomaly detection.

By employing these strategies and tools, administrators can effectively detect and mitigate Linux kernel rootkits, ensuring a secure computing environment.

Finding rootkits in sysfs/tracing by MatheuZSecurity

• /sys/kernel/tracing/available_filter_functions # Lists kernel functions that can be filtered for tracing.
• /sys/debug/kernel/tracing/available_filter_functions # Alternative for listing filterable functions, in debug mode.
• /sys/kernel/tracing/available_filter_functions_addrs # Lists filterable functions with addresses, only in kernel 6.5+
• /sys/debug/kernel/tracing/available_filter_functions_addrs # Like the previous one, but in debug mode, kernel 6.5+
• /sys/kernel/tracing/enabled_functions # Lists kernel functions currently enabled for tracing
• /sys/debug/kernel/tracing/enabled_functions # alternative list of tracked functions, in debug mode
• /sys/kernel/debug/tracing/trace # Where tracking events are recorded in real time
• /sys/kernel/tracing/trace # Like the previous one, it allows you to view the trace, does not require debug mode
• /sys/kernel/debug/dynamic_debug/control # enable/disable real-time kernel debug messages for specific modules
• /sys/kernel/tracing/touched_functions # shows all functions that was every traced by ftrace or a direct trampoline (only for kernel 6.4+)
• /sys/fs/bpf/* # Directory for BPF (eBPF) maps and programs, used for tracking and security
• /sys/module/* # Contains information about loaded kernel modules, such as parameters and states
• /sys/kernel/tracing/kprobe_events # Contains kprobes events

Finding rootkits in procfs

• /proc/kallsyms # Lists all kernel symbols, including function and variable addresses
• /proc/modules # Displays modules loaded in the kernel, with information such as size, usage, etc.
• /proc/vmallocinfo # Shows memory usage allocated by vmalloc
• /proc/sys/kernel/tainted # Indicates the “contamination” state of the kernel, signaling modifications or errors.

Logs

• /var/log/dmesg* # Kernel messages logs, captured by the dmesg command
• /var/log/kern.log # Stores logs of kernel events and messages, useful for monitoring and debugging
• /dev/kmsg # Interface for sending and reading kernel messages in real time
• dmesg # Command to view kernel logs messages and boot
• journalctl -k # Shows kernel logs captured by systemd-journald

Tools open source

• Nitara2
• Tracee
• bpf-hookdetect

Making ftrace based rootkit useless

Most rootkits that still work for the most recent kernel use the famous ftrace hooking based method, as hijacking the syscall table has become old and non-functional for most of the most recent kernels, precisely because of protections. and changes that the kernel underwent.