Hardening WHM & cPanel: cPHulk + ModSecurity OWASP Guide

Running a web‑hosting environment with WHM & cPanel gives you powerful tools, but it also makes you a target for automated attacks. Two of the most effective built‑in defenses are cPHulk Brute Force Protection and ModSecurity with the OWASP Core Rule Set. This guide walks you through the configuration process step by step, so you can tighten security without breaking legitimate traffic.

1. Why Hardening WHM & cPanel Matters

cPanel servers are popular because they simplify site management. That popularity also means attackers frequently scan for vulnerable login pages, weak passwords, and unpatched web applications. A successful brute‑force login or a web‑application exploit can lead to:

• Compromised accounts
• Defacement or data theft
• Spam distribution from your IP range
• Blacklisting by search engines

cPHulk blocks repeated login attempts, while ModSecurity inspects every HTTP request against a curated rule set. Together they provide a layered defense that stops many attacks before they reach your applications.

2. Preparing Your Server

cPHulk and ModSecurity are included in the default WHM installation, but you should ensure your system packages are up to date. The commands below apply to the AlmaLinux 8 / Rocky 8 / RHEL 8 environment that WHM typically runs on.

Update the OS

# dnf update -y
# dnf install -y epel-release

Explanation: dnf update -y refreshes all installed packages to the latest versions, applying security patches. dnf install -y epel-release adds the Extra Packages for Enterprise Linux repository, which provides additional tools that ModSecurity may depend on.

Verify WHM Version

# /usr/local/cpanel/cpanel -V

Confirm you are running a recent WHM release (≥ 108) that includes the latest cPHulk and ModSecurity updates.

3. Enabling and Configuring cPHulk Brute Force Protection

cPHulk monitors authentication attempts for SSH, FTP, cPanel, WHM, and mail services. It can automatically block IPs that exceed configurable thresholds.

Step 1 – Activate cPHulk

1. Log in to WHM as root.
2. Navigate to Security Center → cPHulk Brute Force Protection.
3. Check Enable cPHulk Brute Force Protection and click Save.

Step 2 – Set Global Thresholds

Adjust the following values to suit a typical Indian hosting environment (moderate traffic, many shared accounts):

• Maximum failed login attempts per hour: 5
• Maximum failed login attempts per day: 20
• Maximum failed login attempts per minute (burst): 3

These limits are low enough to stop rapid credential‑stuffing attacks while still allowing occasional user errors.

Step 3 – Whitelist Trusted IPs

If you manage the server from a static office IP or a VPN, add it to the whitelist to avoid accidental lockouts.

# /usr/local/cpanel/bin/cphulkd_whitelist add 203.0.113.45

Explanation: The command adds the IP 203.0.113.45 to cPHulk’s whitelist, bypassing all brute‑force checks for that address.

Step 4 – Review Blocked IPs

Periodically audit the block list to ensure legitimate users haven’t been caught.

# /usr/local/cpanel/bin/cphulkd_blacklist list

4. Installing ModSecurity with the OWASP Core Rule Set

ModSecurity is a web‑application firewall (WAF) that works with Apache, the default web server for cPanel. The OWASP CRS provides a comprehensive set of generic rules that detect SQL injection, XSS, file inclusion, and more.

Step 1 – Install ModSecurity Packages

# dnf install -y mod_security mod_security_crs

Explanation: mod_security installs the engine; mod_security_crs pulls in the OWASP rule set.

Step 2 – Enable ModSecurity in WHM

1. In WHM, go to Security Center → ModSecurity™ Vendors.
2. Select OWASP ModSecurity Core Rule Set (CRS) and click Enable.

Step 3 – Configure the Rule Set

After enabling, you will see a list of rule groups. For a balanced approach, start with the recommended defaults and then fine‑tune:

• Leave SQL Injection Attacks and Cross‑Site Scripting (XSS) enabled.
• Disable the HTTP Protocol Violations group only if you encounter false positives with legitimate API traffic.

Step 4 – Set Action Mode

ModSecurity can operate in Detect (log only) or Block (deny) mode. After reviewing initial logs for false positives, switch to Block for production servers.

# /usr/local/cpanel/scripts/modsecctl --mode=block

Explanation: This command changes ModSecurity from passive monitoring to active blocking.

Step 5 – Test the Configuration

Send a request that triggers a known rule, such as appending ?id=1' OR '1'='1 to a URL. Then check the audit log:

# tail -f /usr/local/apache/logs/modsec_audit.log

If the request is logged and blocked, the WAF is working correctly.

5. Fine‑Tuning and Ongoing Maintenance

Security is not a set‑and‑forget task. Regular review keeps protection effective while minimizing disruption.

Monitor Logs Daily

• cPHulk: /var/log/cphulkd.log
• ModSecurity: /usr/local/apache/logs/modsec_audit.log

Look for repeated IPs, unusual request patterns, or a high volume of rule matches.

Adjust Thresholds When Needed

If many legitimate users are being blocked, raise the cPHulk thresholds slightly (for example, from 5 to 7 attempts per hour). Conversely, if attacks are frequent, consider tightening them.

Update the OWASP CRS Regularly

# dnf update -y mod_security_crs

New rule versions address emerging threats and reduce false positives.

Whitelist Specific URLs or Parameters

Custom APIs may trigger false positives. Add exceptions via WHM → ModSecurity™ Configuration → Exclusions or by editing /usr/local/apache/conf/modsec2.user.conf and restarting Apache:

# systemctl restart httpd

6. Backing Up Your Security Settings

Before making major changes, export the current WHM security configuration. This ensures you can restore a known‑good state if something goes wrong.

Export cPHulk Settings

# /usr/local/cpanel/bin/cphulkd_config export /root/cphulk_backup.json

Export ModSecurity Configuration

# cp -a /usr/local/apache/conf/modsec* /root/modsec_backup/

Store these backups in a secure location (for example, an off‑site object store) and schedule periodic snapshots.

Conclusion

Enabling cPHulk Brute Force Protection and configuring ModSecurity with the OWASP Core Rule Set adds two robust layers of defense to your WHM & cPanel server. Regular monitoring, timely updates, and thoughtful tuning keep the balance between security and usability. Follow the steps above, keep backups handy, and revisit the settings every few weeks to stay ahead of evolving threats.