How to Block Bots and Spiders in cPanel

Posted about 4 months ago 4.1k

 Share

Using robots.txt

Polite blocking: Good bots respect robots.txt.

Create/Edit robots.txt

File Manager → public_html → Create or edit robots.txt.

Block Bad Bots Example

User-agent: MJ12bot
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: SemrushBot
Disallow: /

Block via .htaccess

Block by User-Agent

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (badbot|spammer) [NC]
RewriteRule ^ - [F,L]

Common Bots to Block Table

BotReasonMethod
SemrushBotAggressive scrapingrobots.txt + .htaccess
AhrefsBotHeavy resource userobots.txt
MJ12botBandwidth hogBlock completely


 Share