Affiliate Marketing

When you rank high in the search engines, your competitors will try outrank you by checking your backlinks just by entering your site URL in tools like SEMRush.

Luckily, there is an easy way to prevent that by using the robots.txt file that will tell SEMRush to not crawl your pages.

Add the following text to your robots.txt file to prevent SEMRush from crawling your pages and protect your site from your competitors.

User-agent: SemrushBot
Disallow: /

Check out our robots.txt file to see an example.

After you have updated your robots.txt file, confirm that it works by entering your site URL in the input field below:

SEMRush is one of the best SEO tools and I use it myself to analyze competitors with their tons of features. Free free to check play around by taking advantage of the 7 day trial. All you have to is enter a credit card details. You can cancel your SEMRush subscription within 7 days at no cost.

Explore more topics