AhRefs can't crawl my deployed website

Help
,
1

Set up a website using Vercel’s suggested configuration, namely setting the apex domain that redirects to www subdomain and making the site available there. Site seems available without issue, Google search console can access it just fine. However, AhRefs cannot access the website at all, getting timeouts when crawling any page except when crawling the apex domain, which responds with a 308 redirect.

Current behavior is site is not responding to crawler bot. Expected behavior would be responding with the actual content, like it responds in the browser.

Do an AhRefs site audit.

newslettermetrics.com, using Astro framework, configured as main deployment in Vercel

2 Likes
2

exact same issue with all our projects.

we tried to:
A - whitelist all 9614 IPs used by ahrefs without success. The full IP list to be found here https://api.ahrefs.com/v3/public/crawler-ips/

B - change crawling speed and various settings

C - check the details of the calls in the firewall, but hey all appear as “Default Web Traffic” despite obviously being blocked.

This changed approx. 7 days ago but we could not find any related changelog.

Kindly ask for clarification and prompt turnaround, as we are afraid this could also affect other integrations.

what is unusal as well is that we are receiving a timeout in the ahref logs (status 0) instead a forbidden (status 403) as recently announced for blocked connections (see here: Clients blocked by persistent actions now receive a 403 Forbidden response - Vercel)

1 Like
3

I have the same issue since I switched to the Pro Plan. I tried to set custom firewall rules, so that the User-Agent from Ahresf (Bot and Site Audit) will be bypassed, but it doesn’t work.

1 Like
4

Yes, I’m receiving status 0 (no response) as well.
Tbf it does work sometimes on some pages, seems really random. Setting the “Ignore robots” file in Ahrefs made it so at least it still tries to crawl the rest of the site, but more often than not it gets no reply.

6

Hi, everyone! Thank you for reporting. We’re investigating this now and will share updates in this thread.

7

We’ve identified this as an issue on Ahrefs’ side. Please check their status page for updates and reach out to their support team for assistance.

1 Like
8

Hi, folks! The issue is now resolved on Ahrefs side and the crawlers are working as expected again. Let us know if you have any other issues!

1 Like
9

Thanks a lot! Happy to say this is indeed working correctly now.

Just curious, can you share what seemed to have been the issue? Just out of professional curiosity.