How can I stop the user-agent vercel-screenshot/1.0 from crawling my website? The crawling is aggressive, and once new links (thousands) appear, it causes a load on db. I would prefer to disallow it via robots.txt. So far, I am blocking it using headers.
Just to confirm, will the following robots.txt rules work if there is no logic in the middleware?
User-agent: vercel-screenshot/1.0
Disallow: /