What is MRGbot and what is its use

If you reached this page, your website probably has been crawled by MRGbot. MRGbot is the name for MRG Web Services’s desktop web crawler. MRGbot simulates browsing activity of a user on desktop.

How MRGbot accesses your site

Therefore, there are slight chances that your logs may show visits from several machines at mrg.ro, all with the MRGbot user agent. It is fully agreed that the web is a vast collection of completely uncontrolled heterogeneous documents. Our goal is to crawl few pages from each site without overwhelming your server’s bandwidth. If your site is having trouble keeping up with MRG Web Services’s crawling requests, please use the contact form to request a change in the crawl rate.

MRGbot crawls only over HTTP/1.1.

If you want to prevent MRGbot from crawling content on your site, you have a number of options. Be aware of the difference between preventing MRGbot from crawling a page, preventing MRGbot from indexing a page, and preventing a page from being accessible at all by both crawlers or users.

Verifying MRGbot

If you decide to block MRGbot, be aware that the a user agent string can be spoofed by other crawlers. It’s important to verify that a problematic request actually MRG Web Services. The best way to verify that a request actually comes from MRGbot is to use a reverse DNS lookup on the source IP of the request.