CrawlerProtection
A Guide to Using MediaWiki in a Hosted Environment
An instructional website by the developer of mh370wiki.net - a MediaWiki site about Malaysia Airlines Flight MH370.
Extension: CrawlerProtection
The CrawlerProtection extension blocks anonymous users from performing actions and accessing special pages which are most frequently abused by AI crawler bots.
This Extension was released in 2025. Once installed, the extension requires no configuration.
Before installing a new extension on a live website such as mh370wiki.net I prefer to test it with the same version of MediaWiki.
In this instance, I have installed it on this website, as a trial, and if there are no issues will install it on the mh370wiki.net later.
When the extension was downloaded the file was named CrawlerProtection-main.tar.gz
Consequently, the expanded version was in a folder named CrawlerProtection-main.
This was renamed simply CrawlerProtection prior to upload to the hosting site via FTP.
As I may reuse this upload I copied an instance of the CrawlerProtection folder to the /extensions directory for this website.
I added the following to LocalSettings.php, as per the instructions:-
## Added (date entered) wfLoadExtension( 'CrawlerProtection' );
Using MediaWiki Tools --> Special pages --> Data and tools --> Version, I can see that the extension is installed.
Unlike other installed extensions there is no hyperlink to the extension's page on the MediaWiki website.
There was no noticeable affect on the operation of this website.
Extension Performance
How do you know what the CrawlerProtection extension is doing, how effective it is and what it has blocked?
When I know the answer I will publish it here...
Articles which relate to MediaWiki Extensions
Articles which relate to MediaWiki Extensions are included in Category:Extensions.
The CategoryTree Extension enables a listing of relevant sub-categories and pages:-
Links
- Extension:CrawlerProtection
- https://www.mediawiki.org/wiki/Extension:CrawlerProtection
- Manual:Handling web crawlers
- https://www.mediawiki.org/wiki/Manual:Handling_web_crawlers