CastleCMS Site Crawler

Site Crawler makes CastleCMS' search a "one stop shop" for all information relevant to your users even when that information is spread out across multiple websites, without the need for expensive third-party solutions.

Imagine you'd like your CastleCMS’ search engine to look for content not just on your site but also on other sites. For example, a university may have several separate sites that host information for students, and you'd like your main student site to return search results from all those sites.

CastleCMS Site Crawler does exactly that; It lets you configure the search engine to index ("crawl") other sites and include results from those other sites alongside the results from yours.

A site administrator uses the Site Crawler configuration panel to activate the feature and specify where to find the site maps of the other sites to be crawled.

After the configuration changes have been saved, a process on the CastleCMS server will run on a schedule that reads the provided site maps and indexes the mapped content.

Once this crawling process has run, your site users will start seeing search results from those sites as well as yours when they drop down the search results site menu and select one of the configured other sites:

A site administrator will find Site Crawler statistics on the configuration panel showing when the external sites were last crawled and how many external content items were indexed. This information is useful for ensuring that the crawler is configured correctly and is running on schedule.