AI Websource

The AI Websource plugin allows the INETAPP to use content from configured websites as an AI information source. The crawler periodically fetches the listed URLs, converts the content to a structured format, and makes it available to AI features (e.g. chatbots, search). Configuration is done in Configuration → AI → AI Websource.

• Websources (URLs): List of website URLs to crawl. Add entries with Add website. Each URL is fetched according to the crawl schedule. The content is parsed and indexed for use by the AI.
  • Default value: empty list
• Sitemap detection: For a configured root URL, the crawler automatically tries /sitemap.xml and then /sitemap.txt. A URL containing sitemap.xml or sitemap.txt is treated as a sitemap directly. When a sitemap is found, its listed pages are crawled and links in those pages are not followed recursively. The URL list shows a sitemap icon for the last successful sitemap detection and a web-crawl icon otherwise.
• Crawl interval: How long to wait after each full crawl before running again. You can specify the value in minutes, hours, or days.
  • Default value: 1440 (24 hours)
• Crawl start time: Daily start time for the first crawl run in HH:mm format (e.g. 3:33 for early morning). Subsequent runs follow the crawl interval.
  • Default value: 3:33
• Crawl depth: How many levels of linked pages the crawler follows from each configured URL. 0 means only the given page is fetched; higher values allow following links (e.g. 1 for direct links only). This setting applies when no usable sitemap is found. Limiting depth helps prevent excessive disk usage when the source has many links (e.g. Wikipedia articles).
  • Default value: 10

Note: Only publicly reachable URLs should be added. The crawler respects normal HTTP semantics; pages that require login or that block automated access are not suitable.