mirror of
https://github.com/coleam00/Archon.git
synced 2025-12-24 02:39:17 -05:00
597fc86c39
When a file is selected through discovery, it should be crawled as a single file without following any links contained within it. This preserves the efficiency gains of the discovery feature. Changes: - Skip link extraction when is_discovery_target is true for link collection files - Return sitemap metadata without crawling URLs when is_discovery_target is true - Add clear logging to indicate single-file mode is active This ensures discovered files (llms.txt, sitemap.xml, etc.) are processed as single authoritative sources rather than starting recursive crawls, which aligns with the PR's objective of efficient single-file discovery and crawling.