leex279 c1677a9220 fix: Skip discovery when user provides direct discovery file URLs ...

When a user directly provides a URL to a discovery file (sitemap.xml, llms.txt, robots.txt, etc.),
the system now skips the discovery phase and uses the provided file directly.

This prevents unnecessary discovery attempts and respects the user's explicit choice.

Changes:
- Check if the URL is already a discovery target before running discovery
- Skip discovery for: sitemap files, llms variants, robots.txt, well-known files, and any .txt files
- Add logging to indicate when discovery is skipped

Example: When crawling 'xyz.com/sitemap.xml' directly, the system will now use that sitemap
instead of trying to discover a different file like llms.txt

2025-09-20 13:34:07 +02:00

..

agents

refactor: Remove Socket.IO and implement HTTP polling architecture (#514)

2025-09-02 14:41:35 -05:00

mcp_server

refactor: Phase 2 Query Keys Standardization - Complete TanStack Query v5 patterns implementation (#692)

2025-09-18 11:05:03 +03:00

server

fix: Skip discovery when user provides direct discovery file URLs

2025-09-20 13:34:07 +02:00

__init__.py

The New Archon (Beta) - The Operating System for AI Coding Assistants!

2025-08-13 07:58:24 -05:00