docs: expand README with diverse crawling examples

Added comprehensive list of example documentation sites to crawl, organized into four categories:
- AI/ML Libraries (PydanticAI, LangChain, Anthropic)
- Web Frameworks (React, Next.js, FastAPI)
- Programming Languages (Python, TypeScript, Rust)
- Developer Tools (Docker, Git, Supabase)

This helps users understand the variety of documentation sites that work well with Archon's knowledge base.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Claude
2025-10-21 14:53:56 +00:00
parent 68fb4a8866
commit 02b0b2d531

View File

@@ -113,7 +113,26 @@ This new vision for Archon replaces the old one (the agenteer). Archon used to b
Once everything is running:
1. **Test Web Crawling**: Go to http://localhost:3737 → Knowledge Base → "Crawl Website" → Enter a doc URL (such as https://ai.pydantic.dev/llms.txt)
1. **Test Web Crawling**: Go to http://localhost:3737 → Knowledge Base → "Crawl Website" → Enter a documentation URL
**Example Sites to Crawl**:
- **AI/ML Libraries**:
- PydanticAI: `https://ai.pydantic.dev/llms.txt`
- LangChain: `https://python.langchain.com`
- Anthropic Claude: `https://docs.anthropic.com`
- **Web Frameworks**:
- React: `https://react.dev`
- Next.js: `https://nextjs.org/docs`
- FastAPI: `https://fastapi.tiangolo.com`
- **Programming Languages**:
- Python: `https://docs.python.org/3`
- TypeScript: `https://www.typescriptlang.org/docs`
- Rust: `https://doc.rust-lang.org`
- **Developer Tools**:
- Docker: `https://docs.docker.com`
- Git: `https://git-scm.com/doc`
- Supabase: `https://supabase.com/docs`
2. **Test Document Upload**: Knowledge Base → Upload a PDF
3. **Test Projects**: Projects → Create a new project and add tasks
4. **Integrate with your AI coding assistant**: MCP Dashboard → Copy connection config for your AI coding assistant