Remove the special case that gave robots.txt sitemap declarations highest
priority, which incorrectly overrode the global priority order. Now properly
respects the intended priority: llms-full.txt > llms.txt > llms.md > llms.mdx >
sitemap.xml > robots.txt.
This fixes the issue where supabase.com/docs would return sitemap.xml instead
of llms.txt even though both files exist at /docs/ and llms.txt should have
higher priority.
Changes:
- Removed robots.txt early return that bypassed priority order
- Updated test to verify llms files take precedence over robots.txt sitemaps
- All discovery now follows consistent DISCOVERY_PRIORITY order
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Improve discovery logic to check the same directory as the base URL first before
falling back to root-level and subdirectories. This ensures files like
https://supabase.com/docs/llms.txt are found when crawling
https://supabase.com/docs.
Changes:
- Check same directory as base_url first (e.g., /docs/llms.txt for /docs URL)
- Fall back to root-level urljoin behavior
- Include base directory name in subdirectory checks (e.g., /docs subdirectory)
- Maintain priority order: same-dir > root > subdirectories
- Log discovery location for better debugging
This addresses cases where documentation directories contain their own llms.txt
or sitemap files that should take precedence over root-level files.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Preserve URL case in robots.txt parsing by only lowercasing the sitemap: prefix check
- Add support for relative sitemap paths in robots.txt using urljoin()
- Fix HTML meta tag parsing to use case-insensitive regex instead of lowercasing content
- Add URL scheme validation for discovered sitemaps (http/https only)
- Fix discovery target domain filtering to use discovered URL's domain instead of input URL
- Clean up whitespace and improve dict comprehension usage
These changes improve discovery reliability and prevent URL corruption while maintaining
backward compatibility with existing discovery behavior.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- add trailing slashes to agent-work-orders endpoints to prevent FastAPI mount() redirects
- add defensive null check for repository_url in detail view
- fix backend routes to use relative paths with app.mount()
- resolves ERR_NAME_NOT_RESOLVED when accessing agent work orders
Completes the implementation of test/review workflows with automatic resolution
and integrates them into the orchestrator.
**Phase 3: Test Workflow with Resolution**
- Created test_workflow.py with automatic test failure resolution
- Implements retry loop with max 4 attempts (configurable via MAX_TEST_RETRY_ATTEMPTS)
- Parses JSON test results and resolves failures one by one
- Uses existing test.md and resolve_failed_test.md commands
- Added run_tests() and resolve_test_failure() to workflow_operations.py
**Phase 4: Review Workflow with Resolution**
- Created review_workflow.py with automatic blocker issue resolution
- Implements retry loop with max 3 attempts (configurable via MAX_REVIEW_RETRY_ATTEMPTS)
- Categorizes issues by severity (blocker/tech_debt/skippable)
- Only blocks on blocker issues - tech_debt and skippable allowed to pass
- Created review_runner.md and resolve_failed_review.md commands
- Added run_review() and resolve_review_issue() to workflow_operations.py
- Supports screenshot capture for UI review (configurable via ENABLE_SCREENSHOT_CAPTURE)
**Phase 5: Compositional Integration**
- Updated workflow_orchestrator.py to integrate test and review phases
- Test phase runs between commit and PR creation (if ENABLE_TEST_PHASE=true)
- Review phase runs after tests (if ENABLE_REVIEW_PHASE=true)
- Both phases are optional and controlled by config flags
- Step history tracks test and review execution results
- Proper error handling and logging for all phases
**Supporting Changes**
- Updated agent_names.py to add REVIEWER constant
- Added configuration flags to config.py for test/review phases
- All new code follows structured logging patterns
- Maintains compatibility with existing workflow steps
**Files Changed**: 19 files, 3035+ lines
- New: test_workflow.py, review_workflow.py, review commands
- Modified: orchestrator, workflow_operations, agent_names, config
- Phases 1-2 files (worktree, state, port allocation) also staged
The implementation is complete and ready for testing. All phases now support
parallel execution via worktree isolation with deterministic port allocation.
- Enable SSL certificate verification (verify=True) for all HTTP requests
- Implement streaming with size limits (10MB default) to prevent memory exhaustion
- Add _read_response_with_limit() helper for secure response reading
- Update all test mocks to support streaming API with iter_content()
- Fix test assertions to expect new security parameters
- Enforce deterministic rounding in progress mapper tests
Security improvements:
- Prevents MITM attacks through SSL verification
- Guards against DoS via oversized responses
- Ensures proper resource cleanup with response.close()
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* feat: Add AI-powered release notes generator with Claude
Implements automated release notes generation using Claude AI for releases and branch comparisons.
## Features
- **GitHub Action Workflow**: Automatically generates release notes when tags are pushed or releases are created
- **Local Testing Script**: Test release note generation locally before pushing
- **Branch Comparison Support**: Compare stable vs main branches or any two refs (tags, branches, commits)
- **Smart Branch Resolution**: Automatically resolves local/remote branches (e.g., stable → origin/stable)
- **Comprehensive Release Notes**: Includes features, improvements, bug fixes, technical changes, statistics, and contributors
## Files Added
- `.github/workflows/release-notes.yml` - GitHub Action for automated release notes
- `.github/RELEASE_NOTES_SETUP.md` - Complete setup guide and usage documentation
- `.github/test-release-notes.sh` - Local testing script with branch comparison support
- `.gitignore` - Exclude local release notes test files
## Usage
### Local Testing
```bash
export ANTHROPIC_API_KEY="sk-ant-..."
./.github/test-release-notes.sh # Compare origin/stable..main
./.github/test-release-notes.sh stable main # Explicit branches
./.github/test-release-notes.sh v1.0.0 v2.0.0 # Compare tags
```
### GitHub Action
1. Add `ANTHROPIC_API_KEY` to repository secrets
2. Push a tag: `git tag v1.0.0 && git push origin v1.0.0`
3. Release notes are automatically generated and added to the GitHub release
## Technical Details
- Uses Claude Sonnet 4 for intelligent content analysis
- Properly escapes JSON using jq for robust handling of special characters
- Supports multiple comparison formats: tags, branches, commit hashes
- Cost: ~$0.003 per release (~$0.036/year for monthly releases)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* refactor: Use Claude Code OAuth token for GitHub Action, keep API key for local testing
Changes the GitHub Actions workflow to use Claude Code OAuth authentication (consistent with claude-review workflow) while keeping direct API key authentication for local testing.
## Changes
### GitHub Actions Workflow
- **Before**: Direct API calls with `ANTHROPIC_API_KEY`
- **After**: Uses `anthropics/claude-code-action@beta` with `CLAUDE_CODE_OAUTH_TOKEN`
### Benefits
- ✅ Consistent authentication with existing `claude-review` workflow
- ✅ Better GitHub integration through Claude Code Action
- ✅ No additional API costs (included in Claude Code subscription)
- ✅ Same secret (`CLAUDE_CODE_OAUTH_TOKEN`) works for both workflows
### Local Testing
- **Unchanged**: Still uses `ANTHROPIC_API_KEY` for direct API calls
- Simple, fast iteration during development
- No dependency on Claude Code Action locally
## Implementation Details
The workflow now:
1. Prepares all release context in a `release-context.md` file
2. Uses Claude Code Action to read the context and generate release notes
3. Writes output to `release_notes.md`
4. Validates the generated file before creating/updating the release
## Documentation Updates
- Updated setup instructions to use `CLAUDE_CODE_OAUTH_TOKEN`
- Added section explaining authentication differences
- Clarified cost implications (OAuth has no additional costs)
- Notes that same token works for both `claude-review` and release notes workflows
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
Resolved merge conflicts by integrating features from both branches:
- Added page_storage_ops service initialization from main
- Merged link text extraction with discovery mode features
- Preserved discovery single-file mode and domain filtering
- Maintained link text fallbacks for title extraction
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* fix: Set explicit PLAYWRIGHT_BROWSERS_PATH to fix browser installation
Fixes Playwright browser not found error during web crawling.
The issue was introduced in the uv migration (9f22659) where the
browser installation path was not explicitly set as a persistent
environment variable.
Changes:
- Add ENV PLAYWRIGHT_BROWSERS_PATH=/ms-playwright
- Add --with-deps flag to playwright install command
- Add comprehensive root cause analysis document
Without this fix, Playwright installed browsers to a default location
at build time but couldn't find them at runtime, causing crawling
operations to fail with "Executable doesn't exist" errors.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* fix: Remove --with-deps flag to prevent build conflicts
The --with-deps flag was causing build failures on some systems because:
- We already manually install all Playwright dependencies (lines 26-49)
- --with-deps attempts to reinstall these packages
- This causes package conflicts and build failures on Windows/WSL
The core fix (ENV PLAYWRIGHT_BROWSERS_PATH) remains the same.
* Delete PLAYWRIGHT_FIX_ANALYSIS.md
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Cole Medin <cole@dynamous.ai>
* Initial commit for RAG by document
* Phase 2
* Adding migrations
* Fixing page IDs for chunk metadata
* Fixing unit tests, adding tool to list pages for source
* Fixing page storage upsert issues
* Max file length for retrieval
* Fixing title issue
* Fixing tests
