Archon V3 - MCP Support

This commit is contained in:
Cole Medin
2025-02-17 13:27:24 -06:00
parent e2805b8757
commit 7a9e706581
31 changed files with 1944 additions and 30 deletions

View File

@@ -26,8 +26,10 @@ SUPABASE_SERVICE_KEY=
# The LLM you want to use for the reasoner (o3-mini, R1, QwQ, etc.).
# Example: o3-mini
# Example: deepseek-r1:7b-8k
REASONER_MODEL=
# The LLM you want to use for the primary agent/coder.
# Example: gpt-4o-mini
# Example: qwen2.5:14b-instruct-8k
PRIMARY_MODEL=

123
README.md
View File

@@ -6,8 +6,8 @@
<h3>🚀 **CURRENT VERSION** 🚀</h3>
**[ V2 - Agentic Workflow ]**
*Using LangGraph + Pydantic AI for multi-agent orchestration and planning*
**[ V3 - MCP Support ]**
*Using LangGraph + Pydantic AI with AI IDE integration*
</div>
@@ -18,7 +18,7 @@ Archon will be developed in iterations, starting with just a simple Pydantic AI
all the way to a full agentic workflow using LangGraph that can build other AI agents with any framework.
Through its iterative development, Archon showcases the power of planning, feedback loops, and domain-specific knowledge in creating robust AI agents.
The current version of Archon is V2 as mentioned above - see [V2 Documentation](iterations/v2-agentic-workflow/README.md) for details.
The current version of Archon is V3 as mentioned above - see [V3 Documentation](iterations/v3-mcp-support/README.md) for details.
## Vision
@@ -36,38 +36,50 @@ Archon demonstrates three key principles in modern AI development:
- Simple code generation without validation
- [Learn more about V1](iterations/v1-single-agent/README.md)
### V2: Current - Agentic Workflow (LangGraph)
### V2: Agentic Workflow (LangGraph)
- Multi-agent system with planning and execution separation
- Reasoning LLM (O3-mini/R1) for architecture planning
- LangGraph for workflow orchestration
- Support for local LLMs via Ollama
- [Learn more about V2](iterations/v2-agentic-workflow/README.md)
### V3: Current - MCP Support
- Integration with AI IDEs like Windsurf and Cursor
- Automated file creation and dependency management
- FastAPI service for agent generation
- Improved project structure and organization
- [Learn more about V3](iterations/v3-mcp-support/README.md)
### Future Iterations
- V3: Self-Feedback Loop - Automated validation and error correction
- V4: Tool Library Integration - Pre-built external tool incorporation
- V5: Multi-Framework Support - Framework-agnostic agent generation
- V6: Autonomous Framework Learning - Self-updating framework adapters
- V4: Self-Feedback Loop - Automated validation and error correction
- V5: Tool Library Integration - Pre-built external tool incorporation
- V6: Multi-Framework Support - Framework-agnostic agent generation
- V7: Autonomous Framework Learning - Self-updating framework adapters
### Future Integrations
- Docker
- LangSmith
- MCP
- Other frameworks besides Pydantic AI
- Other vector databases besides Supabase
- Alternative embedding models besides OpenAI
## Getting Started with V2 (current version)
## Getting Started with V3 (current version)
Since V2 is the current version of Archon, all the code for V2 is in both the `archon` and `archon/iterations/v2-agentic-workflow` directories.
Since V3 is the current version of Archon, all the code for V3 is in both the `archon` and `archon/iterations/v3-mcp-support` directories.
### Prerequisites
- Python 3.11+
- Supabase account and database
- OpenAI/OpenRouter API key or Ollama for local LLMs
- Streamlit (for web interface)
- Windsurf, Cursor, or another MCP-compatible AI IDE (optional)
### Installation
There are two ways to install Archon V3:
#### Option 1: Standard Installation (for Streamlit UI)
1. Clone the repository:
```bash
git clone https://github.com/coleam00/archon.git
@@ -81,7 +93,42 @@ source venv/bin/activate # On Windows: venv\Scripts\activate
pip install -r requirements.txt
```
3. Configure environment:
#### Option 2: MCP Server Setup (for AI IDE integration)
1. Clone the repository as above
2. Run the MCP setup script:
```bash
python setup_mcp.py
```
For running the crawler and graph service later, activate the virtual environment too:
```bash
source venv/bin/activate # On Windows: venv\Scripts\activate
```
This will:
- Create a virtual environment if it doesn't exist
- Install dependencies from requirements.txt
- Generate an MCP configuration file
3. Configure your AI IDE:
- **In Windsurf**:
- Click on the hammer icon above the chat input
- Click on "Configure"
- Paste the JSON that `setup_mcp.py` gave you as the MCP config
- Click "Refresh" next to "Configure"
- **In Cursor**:
- Go to Cursor Settings > Features > MCP
- Click on "+ Add New MCP Server"
- Name: Archon
- Type: command (equivalent to stdio)
- Command: Paste the command that `setup_mcp.py` gave for Cursor
### Environment Setup
1. Configure environment:
- Rename `.env.example` to `.env`
- Edit `.env` with your settings:
```env
@@ -97,29 +144,58 @@ pip install -r requirements.txt
### Quick Start
1. Set up the database:
- Execute `site_pages.sql` in your Supabase SQL Editor
- Execute `utils/site_pages.sql` in your Supabase SQL Editor
- This creates tables and enables vector similarity search
2. Crawl documentation:
```bash
python crawl_pydantic_ai_docs.py
python archon/crawl_pydantic_ai_docs.py
```
3. Launch the UI:
3. Run Archon either as an MCP Server or with Streamlit:
### Using with AI IDEs (MCP Support)
1. After crawling the documentation, start the graph service:
```bash
python graph_service.py
```
Archon runs as a separate API endpoint for MCP instead of directly in the MCP server because that way Archon can be updated separately without having to restart the MCP server, and the communication protocols for MCP seemed to interfere with LLM calls when done directly within the MCP server.
2. Restart the MCP server in your AI IDE
3. You can now ask your AI IDE to create agents with Archon
4. Be sure to specify when you want to use Archon - not necessary but it helps a lot
### Using the Streamlit UI
For an interactive web interface:
```bash
streamlit run streamlit_ui.py
```
Visit `http://localhost:8501` to start building AI agents!
The interface will be available at `http://localhost:8501`
## Architecture
### Current V2 Components
- `archon_graph.py`: LangGraph workflow and agent coordination
- `pydantic_ai_coder.py`: Main coding agent with RAG capabilities
- `crawl_pydantic_ai_docs.py`: Documentation processor
- `streamlit_ui.py`: Interactive web interface
- `site_pages.sql`: Database schema
### Core Files
- `mcp_server.py`: MCP server script for AI IDE integration
- `graph_service.py`: FastAPI service that handles the agentic workflow
- `setup_mcp.py`: MCP setup script
- `streamlit_ui.py`: Web interface with streaming support
- `requirements.txt`: Project dependencies
### Archon Package
- `archon/`: Core agent and workflow implementation
- `archon_graph.py`: LangGraph workflow definition and agent coordination
- `pydantic_ai_coder.py`: Main coding agent with RAG capabilities
- `crawl_pydantic_ai_docs.py`: Documentation crawler and processor
### Utilities
- `utils/`: Utility functions and database setup
- `utils.py`: Shared utility functions
- `site_pages.sql`: Database setup commands
### Database Schema
```sql
@@ -148,3 +224,4 @@ We welcome contributions! Whether you're fixing bugs, adding features, or improv
For version-specific details:
- [V1 Documentation](iterations/v1-single-agent/README.md)
- [V2 Documentation](iterations/v2-agentic-workflow/README.md)
- [V3 Documentation](iterations/v3-mcp-support/README.md)

0
archon/__init__.py Normal file
View File

View File

@@ -10,6 +10,7 @@ from openai import AsyncOpenAI
from supabase import Client
import logfire
import os
import sys
# Import the message classes from Pydantic AI
from pydantic_ai.messages import (
@@ -17,7 +18,9 @@ from pydantic_ai.messages import (
ModelMessagesTypeAdapter
)
from pydantic_ai_coder import pydantic_ai_coder, PydanticAIDeps, list_documentation_pages_helper
# Add the parent directory to Python path
sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from archon.pydantic_ai_coder import pydantic_ai_coder, PydanticAIDeps, list_documentation_pages_helper
# Load environment variables
load_dotenv()
@@ -83,9 +86,11 @@ async def define_scope_with_reasoner(state: AgentState):
result = await reasoner.run(prompt)
scope = result.data
# Save the scope to a file
scope_path = os.path.join("workbench", "scope.md")
os.makedirs("workbench", exist_ok=True)
# Get the directory one level up from the current file
current_dir = os.path.dirname(os.path.abspath(__file__))
parent_dir = os.path.dirname(current_dir)
scope_path = os.path.join(parent_dir, "workbench", "scope.md")
os.makedirs(os.path.join(parent_dir, "workbench"), exist_ok=True)
with open(scope_path, "w", encoding="utf-8") as f:
f.write(scope)

View File

@@ -3,5 +3,5 @@
"graphs": {
"agent": "./archon_graph.py:agentic_flow"
},
"env": ".env"
"env": "../.env"
}

View File

@@ -20,7 +20,7 @@ base_url = os.getenv('BASE_URL', 'https://api.openai.com/v1')
api_key = os.getenv('LLM_API_KEY', 'no-llm-api-key-provided')
model = OpenAIModel(llm, base_url=base_url, api_key=api_key)
logfire.configure(send_to_logfire='if-token-present')
# logfire.configure(send_to_logfire='if-token-present')
@dataclass
class PydanticAIDeps:

69
graph_service.py Normal file
View File

@@ -0,0 +1,69 @@
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from typing import Optional, Dict, Any
from archon.archon_graph import agentic_flow
from langgraph.types import Command
from utils.utils import write_to_log
app = FastAPI()
class InvokeRequest(BaseModel):
message: str
thread_id: str
is_first_message: bool = False
config: Optional[Dict[str, Any]] = None
@app.get("/health")
async def health_check():
"""Health check endpoint"""
return {"status": "ok"}
@app.post("/invoke")
async def invoke_agent(request: InvokeRequest):
"""Process a message through the agentic flow and return the complete response.
The agent streams the response but this API endpoint waits for the full output
before returning so it's a synchronous operation for MCP.
Another endpoint will be made later to fully stream the response from the API.
Args:
request: The InvokeRequest containing message and thread info
Returns:
dict: Contains the complete response from the agent
"""
try:
config = request.config or {
"configurable": {
"thread_id": request.thread_id
}
}
response = ""
if request.is_first_message:
write_to_log(f"Processing first message for thread {request.thread_id}")
async for msg in agentic_flow.astream(
{"latest_user_message": request.message},
config,
stream_mode="custom"
):
response += str(msg)
else:
write_to_log(f"Processing continuation for thread {request.thread_id}")
async for msg in agentic_flow.astream(
Command(resume=request.message),
config,
stream_mode="custom"
):
response += str(msg)
write_to_log(f"Final response for thread {request.thread_id}: {response}")
return {"response": response}
except Exception as e:
write_to_log(f"Error processing message for thread {request.thread_id}: {str(e)}")
raise HTTPException(status_code=500, detail=str(e))
if __name__ == "__main__":
import uvicorn
uvicorn.run(app, host="127.0.0.1", port=8100)

View File

@@ -26,8 +26,10 @@ SUPABASE_SERVICE_KEY=
# The LLM you want to use for the reasoner (o3-mini, R1, QwQ, etc.).
# Example: o3-mini
# Example: deepseek-r1:7b-8k
REASONER_MODEL=
# The LLM you want to use for the primary agent/coder.
# Example: gpt-4o-mini
# Example: qwen2.5:14b-instruct-8k
PRIMARY_MODEL=

View File

@@ -0,0 +1,35 @@
# Base URL for the OpenAI instance (default is https://api.openai.com/v1)
# OpenAI: https://api.openai.com/v1
# Ollama (example): http://localhost:11434/v1
# OpenRouter: https://openrouter.ai/api/v1
BASE_URL=
# Get your Open AI API Key by following these instructions -
# https://help.openai.com/en/articles/4936850-where-do-i-find-my-openai-api-key
# Even if using OpenRouter/Ollama, you still need to set this for the embedding model.
# Future versions of Archon will be more flexible with this.
OPENAI_API_KEY=
# For OpenAI: https://help.openai.com/en/articles/4936850-where-do-i-find-my-openai-api-key
# For OpenRouter: https://openrouter.ai/keys
LLM_API_KEY=
# For the Supabase version (sample_supabase_agent.py), set your Supabase URL and Service Key.
# Get your SUPABASE_URL from the API section of your Supabase project settings -
# https://supabase.com/dashboard/project/<your project ID>/settings/api
SUPABASE_URL=
# Get your SUPABASE_SERVICE_KEY from the API section of your Supabase project settings -
# https://supabase.com/dashboard/project/<your project ID>/settings/api
# On this page it is called the service_role secret.
SUPABASE_SERVICE_KEY=
# The LLM you want to use for the reasoner (o3-mini, R1, QwQ, etc.).
# Example: o3-mini
# Example: deepseek-r1:7b-8k
REASONER_MODEL=
# The LLM you want to use for the primary agent/coder.
# Example: gpt-4o-mini
# Example: qwen2.5:14b-instruct-8k
PRIMARY_MODEL=

View File

@@ -0,0 +1,196 @@
# Archon V3 - AI Agent Generator with MCP Support
This is the third iteration of the Archon project, building upon V2 by adding MCP server support for seamless integration with AI IDEs like Windsurf and Cursor. The system starts with a reasoning LLM that analyzes user requirements and documentation to create a detailed scope, which then guides specialized coding and routing agents in generating high-quality Pydantic AI agents.
What makes V3 special is its ability to run as an MCP server, allowing AI IDEs to directly leverage Archon's agent generation capabilities. When you ask your AI IDE to create a new agent, Archon can not only generate the code but the IDE can automatically write it to the appropriate files, manage dependencies, and help you test the agent - creating a powerful synergy between agent generation and development environment.
The core remains an intelligent documentation crawler and RAG (Retrieval-Augmented Generation) system built using Pydantic AI, LangGraph, and Supabase. The system crawls the Pydantic AI documentation, stores content in a vector database, and provides Pydantic AI agent code by retrieving and analyzing relevant documentation chunks.
This version supports both local LLMs with Ollama and cloud-based LLMs through OpenAI/OpenRouter.
Note: We still rely on OpenAI for embeddings, but future versions will add alternatives. I wanted to do this for v3 but MCP support tool a LOT of work to implement.
## Features
- MCP server support for AI IDE integration
- Multi-agent workflow using LangGraph
- Specialized agents for reasoning, routing, and coding
- Pydantic AI documentation crawling and chunking
- Vector database storage with Supabase
- Semantic search using OpenAI embeddings
- RAG-based question answering
- Support for code block preservation
- Streamlit UI for interactive querying
## Prerequisites
- Python 3.11+
- Supabase account and database
- OpenAI/OpenRouter API key or Ollama for local LLMs
- Streamlit (for web interface)
- Windsurf, Cursor, or another MCP-compatible AI IDE (optional)
## Installation
There are two ways to install Archon V3:
### Option 1: Standard Installation (for using the Streamlit UI)
1. Clone the repository:
```bash
git clone https://github.com/coleam00/archon.git
cd archon/iterations/v3-mcp-support
```
2. Install dependencies:
```bash
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
pip install -r requirements.txt
```
### Option 2: MCP Server Setup (for AI IDE integration)
1. Clone the repository as above
2. Run the MCP setup script:
```bash
python setup_mcp.py
```
For running the crawler and graph service later, activate the virtual environment too:
```bash
source venv/bin/activate # On Windows: venv\Scripts\activate
```
This will:
- Create a virtual environment if it doesn't exist
- Install dependencies from requirements.txt
- Generate an MCP configuration file
3. Configure your AI IDE:
- **In Windsurf**:
- Click on the hammer icon above the chat input
- Click on "Configure"
- Paste the JSON that `setup_mcp.py` gave you as the MCP config
- Click "Refresh" next to "Configure"
- **In Cursor**:
- Go to Cursor Settings > Features > MCP
- Click on "+ Add New MCP Server"
- Name: Archon
- Type: command (equivalent to stdio)
- Command: Paste the command that `setup_mcp.py` gave for Cursor
NOTE that this MCP server will only be functional once you complete the steps below!
Be sure to restart your MCP server after finishing all steps.
## Environment Setup
1. Set up environment variables:
- Rename `.env.example` to `.env`
- Edit `.env` with your API keys and preferences:
```env
BASE_URL=https://api.openai.com/v1 for OpenAI, https://api.openrouter.ai/v1 for OpenRouter, or your Ollama URL
LLM_API_KEY=your_openai_or_openrouter_api_key
OPENAI_API_KEY=your_openai_api_key
SUPABASE_URL=your_supabase_url
SUPABASE_SERVICE_KEY=your_supabase_service_key
PRIMARY_MODEL=gpt-4o-mini # or your preferred OpenAI model for main agent
REASONER_MODEL=o3-mini # or your preferred OpenAI model for reasoning
```
## Usage
### Database Setup
Execute the SQL commands in `utils/site_pages.sql` to:
1. Create the necessary tables
2. Enable vector similarity search
3. Set up Row Level Security policies
In Supabase, do this by going to the "SQL Editor" tab and pasting in the SQL into the editor there. Then click "Run".
### Crawl Documentation
To crawl and store documentation in the vector database:
```bash
python archon/crawl_pydantic_ai_docs.py
```
This will:
1. Fetch URLs from the documentation sitemap
2. Crawl each page and split into chunks
3. Generate embeddings and store in Supabase
### Using with AI IDEs (MCP Support)
1. After crawling the documentation, start the graph service:
```bash
python graph_service.py
```
Archon runs as a separate API endpoint for MCP instead of directly in the MCP server because that way Archon can be updated separately without having to restart the MCP server, and the communication protocols for MCP seemed to interfere with LLM calls when done directly within the MCP server.
2. Restart the MCP server in your AI IDE
3. You can now ask your AI IDE to create agents with Archon
4. Be sure to specify when you want to use Archon - not necessary but it helps a lot
### Using the Streamlit UI
For an interactive web interface:
```bash
streamlit run streamlit_ui.py
```
The interface will be available at `http://localhost:8501`
## Configuration
### Database Schema
The Supabase database uses the following schema:
```sql
CREATE TABLE site_pages (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
url TEXT,
chunk_number INTEGER,
title TEXT,
summary TEXT,
content TEXT,
metadata JSONB,
embedding VECTOR(1536)
);
```
## Project Structure
### Core Files
- `mcp_server.py`: MCP server script for AI IDE integration
- `graph_service.py`: FastAPI service that handles the agentic workflow
- `setup_mcp.py`: MCP setup script
- `streamlit_ui.py`: Web interface with streaming support
- `requirements.txt`: Project dependencies
- `.env.example`: Example environment variables
### Archon Package
- `archon/`: Core agent and workflow implementation
- `archon_graph.py`: LangGraph workflow definition and agent coordination
- `pydantic_ai_coder.py`: Main coding agent with RAG capabilities
- `crawl_pydantic_ai_docs.py`: Documentation crawler and processor
### Utilities
- `utils/`: Utility functions and database setup
- `utils.py`: Shared utility functions
- `site_pages.sql`: Database setup commands
### Runtime
- `workbench/`: Runtime files and logs
- `venv/`: Python virtual environment (created by setup)
## Contributing
Contributions are welcome! Please feel free to submit a Pull Request.

View File

@@ -0,0 +1,206 @@
from pydantic_ai.models.openai import OpenAIModel
from pydantic_ai import Agent, RunContext
from langgraph.graph import StateGraph, START, END
from langgraph.checkpoint.memory import MemorySaver
from typing import TypedDict, Annotated, List, Any
from langgraph.config import get_stream_writer
from langgraph.types import interrupt
from dotenv import load_dotenv
from openai import AsyncOpenAI
from supabase import Client
import logfire
import os
import sys
# Import the message classes from Pydantic AI
from pydantic_ai.messages import (
ModelMessage,
ModelMessagesTypeAdapter
)
# Add the parent directory to Python path
sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from archon.pydantic_ai_coder import pydantic_ai_coder, PydanticAIDeps, list_documentation_pages_helper
# Load environment variables
load_dotenv()
# Configure logfire to suppress warnings (optional)
logfire.configure(send_to_logfire='never')
base_url = os.getenv('BASE_URL', 'https://api.openai.com/v1')
api_key = os.getenv('LLM_API_KEY', 'no-llm-api-key-provided')
is_ollama = "localhost" in base_url.lower()
reasoner_llm_model = os.getenv('REASONER_MODEL', 'o3-mini')
reasoner = Agent(
OpenAIModel(reasoner_llm_model, base_url=base_url, api_key=api_key),
system_prompt='You are an expert at coding AI agents with Pydantic AI and defining the scope for doing so.',
)
primary_llm_model = os.getenv('PRIMARY_MODEL', 'gpt-4o-mini')
router_agent = Agent(
OpenAIModel(primary_llm_model, base_url=base_url, api_key=api_key),
system_prompt='Your job is to route the user message either to the end of the conversation or to continue coding the AI agent.',
)
end_conversation_agent = Agent(
OpenAIModel(primary_llm_model, base_url=base_url, api_key=api_key),
system_prompt='Your job is to end a conversation for creating an AI agent by giving instructions for how to execute the agent and they saying a nice goodbye to the user.',
)
openai_client = AsyncOpenAI(api_key=os.getenv("OPENAI_API_KEY"))
supabase: Client = Client(
os.getenv("SUPABASE_URL"),
os.getenv("SUPABASE_SERVICE_KEY")
)
# Define state schema
class AgentState(TypedDict):
latest_user_message: str
messages: Annotated[List[bytes], lambda x, y: x + y]
scope: str
# Scope Definition Node with Reasoner LLM
async def define_scope_with_reasoner(state: AgentState):
# First, get the documentation pages so the reasoner can decide which ones are necessary
documentation_pages = await list_documentation_pages_helper(supabase)
documentation_pages_str = "\n".join(documentation_pages)
# Then, use the reasoner to define the scope
prompt = f"""
User AI Agent Request: {state['latest_user_message']}
Create detailed scope document for the AI agent including:
- Architecture diagram
- Core components
- External dependencies
- Testing strategy
Also based on these documentation pages available:
{documentation_pages_str}
Include a list of documentation pages that are relevant to creating this agent for the user in the scope document.
"""
result = await reasoner.run(prompt)
scope = result.data
# Get the directory one level up from the current file
current_dir = os.path.dirname(os.path.abspath(__file__))
parent_dir = os.path.dirname(current_dir)
scope_path = os.path.join(parent_dir, "workbench", "scope.md")
os.makedirs(os.path.join(parent_dir, "workbench"), exist_ok=True)
with open(scope_path, "w", encoding="utf-8") as f:
f.write(scope)
return {"scope": scope}
# Coding Node with Feedback Handling
async def coder_agent(state: AgentState, writer):
# Prepare dependencies
deps = PydanticAIDeps(
supabase=supabase,
openai_client=openai_client,
reasoner_output=state['scope']
)
# Get the message history into the format for Pydantic AI
message_history: list[ModelMessage] = []
for message_row in state['messages']:
message_history.extend(ModelMessagesTypeAdapter.validate_json(message_row))
# Run the agent in a stream
if is_ollama:
writer = get_stream_writer()
result = await pydantic_ai_coder.run(state['latest_user_message'], deps=deps, message_history= message_history)
writer(result.data)
else:
async with pydantic_ai_coder.run_stream(
state['latest_user_message'],
deps=deps,
message_history= message_history
) as result:
# Stream partial text as it arrives
async for chunk in result.stream_text(delta=True):
writer(chunk)
# print(ModelMessagesTypeAdapter.validate_json(result.new_messages_json()))
return {"messages": [result.new_messages_json()]}
# Interrupt the graph to get the user's next message
def get_next_user_message(state: AgentState):
value = interrupt({})
# Set the user's latest message for the LLM to continue the conversation
return {
"latest_user_message": value
}
# Determine if the user is finished creating their AI agent or not
async def route_user_message(state: AgentState):
prompt = f"""
The user has sent a message:
{state['latest_user_message']}
If the user wants to end the conversation, respond with just the text "finish_conversation".
If the user wants to continue coding the AI agent, respond with just the text "coder_agent".
"""
result = await router_agent.run(prompt)
next_action = result.data
if next_action == "finish_conversation":
return "finish_conversation"
else:
return "coder_agent"
# End of conversation agent to give instructions for executing the agent
async def finish_conversation(state: AgentState, writer):
# Get the message history into the format for Pydantic AI
message_history: list[ModelMessage] = []
for message_row in state['messages']:
message_history.extend(ModelMessagesTypeAdapter.validate_json(message_row))
# Run the agent in a stream
if is_ollama:
writer = get_stream_writer()
result = await end_conversation_agent.run(state['latest_user_message'], message_history= message_history)
writer(result.data)
else:
async with end_conversation_agent.run_stream(
state['latest_user_message'],
message_history= message_history
) as result:
# Stream partial text as it arrives
async for chunk in result.stream_text(delta=True):
writer(chunk)
return {"messages": [result.new_messages_json()]}
# Build workflow
builder = StateGraph(AgentState)
# Add nodes
builder.add_node("define_scope_with_reasoner", define_scope_with_reasoner)
builder.add_node("coder_agent", coder_agent)
builder.add_node("get_next_user_message", get_next_user_message)
builder.add_node("finish_conversation", finish_conversation)
# Set edges
builder.add_edge(START, "define_scope_with_reasoner")
builder.add_edge("define_scope_with_reasoner", "coder_agent")
builder.add_edge("coder_agent", "get_next_user_message")
builder.add_conditional_edges(
"get_next_user_message",
route_user_message,
{"coder_agent": "coder_agent", "finish_conversation": "finish_conversation"}
)
builder.add_edge("finish_conversation", END)
# Configure persistence
memory = MemorySaver()
agentic_flow = builder.compile(checkpointer=memory)

View File

@@ -0,0 +1,245 @@
import os
import sys
import json
import asyncio
import requests
from xml.etree import ElementTree
from typing import List, Dict, Any
from dataclasses import dataclass
from datetime import datetime, timezone
from urllib.parse import urlparse
from dotenv import load_dotenv
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
from openai import AsyncOpenAI
from supabase import create_client, Client
load_dotenv()
# Initialize OpenAI and Supabase clients
openai_client = AsyncOpenAI(api_key=os.getenv("OPENAI_API_KEY"))
supabase: Client = create_client(
os.getenv("SUPABASE_URL"),
os.getenv("SUPABASE_SERVICE_KEY")
)
@dataclass
class ProcessedChunk:
url: str
chunk_number: int
title: str
summary: str
content: str
metadata: Dict[str, Any]
embedding: List[float]
def chunk_text(text: str, chunk_size: int = 5000) -> List[str]:
"""Split text into chunks, respecting code blocks and paragraphs."""
chunks = []
start = 0
text_length = len(text)
while start < text_length:
# Calculate end position
end = start + chunk_size
# If we're at the end of the text, just take what's left
if end >= text_length:
chunks.append(text[start:].strip())
break
# Try to find a code block boundary first (```)
chunk = text[start:end]
code_block = chunk.rfind('```')
if code_block != -1 and code_block > chunk_size * 0.3:
end = start + code_block
# If no code block, try to break at a paragraph
elif '\n\n' in chunk:
# Find the last paragraph break
last_break = chunk.rfind('\n\n')
if last_break > chunk_size * 0.3: # Only break if we're past 30% of chunk_size
end = start + last_break
# If no paragraph break, try to break at a sentence
elif '. ' in chunk:
# Find the last sentence break
last_period = chunk.rfind('. ')
if last_period > chunk_size * 0.3: # Only break if we're past 30% of chunk_size
end = start + last_period + 1
# Extract chunk and clean it up
chunk = text[start:end].strip()
if chunk:
chunks.append(chunk)
# Move start position for next chunk
start = max(start + 1, end)
return chunks
async def get_title_and_summary(chunk: str, url: str) -> Dict[str, str]:
"""Extract title and summary using GPT-4."""
system_prompt = """You are an AI that extracts titles and summaries from documentation chunks.
Return a JSON object with 'title' and 'summary' keys.
For the title: If this seems like the start of a document, extract its title. If it's a middle chunk, derive a descriptive title.
For the summary: Create a concise summary of the main points in this chunk.
Keep both title and summary concise but informative."""
try:
response = await openai_client.chat.completions.create(
model=os.getenv("LLM_MODEL", "gpt-4o-mini"),
messages=[
{"role": "system", "content": system_prompt},
{"role": "user", "content": f"URL: {url}\n\nContent:\n{chunk[:1000]}..."} # Send first 1000 chars for context
],
response_format={ "type": "json_object" }
)
return json.loads(response.choices[0].message.content)
except Exception as e:
print(f"Error getting title and summary: {e}")
return {"title": "Error processing title", "summary": "Error processing summary"}
async def get_embedding(text: str) -> List[float]:
"""Get embedding vector from OpenAI."""
try:
response = await openai_client.embeddings.create(
model="text-embedding-3-small",
input=text
)
return response.data[0].embedding
except Exception as e:
print(f"Error getting embedding: {e}")
return [0] * 1536 # Return zero vector on error
async def process_chunk(chunk: str, chunk_number: int, url: str) -> ProcessedChunk:
"""Process a single chunk of text."""
# Get title and summary
extracted = await get_title_and_summary(chunk, url)
# Get embedding
embedding = await get_embedding(chunk)
# Create metadata
metadata = {
"source": "pydantic_ai_docs",
"chunk_size": len(chunk),
"crawled_at": datetime.now(timezone.utc).isoformat(),
"url_path": urlparse(url).path
}
return ProcessedChunk(
url=url,
chunk_number=chunk_number,
title=extracted['title'],
summary=extracted['summary'],
content=chunk, # Store the original chunk content
metadata=metadata,
embedding=embedding
)
async def insert_chunk(chunk: ProcessedChunk):
"""Insert a processed chunk into Supabase."""
try:
data = {
"url": chunk.url,
"chunk_number": chunk.chunk_number,
"title": chunk.title,
"summary": chunk.summary,
"content": chunk.content,
"metadata": chunk.metadata,
"embedding": chunk.embedding
}
result = supabase.table("site_pages").insert(data).execute()
print(f"Inserted chunk {chunk.chunk_number} for {chunk.url}")
return result
except Exception as e:
print(f"Error inserting chunk: {e}")
return None
async def process_and_store_document(url: str, markdown: str):
"""Process a document and store its chunks in parallel."""
# Split into chunks
chunks = chunk_text(markdown)
# Process chunks in parallel
tasks = [
process_chunk(chunk, i, url)
for i, chunk in enumerate(chunks)
]
processed_chunks = await asyncio.gather(*tasks)
# Store chunks in parallel
insert_tasks = [
insert_chunk(chunk)
for chunk in processed_chunks
]
await asyncio.gather(*insert_tasks)
async def crawl_parallel(urls: List[str], max_concurrent: int = 5):
"""Crawl multiple URLs in parallel with a concurrency limit."""
browser_config = BrowserConfig(
headless=True,
verbose=False,
extra_args=["--disable-gpu", "--disable-dev-shm-usage", "--no-sandbox"],
)
crawl_config = CrawlerRunConfig(cache_mode=CacheMode.BYPASS)
# Create the crawler instance
crawler = AsyncWebCrawler(config=browser_config)
await crawler.start()
try:
# Create a semaphore to limit concurrency
semaphore = asyncio.Semaphore(max_concurrent)
async def process_url(url: str):
async with semaphore:
result = await crawler.arun(
url=url,
config=crawl_config,
session_id="session1"
)
if result.success:
print(f"Successfully crawled: {url}")
await process_and_store_document(url, result.markdown_v2.raw_markdown)
else:
print(f"Failed: {url} - Error: {result.error_message}")
# Process all URLs in parallel with limited concurrency
await asyncio.gather(*[process_url(url) for url in urls])
finally:
await crawler.close()
def get_pydantic_ai_docs_urls() -> List[str]:
"""Get URLs from Pydantic AI docs sitemap."""
sitemap_url = "https://ai.pydantic.dev/sitemap.xml"
try:
response = requests.get(sitemap_url)
response.raise_for_status()
# Parse the XML
root = ElementTree.fromstring(response.content)
# Extract all URLs from the sitemap
namespace = {'ns': 'http://www.sitemaps.org/schemas/sitemap/0.9'}
urls = [loc.text for loc in root.findall('.//ns:loc', namespace)]
return urls
except Exception as e:
print(f"Error fetching sitemap: {e}")
return []
async def main():
# Get URLs from Pydantic AI docs
urls = get_pydantic_ai_docs_urls()
if not urls:
print("No URLs found to crawl")
return
print(f"Found {len(urls)} URLs to crawl")
await crawl_parallel(urls)
if __name__ == "__main__":
asyncio.run(main())

View File

@@ -0,0 +1,7 @@
{
"dependencies": ["."],
"graphs": {
"agent": "./archon_graph.py:agentic_flow"
},
"env": "../.env"
}

View File

@@ -0,0 +1,219 @@
from __future__ import annotations as _annotations
from dataclasses import dataclass
from dotenv import load_dotenv
import logfire
import asyncio
import httpx
import os
from pydantic_ai import Agent, ModelRetry, RunContext
from pydantic_ai.models.openai import OpenAIModel
from openai import AsyncOpenAI
from supabase import Client
from typing import List
load_dotenv()
llm = os.getenv('PRIMARY_MODEL', 'gpt-4o-mini')
base_url = os.getenv('BASE_URL', 'https://api.openai.com/v1')
api_key = os.getenv('LLM_API_KEY', 'no-llm-api-key-provided')
model = OpenAIModel(llm, base_url=base_url, api_key=api_key)
# logfire.configure(send_to_logfire='if-token-present')
@dataclass
class PydanticAIDeps:
supabase: Client
openai_client: AsyncOpenAI
reasoner_output: str
system_prompt = """
~~ CONTEXT: ~~
You are an expert at Pydantic AI - a Python AI agent framework that you have access to all the documentation to,
including examples, an API reference, and other resources to help you build Pydantic AI agents.
~~ GOAL: ~~
Your only job is to help the user create an AI agent with Pydantic AI.
The user will describe the AI agent they want to build, or if they don't, guide them towards doing so.
You will take their requirements, and then search through the Pydantic AI documentation with the tools provided
to find all the necessary information to create the AI agent with correct code.
It's important for you to search through multiple Pydantic AI documentation pages to get all the information you need.
Almost never stick to just one page - use RAG and the other documentation tools multiple times when you are creating
an AI agent from scratch for the user.
~~ STRUCTURE: ~~
When you build an AI agent from scratch, split the agent into this files and give the code for each:
- `agent.py`: The main agent file, which is where the Pydantic AI agent is defined.
- `agent_tools.py`: A tools file for the agent, which is where all the tool functions are defined. Use this for more complex agents.
- `agent_prompts.py`: A prompts file for the agent, which includes all system prompts and other prompts used by the agent. Use this when there are many prompts or large ones.
- `.env.example`: An example `.env` file - specify each variable that the user will need to fill in and a quick comment above each one for how to do so.
- `requirements.txt`: Don't include any versions, just the top level package names needed for the agent.
~~ INSTRUCTIONS: ~~
- Don't ask the user before taking an action, just do it. Always make sure you look at the documentation with the provided tools before writing any code.
- When you first look at the documentation, always start with RAG.
Then also always check the list of available documentation pages and retrieve the content of page(s) if it'll help.
- Always let the user know when you didn't find the answer in the documentation or the right URL - be honest.
- Helpful tip: when starting a new AI agent build, it's a good idea to look at the 'weather agent' in the docs as an example.
- When starting a new AI agent build, always produce the full code for the AI agent - never tell the user to finish a tool/function.
- When refining an existing AI agent build in a conversation, just share the code changes necessary.
- Each time you respond to the user, ask them to let you know either if they need changes or the code looks good.
"""
pydantic_ai_coder = Agent(
model,
system_prompt=system_prompt,
deps_type=PydanticAIDeps,
retries=2
)
@pydantic_ai_coder.system_prompt
def add_reasoner_output(ctx: RunContext[str]) -> str:
return f"""
\n\nAdditional thoughts/instructions from the reasoner LLM.
This scope includes documentation pages for you to search as well:
{ctx.deps.reasoner_output}
"""
# Add this in to get some crazy tool calling:
# You must get ALL documentation pages listed in the scope.
async def get_embedding(text: str, openai_client: AsyncOpenAI) -> List[float]:
"""Get embedding vector from OpenAI."""
try:
response = await openai_client.embeddings.create(
model="text-embedding-3-small",
input=text
)
return response.data[0].embedding
except Exception as e:
print(f"Error getting embedding: {e}")
return [0] * 1536 # Return zero vector on error
@pydantic_ai_coder.tool
async def retrieve_relevant_documentation(ctx: RunContext[PydanticAIDeps], user_query: str) -> str:
"""
Retrieve relevant documentation chunks based on the query with RAG.
Args:
ctx: The context including the Supabase client and OpenAI client
user_query: The user's question or query
Returns:
A formatted string containing the top 5 most relevant documentation chunks
"""
try:
# Get the embedding for the query
query_embedding = await get_embedding(user_query, ctx.deps.openai_client)
# Query Supabase for relevant documents
result = ctx.deps.supabase.rpc(
'match_site_pages',
{
'query_embedding': query_embedding,
'match_count': 5,
'filter': {'source': 'pydantic_ai_docs'}
}
).execute()
if not result.data:
return "No relevant documentation found."
# Format the results
formatted_chunks = []
for doc in result.data:
chunk_text = f"""
# {doc['title']}
{doc['content']}
"""
formatted_chunks.append(chunk_text)
# Join all chunks with a separator
return "\n\n---\n\n".join(formatted_chunks)
except Exception as e:
print(f"Error retrieving documentation: {e}")
return f"Error retrieving documentation: {str(e)}"
async def list_documentation_pages_helper(supabase: Client) -> List[str]:
"""
Function to retrieve a list of all available Pydantic AI documentation pages.
This is called by the list_documentation_pages tool and also externally
to fetch documentation pages for the reasoner LLM.
Returns:
List[str]: List of unique URLs for all documentation pages
"""
try:
# Query Supabase for unique URLs where source is pydantic_ai_docs
result = supabase.from_('site_pages') \
.select('url') \
.eq('metadata->>source', 'pydantic_ai_docs') \
.execute()
if not result.data:
return []
# Extract unique URLs
urls = sorted(set(doc['url'] for doc in result.data))
return urls
except Exception as e:
print(f"Error retrieving documentation pages: {e}")
return []
@pydantic_ai_coder.tool
async def list_documentation_pages(ctx: RunContext[PydanticAIDeps]) -> List[str]:
"""
Retrieve a list of all available Pydantic AI documentation pages.
Returns:
List[str]: List of unique URLs for all documentation pages
"""
return await list_documentation_pages_helper(ctx.deps.supabase)
@pydantic_ai_coder.tool
async def get_page_content(ctx: RunContext[PydanticAIDeps], url: str) -> str:
"""
Retrieve the full content of a specific documentation page by combining all its chunks.
Args:
ctx: The context including the Supabase client
url: The URL of the page to retrieve
Returns:
str: The complete page content with all chunks combined in order
"""
try:
# Query Supabase for all chunks of this URL, ordered by chunk_number
result = ctx.deps.supabase.from_('site_pages') \
.select('title, content, chunk_number') \
.eq('url', url) \
.eq('metadata->>source', 'pydantic_ai_docs') \
.order('chunk_number') \
.execute()
if not result.data:
return f"No content found for URL: {url}"
# Format the page with its title and all chunks
page_title = result.data[0]['title'].split(' - ')[0] # Get the main title
formatted_content = [f"# {page_title}\n"]
# Add each chunk's content
for chunk in result.data:
formatted_content.append(chunk['content'])
# Join everything together
return "\n\n".join(formatted_content)
except Exception as e:
print(f"Error retrieving page content: {e}")
return f"Error retrieving page content: {str(e)}"

View File

@@ -0,0 +1,69 @@
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from typing import Optional, Dict, Any
from archon.archon_graph import agentic_flow
from langgraph.types import Command
from utils.utils import write_to_log
app = FastAPI()
class InvokeRequest(BaseModel):
message: str
thread_id: str
is_first_message: bool = False
config: Optional[Dict[str, Any]] = None
@app.get("/health")
async def health_check():
"""Health check endpoint"""
return {"status": "ok"}
@app.post("/invoke")
async def invoke_agent(request: InvokeRequest):
"""Process a message through the agentic flow and return the complete response.
The agent streams the response but this API endpoint waits for the full output
before returning so it's a synchronous operation for MCP.
Another endpoint will be made later to fully stream the response from the API.
Args:
request: The InvokeRequest containing message and thread info
Returns:
dict: Contains the complete response from the agent
"""
try:
config = request.config or {
"configurable": {
"thread_id": request.thread_id
}
}
response = ""
if request.is_first_message:
write_to_log(f"Processing first message for thread {request.thread_id}")
async for msg in agentic_flow.astream(
{"latest_user_message": request.message},
config,
stream_mode="custom"
):
response += str(msg)
else:
write_to_log(f"Processing continuation for thread {request.thread_id}")
async for msg in agentic_flow.astream(
Command(resume=request.message),
config,
stream_mode="custom"
):
response += str(msg)
write_to_log(f"Final response for thread {request.thread_id}: {response}")
return {"response": response}
except Exception as e:
write_to_log(f"Error processing message for thread {request.thread_id}: {str(e)}")
raise HTTPException(status_code=500, detail=str(e))
if __name__ == "__main__":
import uvicorn
uvicorn.run(app, host="127.0.0.1", port=8100)

View File

@@ -0,0 +1,10 @@
{
"mcpServers": {
"archon": {
"command": "[path to Archon]\\archon\\iterations\\v3-mcp-support\\venv\\Scripts\\python.exe",
"args": [
"[path to Archon]\\archon\\iterations\\v3-mcp-support\\mcp_server.py"
]
}
}
}

View File

@@ -0,0 +1,95 @@
import os
import sys
import asyncio
import threading
from mcp.server.fastmcp import FastMCP
import requests
from typing import Dict, List
import uuid
from utils.utils import write_to_log
from graph_service import app
import uvicorn
# Initialize FastMCP server
mcp = FastMCP("archon")
# Store active threads
active_threads: Dict[str, List[str]] = {}
# FastAPI service URL
GRAPH_SERVICE_URL = "http://127.0.0.1:8100"
@mcp.tool()
async def create_thread() -> str:
"""Create a new conversation thread for Archon.
Always call this tool before invoking Archon for the first time in a conversation.
(if you don't already have a thread ID)
Returns:
str: A unique thread ID for the conversation
"""
thread_id = str(uuid.uuid4())
active_threads[thread_id] = []
write_to_log(f"Created new thread: {thread_id}")
return thread_id
def _make_request(thread_id: str, user_input: str, config: dict) -> str:
"""Make synchronous request to graph service"""
response = requests.post(
f"{GRAPH_SERVICE_URL}/invoke",
json={
"message": user_input,
"thread_id": thread_id,
"is_first_message": not active_threads[thread_id],
"config": config
}
)
response.raise_for_status()
return response.json()
@mcp.tool()
async def run_agent(thread_id: str, user_input: str) -> str:
"""Run the Archon agent with user input.
Only use this tool after you have called create_thread in this conversation to get a unique thread ID.
If you already created a thread ID in this conversation, do not create another one. Reuse the same ID.
After you receive the code from Archon, always implement it into the codebase unless asked not to.
Args:
thread_id: The conversation thread ID
user_input: The user's message to process
Returns:
str: The agent's response which generally includes the code for the agent
"""
if thread_id not in active_threads:
write_to_log(f"Error: Thread not found - {thread_id}")
raise ValueError("Thread not found")
write_to_log(f"Processing message for thread {thread_id}: {user_input}")
config = {
"configurable": {
"thread_id": thread_id
}
}
try:
result = await asyncio.to_thread(_make_request, thread_id, user_input, config)
active_threads[thread_id].append(user_input)
return result['response']
except Exception as e:
raise
if __name__ == "__main__":
write_to_log("Starting MCP server")
# Run MCP server
mcp.run(transport='stdio')

View File

@@ -0,0 +1,176 @@
aiofiles==24.1.0
aiohappyeyeballs==2.4.4
aiohttp==3.11.11
aiosignal==1.3.2
aiosqlite==0.20.0
altair==5.5.0
annotated-types==0.7.0
anthropic==0.42.0
anyio==4.8.0
attrs==24.3.0
beautifulsoup4==4.12.3
blinker==1.9.0
cachetools==5.5.0
certifi==2024.12.14
cffi==1.17.1
charset-normalizer==3.4.1
click==8.1.8
cohere==5.13.12
colorama==0.4.6
Crawl4AI==0.4.247
cryptography==43.0.3
Deprecated==1.2.15
deprecation==2.1.0
distro==1.9.0
dnspython==2.7.0
email_validator==2.2.0
eval_type_backport==0.2.2
executing==2.1.0
fake-http-header==0.3.5
fastapi==0.115.8
fastapi-cli==0.0.7
fastavro==1.10.0
filelock==3.16.1
frozenlist==1.5.0
fsspec==2024.12.0
gitdb==4.0.12
GitPython==3.1.44
google-auth==2.37.0
googleapis-common-protos==1.66.0
gotrue==2.11.1
greenlet==3.1.1
griffe==1.5.4
groq==0.15.0
h11==0.14.0
h2==4.1.0
hpack==4.0.0
httpcore==1.0.7
httptools==0.6.4
httpx==0.27.2
httpx-sse==0.4.0
huggingface-hub==0.27.1
hyperframe==6.0.1
idna==3.10
importlib_metadata==8.5.0
iniconfig==2.0.0
itsdangerous==2.2.0
Jinja2==3.1.5
jiter==0.8.2
joblib==1.4.2
jsonpatch==1.33
jsonpath-python==1.0.6
jsonpointer==3.0.0
jsonschema==4.23.0
jsonschema-specifications==2024.10.1
jsonschema_rs==0.25.1
langchain-core==0.3.33
langgraph==0.2.69
langgraph-api==0.0.22
langgraph-checkpoint==2.0.10
langgraph-cli==0.1.71
langgraph-sdk==0.1.51
langsmith==0.3.6
litellm==1.57.8
logfire==3.1.0
logfire-api==3.1.0
lxml==5.3.0
markdown-it-py==3.0.0
MarkupSafe==3.0.2
mcp==1.2.1
mdurl==0.1.2
mistralai==1.2.6
mockito==1.5.3
msgpack==1.1.0
multidict==6.1.0
mypy-extensions==1.0.0
narwhals==1.21.1
nltk==3.9.1
numpy==2.2.1
openai==1.59.6
opentelemetry-api==1.29.0
opentelemetry-exporter-otlp-proto-common==1.29.0
opentelemetry-exporter-otlp-proto-http==1.29.0
opentelemetry-instrumentation==0.50b0
opentelemetry-proto==1.29.0
opentelemetry-sdk==1.29.0
opentelemetry-semantic-conventions==0.50b0
orjson==3.10.15
packaging==24.2
pandas==2.2.3
pillow==10.4.0
playwright==1.49.1
pluggy==1.5.0
postgrest==0.19.1
propcache==0.2.1
protobuf==5.29.3
psutil==6.1.1
pyarrow==18.1.0
pyasn1==0.6.1
pyasn1_modules==0.4.1
pycparser==2.22
pydantic==2.10.5
pydantic-ai==0.0.22
pydantic-ai-slim==0.0.22
pydantic-extra-types==2.10.2
pydantic-graph==0.0.22
pydantic-settings==2.7.1
pydantic_core==2.27.2
pydeck==0.9.1
pyee==12.0.0
Pygments==2.19.1
PyJWT==2.10.1
pyOpenSSL==24.3.0
pytest==8.3.4
pytest-mockito==0.0.4
python-dateutil==2.9.0.post0
python-dotenv==1.0.1
python-multipart==0.0.20
pytz==2024.2
PyYAML==6.0.2
rank-bm25==0.2.2
realtime==2.1.0
referencing==0.35.1
regex==2024.11.6
requests==2.32.3
requests-toolbelt==1.0.0
rich==13.9.4
rich-toolkit==0.13.2
rpds-py==0.22.3
rsa==4.9
shellingham==1.5.4
six==1.17.0
smmap==5.0.2
sniffio==1.3.1
snowballstemmer==2.2.0
soupsieve==2.6
sse-starlette==2.1.3
starlette==0.45.3
storage3==0.11.0
streamlit==1.41.1
StrEnum==0.4.15
structlog==24.4.0
supabase==2.11.0
supafunc==0.9.0
tenacity==9.0.0
tf-playwright-stealth==1.1.0
tiktoken==0.8.0
tokenizers==0.21.0
toml==0.10.2
tornado==6.4.2
tqdm==4.67.1
typer==0.15.1
types-requests==2.32.0.20241016
typing-inspect==0.9.0
typing_extensions==4.12.2
tzdata==2024.2
ujson==5.10.0
urllib3==2.3.0
uvicorn==0.34.0
watchdog==6.0.0
watchfiles==1.0.4
websockets==13.1
wrapt==1.17.1
xxhash==3.5.0
yarl==1.18.3
zipp==3.21.0
zstandard==0.23.0

View File

@@ -0,0 +1,60 @@
import os
import json
import subprocess
import sys
def setup_venv():
# Get the absolute path to the current directory
base_path = os.path.abspath(os.path.dirname(__file__))
venv_path = os.path.join(base_path, 'venv')
venv_created = False
# Create virtual environment if it doesn't exist
if not os.path.exists(venv_path):
print("Creating virtual environment...")
subprocess.run([sys.executable, '-m', 'venv', venv_path], check=True)
print("Virtual environment created successfully!")
venv_created = True
else:
print("Virtual environment already exists.")
# Install requirements if we just created the venv
if venv_created:
print("\nInstalling requirements...")
# Use the venv's pip to install requirements
pip_path = os.path.join(venv_path, 'Scripts', 'pip.exe')
requirements_path = os.path.join(base_path, 'requirements.txt')
subprocess.run([pip_path, 'install', '-r', requirements_path], check=True)
print("Requirements installed successfully!")
def generate_mcp_config():
# Get the absolute path to the current directory
base_path = os.path.abspath(os.path.dirname(__file__))
# Construct the paths
python_path = os.path.join(base_path, 'venv', 'Scripts', 'python.exe')
server_script_path = os.path.join(base_path, 'mcp_server.py')
# Create the config dictionary
config = {
"mcpServers": {
"archon": {
"command": python_path,
"args": [server_script_path]
}
}
}
# Write the config to a file
config_path = os.path.join(base_path, 'mcp-config.json')
with open(config_path, 'w') as f:
json.dump(config, f, indent=2)
print(f"\nMCP configuration has been written to: {config_path}")
print(f"\nMCP configuration for Cursor:\n\n{python_path} {server_script_path}")
print("\nMCP configuration for Windsurf/Claude Desktop:")
print(json.dumps(config, indent=2))
if __name__ == '__main__':
setup_venv()
generate_mcp_config()

View File

@@ -0,0 +1,117 @@
from __future__ import annotations
from typing import Literal, TypedDict
from langgraph.types import Command
from openai import AsyncOpenAI
from supabase import Client
import streamlit as st
import logfire
import asyncio
import json
import uuid
import os
import sys
# Import all the message part classes
from pydantic_ai.messages import (
ModelMessage,
ModelRequest,
ModelResponse,
SystemPromptPart,
UserPromptPart,
TextPart,
ToolCallPart,
ToolReturnPart,
RetryPromptPart,
ModelMessagesTypeAdapter
)
# Add the current directory to Python path
sys.path.append(os.path.dirname(os.path.abspath(__file__)))
from archon.archon_graph import agentic_flow
# Load environment variables
from dotenv import load_dotenv
load_dotenv()
openai_client = AsyncOpenAI(api_key=os.getenv("OPENAI_API_KEY"))
supabase: Client = Client(
os.getenv("SUPABASE_URL"),
os.getenv("SUPABASE_SERVICE_KEY")
)
# Configure logfire to suppress warnings (optional)
logfire.configure(send_to_logfire='never')
@st.cache_resource
def get_thread_id():
return str(uuid.uuid4())
thread_id = get_thread_id()
async def run_agent_with_streaming(user_input: str):
"""
Run the agent with streaming text for the user_input prompt,
while maintaining the entire conversation in `st.session_state.messages`.
"""
config = {
"configurable": {
"thread_id": thread_id
}
}
# First message from user
if len(st.session_state.messages) == 1:
async for msg in agentic_flow.astream(
{"latest_user_message": user_input}, config, stream_mode="custom"
):
yield msg
# Continue the conversation
else:
async for msg in agentic_flow.astream(
Command(resume=user_input), config, stream_mode="custom"
):
yield msg
async def main():
st.title("Archon - Agent Builder")
st.write("Describe to me an AI agent you want to build and I'll code it for you with Pydantic AI.")
st.write("Example: Build me an AI agent that can search the web with the Brave API.")
# Initialize chat history in session state if not present
if "messages" not in st.session_state:
st.session_state.messages = []
# Display chat messages from history on app rerun
for message in st.session_state.messages:
message_type = message["type"]
if message_type in ["human", "ai", "system"]:
with st.chat_message(message_type):
st.markdown(message["content"])
# Chat input for the user
user_input = st.chat_input("What do you want to build today?")
if user_input:
# We append a new request to the conversation explicitly
st.session_state.messages.append({"type": "human", "content": user_input})
# Display user prompt in the UI
with st.chat_message("user"):
st.markdown(user_input)
# Display assistant response in chat message container
response_content = ""
with st.chat_message("assistant"):
message_placeholder = st.empty() # Placeholder for updating the message
# Run the async generator to fetch responses
async for chunk in run_agent_with_streaming(user_input):
response_content += chunk
# Update the placeholder with the current response content
message_placeholder.markdown(response_content)
st.session_state.messages.append({"type": "ai", "content": response_content})
if __name__ == "__main__":
asyncio.run(main())

View File

@@ -0,0 +1,42 @@
import os
from datetime import datetime
from functools import wraps
import inspect
def write_to_log(message: str):
"""Write a message to the logs.txt file in the workbench directory.
Args:
message: The message to log
"""
# Get the directory one level up from the current file
current_dir = os.path.dirname(os.path.abspath(__file__))
parent_dir = os.path.dirname(current_dir)
workbench_dir = os.path.join(parent_dir, "workbench")
log_path = os.path.join(workbench_dir, "logs.txt")
os.makedirs(workbench_dir, exist_ok=True)
timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
log_entry = f"[{timestamp}] {message}\n"
with open(log_path, "a", encoding="utf-8") as f:
f.write(log_entry)
def log_node_execution(func):
"""Decorator to log the start and end of graph node execution.
Args:
func: The async function to wrap
"""
@wraps(func)
async def wrapper(*args, **kwargs):
func_name = func.__name__
write_to_log(f"Starting node: {func_name}")
try:
result = await func(*args, **kwargs)
write_to_log(f"Completed node: {func_name}")
return result
except Exception as e:
write_to_log(f"Error in node {func_name}: {str(e)}")
raise
return wrapper

10
mcp-config.json Normal file
View File

@@ -0,0 +1,10 @@
{
"mcpServers": {
"archon": {
"command": "[path to Archon]\\archon\\venv\\Scripts\\python.exe",
"args": [
"[path to Archon]\\archon\\mcp_server.py"
]
}
}
}

95
mcp_server.py Normal file
View File

@@ -0,0 +1,95 @@
import os
import sys
import asyncio
import threading
from mcp.server.fastmcp import FastMCP
import requests
from typing import Dict, List
import uuid
from utils.utils import write_to_log
from graph_service import app
import uvicorn
# Initialize FastMCP server
mcp = FastMCP("archon")
# Store active threads
active_threads: Dict[str, List[str]] = {}
# FastAPI service URL
GRAPH_SERVICE_URL = "http://127.0.0.1:8100"
@mcp.tool()
async def create_thread() -> str:
"""Create a new conversation thread for Archon.
Always call this tool before invoking Archon for the first time in a conversation.
(if you don't already have a thread ID)
Returns:
str: A unique thread ID for the conversation
"""
thread_id = str(uuid.uuid4())
active_threads[thread_id] = []
write_to_log(f"Created new thread: {thread_id}")
return thread_id
def _make_request(thread_id: str, user_input: str, config: dict) -> str:
"""Make synchronous request to graph service"""
response = requests.post(
f"{GRAPH_SERVICE_URL}/invoke",
json={
"message": user_input,
"thread_id": thread_id,
"is_first_message": not active_threads[thread_id],
"config": config
}
)
response.raise_for_status()
return response.json()
@mcp.tool()
async def run_agent(thread_id: str, user_input: str) -> str:
"""Run the Archon agent with user input.
Only use this tool after you have called create_thread in this conversation to get a unique thread ID.
If you already created a thread ID in this conversation, do not create another one. Reuse the same ID.
After you receive the code from Archon, always implement it into the codebase unless asked not to.
Args:
thread_id: The conversation thread ID
user_input: The user's message to process
Returns:
str: The agent's response which generally includes the code for the agent
"""
if thread_id not in active_threads:
write_to_log(f"Error: Thread not found - {thread_id}")
raise ValueError("Thread not found")
write_to_log(f"Processing message for thread {thread_id}: {user_input}")
config = {
"configurable": {
"thread_id": thread_id
}
}
try:
result = await asyncio.to_thread(_make_request, thread_id, user_input, config)
active_threads[thread_id].append(user_input)
return result['response']
except Exception as e:
raise
if __name__ == "__main__":
write_to_log("Starting MCP server")
# Run MCP server
mcp.run(transport='stdio')

Binary file not shown.

60
setup_mcp.py Normal file
View File

@@ -0,0 +1,60 @@
import os
import json
import subprocess
import sys
def setup_venv():
# Get the absolute path to the current directory
base_path = os.path.abspath(os.path.dirname(__file__))
venv_path = os.path.join(base_path, 'venv')
venv_created = False
# Create virtual environment if it doesn't exist
if not os.path.exists(venv_path):
print("Creating virtual environment...")
subprocess.run([sys.executable, '-m', 'venv', venv_path], check=True)
print("Virtual environment created successfully!")
venv_created = True
else:
print("Virtual environment already exists.")
# Install requirements if we just created the venv
if venv_created:
print("\nInstalling requirements...")
# Use the venv's pip to install requirements
pip_path = os.path.join(venv_path, 'Scripts', 'pip.exe')
requirements_path = os.path.join(base_path, 'requirements.txt')
subprocess.run([pip_path, 'install', '-r', requirements_path], check=True)
print("Requirements installed successfully!")
def generate_mcp_config():
# Get the absolute path to the current directory
base_path = os.path.abspath(os.path.dirname(__file__))
# Construct the paths
python_path = os.path.join(base_path, 'venv', 'Scripts', 'python.exe')
server_script_path = os.path.join(base_path, 'mcp_server.py')
# Create the config dictionary
config = {
"mcpServers": {
"archon": {
"command": python_path,
"args": [server_script_path]
}
}
}
# Write the config to a file
config_path = os.path.join(base_path, 'mcp-config.json')
with open(config_path, 'w') as f:
json.dump(config, f, indent=2)
print(f"\nMCP configuration has been written to: {config_path}")
print(f"\nMCP configuration for Cursor:\n\n{python_path} {server_script_path}")
print("\nMCP configuration for Windsurf/Claude Desktop:")
print(json.dumps(config, indent=2))
if __name__ == '__main__':
setup_venv()
generate_mcp_config()

View File

@@ -9,6 +9,7 @@ import asyncio
import json
import uuid
import os
import sys
# Import all the message part classes
from pydantic_ai.messages import (
@@ -24,7 +25,9 @@ from pydantic_ai.messages import (
ModelMessagesTypeAdapter
)
from archon_graph import agentic_flow
# Add the current directory to Python path
sys.path.append(os.path.dirname(os.path.abspath(__file__)))
from archon.archon_graph import agentic_flow
# Load environment variables
from dotenv import load_dotenv

72
utils/site_pages.sql Normal file
View File

@@ -0,0 +1,72 @@
-- Enable the pgvector extension
create extension if not exists vector;
-- Create the documentation chunks table
create table site_pages (
id bigserial primary key,
url varchar not null,
chunk_number integer not null,
title varchar not null,
summary varchar not null,
content text not null, -- Added content column
metadata jsonb not null default '{}'::jsonb, -- Added metadata column
embedding vector(1536), -- OpenAI embeddings are 1536 dimensions
created_at timestamp with time zone default timezone('utc'::text, now()) not null,
-- Add a unique constraint to prevent duplicate chunks for the same URL
unique(url, chunk_number)
);
-- Create an index for better vector similarity search performance
create index on site_pages using ivfflat (embedding vector_cosine_ops);
-- Create an index on metadata for faster filtering
create index idx_site_pages_metadata on site_pages using gin (metadata);
-- Create a function to search for documentation chunks
create function match_site_pages (
query_embedding vector(1536),
match_count int default 10,
filter jsonb DEFAULT '{}'::jsonb
) returns table (
id bigint,
url varchar,
chunk_number integer,
title varchar,
summary varchar,
content text,
metadata jsonb,
similarity float
)
language plpgsql
as $$
#variable_conflict use_column
begin
return query
select
id,
url,
chunk_number,
title,
summary,
content,
metadata,
1 - (site_pages.embedding <=> query_embedding) as similarity
from site_pages
where metadata @> filter
order by site_pages.embedding <=> query_embedding
limit match_count;
end;
$$;
-- Everything above will work for any PostgreSQL database. The below commands are for Supabase security
-- Enable RLS on the table
alter table site_pages enable row level security;
-- Create a policy that allows anyone to read
create policy "Allow public read access"
on site_pages
for select
to public
using (true);

42
utils/utils.py Normal file
View File

@@ -0,0 +1,42 @@
import os
from datetime import datetime
from functools import wraps
import inspect
def write_to_log(message: str):
"""Write a message to the logs.txt file in the workbench directory.
Args:
message: The message to log
"""
# Get the directory one level up from the current file
current_dir = os.path.dirname(os.path.abspath(__file__))
parent_dir = os.path.dirname(current_dir)
workbench_dir = os.path.join(parent_dir, "workbench")
log_path = os.path.join(workbench_dir, "logs.txt")
os.makedirs(workbench_dir, exist_ok=True)
timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
log_entry = f"[{timestamp}] {message}\n"
with open(log_path, "a", encoding="utf-8") as f:
f.write(log_entry)
def log_node_execution(func):
"""Decorator to log the start and end of graph node execution.
Args:
func: The async function to wrap
"""
@wraps(func)
async def wrapper(*args, **kwargs):
func_name = func.__name__
write_to_log(f"Starting node: {func_name}")
try:
result = await func(*args, **kwargs)
write_to_log(f"Completed node: {func_name}")
return result
except Exception as e:
write_to_log(f"Error in node {func_name}: {str(e)}")
raise
return wrapper