The New Archon (Beta) - The Operating System for AI Coding Assistants!

original_archon/iterations/v2-agentic-workflow/.env.example

@@ -0,0 +1,38 @@
+# Base URL for the OpenAI instance (default is https://api.openai.com/v1)
+# OpenAI: https://api.openai.com/v1
+# Ollama (example): http://localhost:11434/v1
+# OpenRouter: https://openrouter.ai/api/v1
+BASE_URL=
+
+# Get your Open AI API Key by following these instructions -
+# https://help.openai.com/en/articles/4936850-where-do-i-find-my-openai-api-key
+# Even if using OpenRouter/Ollama, you still need to set this for the embedding model.
+# Future versions of Archon will be more flexible with this.
+OPENAI_API_KEY=
+
+# For OpenAI: https://help.openai.com/en/articles/4936850-where-do-i-find-my-openai-api-key
+# For OpenRouter: https://openrouter.ai/keys
+LLM_API_KEY=
+
+# For the Supabase version (sample_supabase_agent.py), set your Supabase URL and Service Key.
+# Get your SUPABASE_URL from the API section of your Supabase project settings -
+# https://supabase.com/dashboard/project/<your project ID>/settings/api
+SUPABASE_URL=
+
+# Get your SUPABASE_SERVICE_KEY from the API section of your Supabase project settings -
+# https://supabase.com/dashboard/project/<your project ID>/settings/api
+# On this page it is called the service_role secret.
+SUPABASE_SERVICE_KEY=
+
+# The LLM you want to use for the reasoner (o3-mini, R1, QwQ, etc.).
+# Example: o3-mini
+# Example: deepseek-r1:7b-8k
+REASONER_MODEL=
+
+# The LLM you want to use for the primary agent/coder.
+# Example: qwen2.5:14b-instruct-8k
+PRIMARY_MODEL=
+
+# Embedding model you want to use (nomic-embed-text:latest, text-embedding-3-small)
+# Example: nomic-embed-text:latest
+EMBEDDING_MODEL=

original_archon/iterations/v2-agentic-workflow/.gitignore

@@ -0,0 +1 @@
+.env

original_archon/iterations/v2-agentic-workflow/README.md

@@ -0,0 +1,132 @@
+# Archon V2 - Agentic Workflow for Building Pydantic AI Agents
+
+This is the second iteration of the Archon project, building upon V1 by introducing LangGraph for a full agentic workflow. The system starts with a reasoning LLM (like O3-mini or R1) that analyzes user requirements and documentation to create a detailed scope, which then guides specialized coding and routing agents in generating high-quality Pydantic AI agents.
+
+An intelligent documentation crawler and RAG (Retrieval-Augmented Generation) system built using Pydantic AI, LangGraph, and Supabase that is capable of building other Pydantic AI agents. The system crawls the Pydantic AI documentation, stores content in a vector database, and provides Pydantic AI agent code by retrieving and analyzing relevant documentation chunks.
+
+This version also supports local LLMs with Ollama for the main agent and reasoning LLM.
+
+Note that we are still relying on OpenAI for embeddings no matter what, but future versions of Archon will change that.
+
+## Features
+
+- Multi-agent workflow using LangGraph
+- Specialized agents for reasoning, routing, and coding
+- Pydantic AI documentation crawling and chunking
+- Vector database storage with Supabase
+- Semantic search using OpenAI embeddings
+- RAG-based question answering
+- Support for code block preservation
+- Streamlit UI for interactive querying
+
+## Prerequisites
+
+- Python 3.11+
+- Supabase account and database
+- OpenAI/OpenRouter API key or Ollama for local LLMs
+- Streamlit (for web interface)
+
+## Installation
+
+1. Clone the repository:
+```bash
+git clone https://github.com/coleam00/archon.git
+cd archon/iterations/v2-agentic-workflow
+```
+
+2. Install dependencies (recommended to use a Python virtual environment):
+```bash
+python -m venv venv
+source venv/bin/activate  # On Windows: venv\Scripts\activate
+pip install -r requirements.txt
+```
+
+3. Set up environment variables:
+   - Rename `.env.example` to `.env`
+   - Edit `.env` with your API keys and preferences:
+   ```env
+   BASE_URL=https://api.openai.com/v1 for OpenAI, https://api.openrouter.ai/v1 for OpenRouter, or your Ollama URL
+   LLM_API_KEY=your_openai_or_openrouter_api_key
+   OPENAI_API_KEY=your_openai_api_key
+   SUPABASE_URL=your_supabase_url
+   SUPABASE_SERVICE_KEY=your_supabase_service_key
+   PRIMARY_MODEL=gpt-4o-mini  # or your preferred OpenAI model for main agent
+   REASONER_MODEL=o3-mini     # or your preferred OpenAI model for reasoning
+   ```
+
+## Usage
+
+### Database Setup
+
+Execute the SQL commands in `site_pages.sql` to:
+1. Create the necessary tables
+2. Enable vector similarity search
+3. Set up Row Level Security policies
+
+In Supabase, do this by going to the "SQL Editor" tab and pasting in the SQL into the editor there. Then click "Run".
+
+### Crawl Documentation
+
+To crawl and store documentation in the vector database:
+
+```bash
+python crawl_pydantic_ai_docs.py
+```
+
+This will:
+1. Fetch URLs from the documentation sitemap
+2. Crawl each page and split into chunks
+3. Generate embeddings and store in Supabase
+
+### Chunking Configuration
+
+You can configure chunking parameters in `crawl_pydantic_ai_docs.py`:
+```python
+chunk_size = 5000  # Characters per chunk
+```
+
+The chunker intelligently preserves:
+- Code blocks
+- Paragraph boundaries
+- Sentence boundaries
+
+### Streamlit Web Interface
+
+For an interactive web interface to query the documentation and create agents:
+
+```bash
+streamlit run streamlit_ui.py
+```
+
+The interface will be available at `http://localhost:8501`
+
+## Configuration
+
+### Database Schema
+
+The Supabase database uses the following schema:
+```sql
+CREATE TABLE site_pages (
+    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
+    url TEXT,
+    chunk_number INTEGER,
+    title TEXT,
+    summary TEXT,
+    content TEXT,
+    metadata JSONB,
+    embedding VECTOR(1536)
+);
+```
+
+## Project Structure
+
+- `archon_graph.py`: LangGraph workflow definition and agent coordination
+- `pydantic_ai_coder.py`: Main coding agent with RAG capabilities
+- `crawl_pydantic_ai_docs.py`: Documentation crawler and processor
+- `streamlit_ui.py`: Web interface with streaming support
+- `site_pages.sql`: Database setup commands
+- `requirements.txt`: Project dependencies
+
+## Contributing
+
+Contributions are welcome! Please feel free to submit a Pull Request.

original_archon/iterations/v2-agentic-workflow/archon_graph.py

@@ -0,0 +1,207 @@
+from pydantic_ai.models.openai import OpenAIModel
+from pydantic_ai import Agent, RunContext
+from langgraph.graph import StateGraph, START, END
+from langgraph.checkpoint.memory import MemorySaver
+from typing import TypedDict, Annotated, List, Any
+from langgraph.config import get_stream_writer
+from langgraph.types import interrupt
+from dotenv import load_dotenv
+from openai import AsyncOpenAI
+from supabase import Client
+import logfire
+import os
+
+# Import the message classes from Pydantic AI
+from pydantic_ai.messages import (
+    ModelMessage,
+    ModelMessagesTypeAdapter
+)
+
+from pydantic_ai_coder import pydantic_ai_coder, PydanticAIDeps, list_documentation_pages_helper
+
+# Load environment variables
+load_dotenv()
+
+# Configure logfire to suppress warnings (optional)
+logfire.configure(send_to_logfire='never')
+
+base_url = os.getenv('BASE_URL', 'https://api.openai.com/v1')
+api_key = os.getenv('LLM_API_KEY', 'no-llm-api-key-provided')
+is_ollama = "localhost" in base_url.lower()
+reasoner_llm_model = os.getenv('REASONER_MODEL', 'o3-mini')
+reasoner = Agent(  
+    OpenAIModel(reasoner_llm_model, base_url=base_url, api_key=api_key),
+    system_prompt='You are an expert at coding AI agents with Pydantic AI and defining the scope for doing so.',  
+)
+
+primary_llm_model = os.getenv('PRIMARY_MODEL', 'gpt-4o-mini')
+router_agent = Agent(  
+    OpenAIModel(primary_llm_model, base_url=base_url, api_key=api_key),
+    system_prompt='Your job is to route the user message either to the end of the conversation or to continue coding the AI agent.',  
+)
+
+end_conversation_agent = Agent(  
+    OpenAIModel(primary_llm_model, base_url=base_url, api_key=api_key),
+    system_prompt='Your job is to end a conversation for creating an AI agent by giving instructions for how to execute the agent and they saying a nice goodbye to the user.',  
+)
+
+openai_client=None
+
+if is_ollama:
+    openai_client = AsyncOpenAI(base_url=base_url,api_key=api_key)
+else:
+    openai_client = AsyncOpenAI(api_key=os.getenv("OPENAI_API_KEY"))
+
+supabase: Client = Client(
+    os.getenv("SUPABASE_URL"),
+    os.getenv("SUPABASE_SERVICE_KEY")
+)
+
+# Define state schema
+class AgentState(TypedDict):
+    latest_user_message: str
+    messages: Annotated[List[bytes], lambda x, y: x + y]
+    scope: str
+
+# Scope Definition Node with Reasoner LLM
+async def define_scope_with_reasoner(state: AgentState):
+    # First, get the documentation pages so the reasoner can decide which ones are necessary
+    documentation_pages = await list_documentation_pages_helper(supabase)
+    documentation_pages_str = "\n".join(documentation_pages)
+
+    # Then, use the reasoner to define the scope
+    prompt = f"""
+    User AI Agent Request: {state['latest_user_message']}
+    
+    Create detailed scope document for the AI agent including:
+    - Architecture diagram
+    - Core components
+    - External dependencies
+    - Testing strategy
+
+    Also based on these documentation pages available:
+
+    {documentation_pages_str}
+
+    Include a list of documentation pages that are relevant to creating this agent for the user in the scope document.
+    """
+
+    result = await reasoner.run(prompt)
+    scope = result.data
+
+    # Save the scope to a file
+    scope_path = os.path.join("workbench", "scope.md")
+    os.makedirs("workbench", exist_ok=True)
+
+    with open(scope_path, "w", encoding="utf-8") as f:
+        f.write(scope)
+    
+    return {"scope": scope}
+
+# Coding Node with Feedback Handling
+async def coder_agent(state: AgentState, writer):    
+    # Prepare dependencies
+    deps = PydanticAIDeps(
+        supabase=supabase,
+        openai_client=openai_client,
+        reasoner_output=state['scope']
+    )
+
+    # Get the message history into the format for Pydantic AI
+    message_history: list[ModelMessage] = []
+    for message_row in state['messages']:
+        message_history.extend(ModelMessagesTypeAdapter.validate_json(message_row))
+
+    # Run the agent in a stream
+    if is_ollama:
+        writer = get_stream_writer()
+        result = await pydantic_ai_coder.run(state['latest_user_message'], deps=deps, message_history= message_history)
+        writer(result.data)
+    else:
+        async with pydantic_ai_coder.run_stream(
+            state['latest_user_message'],
+            deps=deps,
+            message_history= message_history
+        ) as result:
+            # Stream partial text as it arrives
+            async for chunk in result.stream_text(delta=True):
+                writer(chunk)
+
+    # print(ModelMessagesTypeAdapter.validate_json(result.new_messages_json()))
+
+    return {"messages": [result.new_messages_json()]}
+
+# Interrupt the graph to get the user's next message
+def get_next_user_message(state: AgentState):
+    value = interrupt({})
+
+    # Set the user's latest message for the LLM to continue the conversation
+    return {
+        "latest_user_message": value
+    }
+
+# Determine if the user is finished creating their AI agent or not
+async def route_user_message(state: AgentState):
+    prompt = f"""
+    The user has sent a message: 
+    
+    {state['latest_user_message']}
+
+    If the user wants to end the conversation, respond with just the text "finish_conversation".
+    If the user wants to continue coding the AI agent, respond with just the text "coder_agent".
+    """
+
+    result = await router_agent.run(prompt)
+    next_action = result.data
+
+    if next_action == "finish_conversation":
+        return "finish_conversation"
+    else:
+        return "coder_agent"
+
+# End of conversation agent to give instructions for executing the agent
+async def finish_conversation(state: AgentState, writer):    
+    # Get the message history into the format for Pydantic AI
+    message_history: list[ModelMessage] = []
+    for message_row in state['messages']:
+        message_history.extend(ModelMessagesTypeAdapter.validate_json(message_row))
+
+    # Run the agent in a stream
+    if is_ollama:
+        writer = get_stream_writer()
+        result = await end_conversation_agent.run(state['latest_user_message'], message_history= message_history)
+        writer(result.data)   
+    else: 
+        async with end_conversation_agent.run_stream(
+            state['latest_user_message'],
+            message_history= message_history
+        ) as result:
+            # Stream partial text as it arrives
+            async for chunk in result.stream_text(delta=True):
+                writer(chunk)
+
+    return {"messages": [result.new_messages_json()]}        
+
+# Build workflow
+builder = StateGraph(AgentState)
+
+# Add nodes
+builder.add_node("define_scope_with_reasoner", define_scope_with_reasoner)
+builder.add_node("coder_agent", coder_agent)
+builder.add_node("get_next_user_message", get_next_user_message)
+builder.add_node("finish_conversation", finish_conversation)
+
+# Set edges
+builder.add_edge(START, "define_scope_with_reasoner")
+builder.add_edge("define_scope_with_reasoner", "coder_agent")
+builder.add_edge("coder_agent", "get_next_user_message")
+builder.add_conditional_edges(
+    "get_next_user_message",
+    route_user_message,
+    {"coder_agent": "coder_agent", "finish_conversation": "finish_conversation"}
+)
+builder.add_edge("finish_conversation", END)
+
+# Configure persistence
+memory = MemorySaver()
+agentic_flow = builder.compile(checkpointer=memory)

original_archon/iterations/v2-agentic-workflow/crawl_pydantic_ai_docs.py

@@ -0,0 +1,258 @@
+import os
+import sys
+import json
+import asyncio
+import requests
+from xml.etree import ElementTree
+from typing import List, Dict, Any
+from dataclasses import dataclass
+from datetime import datetime, timezone
+from urllib.parse import urlparse
+from dotenv import load_dotenv
+
+from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
+from openai import AsyncOpenAI
+from supabase import create_client, Client
+
+load_dotenv()
+
+# Initialize OpenAI and Supabase clients
+
+base_url = os.getenv('BASE_URL', 'https://api.openai.com/v1')
+api_key = os.getenv('LLM_API_KEY', 'no-llm-api-key-provided')
+is_ollama = "localhost" in base_url.lower()
+
+embedding_model = os.getenv('EMBEDDING_MODEL', 'text-embedding-3-small')
+
+openai_client=None
+
+if is_ollama:
+    openai_client = AsyncOpenAI(base_url=base_url,api_key=api_key)
+else:
+    openai_client = AsyncOpenAI(api_key=os.getenv("OPENAI_API_KEY"))
+
+supabase: Client = create_client(
+    os.getenv("SUPABASE_URL"),
+    os.getenv("SUPABASE_SERVICE_KEY")
+)
+
+@dataclass
+class ProcessedChunk:
+    url: str
+    chunk_number: int
+    title: str
+    summary: str
+    content: str
+    metadata: Dict[str, Any]
+    embedding: List[float]
+
+def chunk_text(text: str, chunk_size: int = 5000) -> List[str]:
+    """Split text into chunks, respecting code blocks and paragraphs."""
+    chunks = []
+    start = 0
+    text_length = len(text)
+
+    while start < text_length:
+        # Calculate end position
+        end = start + chunk_size
+
+        # If we're at the end of the text, just take what's left
+        if end >= text_length:
+            chunks.append(text[start:].strip())
+            break
+
+        # Try to find a code block boundary first (```)
+        chunk = text[start:end]
+        code_block = chunk.rfind('```')
+        if code_block != -1 and code_block > chunk_size * 0.3:
+            end = start + code_block
+
+        # If no code block, try to break at a paragraph
+        elif '\n\n' in chunk:
+            # Find the last paragraph break
+            last_break = chunk.rfind('\n\n')
+            if last_break > chunk_size * 0.3:  # Only break if we're past 30% of chunk_size
+                end = start + last_break
+
+        # If no paragraph break, try to break at a sentence
+        elif '. ' in chunk:
+            # Find the last sentence break
+            last_period = chunk.rfind('. ')
+            if last_period > chunk_size * 0.3:  # Only break if we're past 30% of chunk_size
+                end = start + last_period + 1
+
+        # Extract chunk and clean it up
+        chunk = text[start:end].strip()
+        if chunk:
+            chunks.append(chunk)
+
+        # Move start position for next chunk
+        start = max(start + 1, end)
+
+    return chunks
+
+async def get_title_and_summary(chunk: str, url: str) -> Dict[str, str]:
+    """Extract title and summary using GPT-4."""
+    system_prompt = """You are an AI that extracts titles and summaries from documentation chunks.
+    Return a JSON object with 'title' and 'summary' keys.
+    For the title: If this seems like the start of a document, extract its title. If it's a middle chunk, derive a descriptive title.
+    For the summary: Create a concise summary of the main points in this chunk.
+    Keep both title and summary concise but informative."""
+    
+    try:
+        response = await openai_client.chat.completions.create(
+            model=os.getenv("PRIMARY_MODEL", "gpt-4o-mini"),
+            messages=[
+                {"role": "system", "content": system_prompt},
+                {"role": "user", "content": f"URL: {url}\n\nContent:\n{chunk[:1000]}..."}  # Send first 1000 chars for context
+            ],
+            response_format={ "type": "json_object" }
+        )
+        return json.loads(response.choices[0].message.content)
+    except Exception as e:
+        print(f"Error getting title and summary: {e}")
+        return {"title": "Error processing title", "summary": "Error processing summary"}
+
+async def get_embedding(text: str) -> List[float]:
+    """Get embedding vector from OpenAI."""
+    try:
+        response = await openai_client.embeddings.create(
+            model= embedding_model,
+            input=text
+        )
+        return response.data[0].embedding
+    except Exception as e:
+        print(f"Error getting embedding: {e}")
+        return [0] * 1536  # Return zero vector on error
+
+async def process_chunk(chunk: str, chunk_number: int, url: str) -> ProcessedChunk:
+    """Process a single chunk of text."""
+    # Get title and summary
+    extracted = await get_title_and_summary(chunk, url)
+    
+    # Get embedding
+    embedding = await get_embedding(chunk)
+    
+    # Create metadata
+    metadata = {
+        "source": "pydantic_ai_docs",
+        "chunk_size": len(chunk),
+        "crawled_at": datetime.now(timezone.utc).isoformat(),
+        "url_path": urlparse(url).path
+    }
+    
+    return ProcessedChunk(
+        url=url,
+        chunk_number=chunk_number,
+        title=extracted['title'],
+        summary=extracted['summary'],
+        content=chunk,  # Store the original chunk content
+        metadata=metadata,
+        embedding=embedding
+    )
+
+async def insert_chunk(chunk: ProcessedChunk):
+    """Insert a processed chunk into Supabase."""
+    try:
+        data = {
+            "url": chunk.url,
+            "chunk_number": chunk.chunk_number,
+            "title": chunk.title,
+            "summary": chunk.summary,
+            "content": chunk.content,
+            "metadata": chunk.metadata,
+            "embedding": chunk.embedding
+        }
+        
+        result = supabase.table("site_pages").insert(data).execute()
+        print(f"Inserted chunk {chunk.chunk_number} for {chunk.url}")
+        return result
+    except Exception as e:
+        print(f"Error inserting chunk: {e}")
+        return None
+
+async def process_and_store_document(url: str, markdown: str):
+    """Process a document and store its chunks in parallel."""
+    # Split into chunks
+    chunks = chunk_text(markdown)
+    
+    # Process chunks in parallel
+    tasks = [
+        process_chunk(chunk, i, url) 
+        for i, chunk in enumerate(chunks)
+    ]
+    processed_chunks = await asyncio.gather(*tasks)
+    
+    # Store chunks in parallel
+    insert_tasks = [
+        insert_chunk(chunk) 
+        for chunk in processed_chunks
+    ]
+    await asyncio.gather(*insert_tasks)
+
+async def crawl_parallel(urls: List[str], max_concurrent: int = 5):
+    """Crawl multiple URLs in parallel with a concurrency limit."""
+    browser_config = BrowserConfig(
+        headless=True,
+        verbose=False,
+        extra_args=["--disable-gpu", "--disable-dev-shm-usage", "--no-sandbox"],
+    )
+    crawl_config = CrawlerRunConfig(cache_mode=CacheMode.BYPASS)
+
+    # Create the crawler instance
+    crawler = AsyncWebCrawler(config=browser_config)
+    await crawler.start()
+
+    try:
+        # Create a semaphore to limit concurrency
+        semaphore = asyncio.Semaphore(max_concurrent)
+        
+        async def process_url(url: str):
+            async with semaphore:
+                result = await crawler.arun(
+                    url=url,
+                    config=crawl_config,
+                    session_id="session1"
+                )
+                if result.success:
+                    print(f"Successfully crawled: {url}")
+                    await process_and_store_document(url, result.markdown_v2.raw_markdown)
+                else:
+                    print(f"Failed: {url} - Error: {result.error_message}")
+        
+        # Process all URLs in parallel with limited concurrency
+        await asyncio.gather(*[process_url(url) for url in urls])
+    finally:
+        await crawler.close()
+
+def get_pydantic_ai_docs_urls() -> List[str]:
+    """Get URLs from Pydantic AI docs sitemap."""
+    sitemap_url = "https://ai.pydantic.dev/sitemap.xml"
+    try:
+        response = requests.get(sitemap_url)
+        response.raise_for_status()
+        
+        # Parse the XML
+        root = ElementTree.fromstring(response.content)
+        
+        # Extract all URLs from the sitemap
+        namespace = {'ns': 'http://www.sitemaps.org/schemas/sitemap/0.9'}
+        urls = [loc.text for loc in root.findall('.//ns:loc', namespace)]
+        
+        return urls
+    except Exception as e:
+        print(f"Error fetching sitemap: {e}")
+        return []
+
+async def main():
+    # Get URLs from Pydantic AI docs
+    urls = get_pydantic_ai_docs_urls()
+    if not urls:
+        print("No URLs found to crawl")
+        return
+    
+    print(f"Found {len(urls)} URLs to crawl")
+    await crawl_parallel(urls)
+
+if __name__ == "__main__":
+    asyncio.run(main())

original_archon/iterations/v2-agentic-workflow/langgraph.json

@@ -0,0 +1,7 @@
+{
+  "dependencies": ["."],
+  "graphs": {
+    "agent": "./archon_graph.py:agentic_flow"
+  },
+  "env": ".env"
+}

original_archon/iterations/v2-agentic-workflow/ollama_site_pages.sql

@@ -0,0 +1,72 @@
+-- Enable the pgvector extension
+create extension if not exists vector;
+
+-- Create the documentation chunks table
+create table site_pages (
+    id bigserial primary key,
+    url varchar not null,
+    chunk_number integer not null,
+    title varchar not null,
+    summary varchar not null,
+    content text not null,  -- Added content column
+    metadata jsonb not null default '{}'::jsonb,  -- Added metadata column
+    embedding vector(768),  -- Ollama nomic-embed-text embeddings are 768 dimensions
+    created_at timestamp with time zone default timezone('utc'::text, now()) not null,
+    
+    -- Add a unique constraint to prevent duplicate chunks for the same URL
+    unique(url, chunk_number)
+);
+
+-- Create an index for better vector similarity search performance
+create index on site_pages using ivfflat (embedding vector_cosine_ops);
+
+-- Create an index on metadata for faster filtering
+create index idx_site_pages_metadata on site_pages using gin (metadata);
+
+-- Create a function to search for documentation chunks
+create function match_site_pages (
+  query_embedding vector(768),
+  match_count int default 10,
+  filter jsonb DEFAULT '{}'::jsonb
+) returns table (
+  id bigint,
+  url varchar,
+  chunk_number integer,
+  title varchar,
+  summary varchar,
+  content text,
+  metadata jsonb,
+  similarity float
+)
+language plpgsql
+as $$
+#variable_conflict use_column
+begin
+  return query
+  select
+    id,
+    url,
+    chunk_number,
+    title,
+    summary,
+    content,
+    metadata,
+    1 - (site_pages.embedding <=> query_embedding) as similarity
+  from site_pages
+  where metadata @> filter
+  order by site_pages.embedding <=> query_embedding
+  limit match_count;
+end;
+$$;
+
+-- Everything above will work for any PostgreSQL database. The below commands are for Supabase security
+
+-- Enable RLS on the table
+alter table site_pages enable row level security;
+
+-- Create a policy that allows anyone to read
+create policy "Allow public read access"
+  on site_pages
+  for select
+  to public
+  using (true);

original_archon/iterations/v2-agentic-workflow/pydantic_ai_coder.py

@@ -0,0 +1,270 @@
+from __future__ import annotations as _annotations
+
+from dataclasses import dataclass
+from dotenv import load_dotenv
+import logfire
+import asyncio
+import httpx
+import os
+
+from pydantic_ai import Agent, ModelRetry, RunContext
+from pydantic_ai.models.openai import OpenAIModel
+from openai import AsyncOpenAI
+from supabase import Client
+from typing import List
+
+load_dotenv()
+
+llm = os.getenv('PRIMARY_MODEL', 'gpt-4o-mini')
+base_url = os.getenv('BASE_URL', 'https://api.openai.com/v1')
+api_key = os.getenv('LLM_API_KEY', 'no-llm-api-key-provided')
+model = OpenAIModel(llm, base_url=base_url, api_key=api_key)
+embedding_model = os.getenv('EMBEDDING_MODEL', 'text-embedding-3-small')
+
+logfire.configure(send_to_logfire='if-token-present')
+
+is_ollama = "localhost" in base_url.lower()
+
+@dataclass
+class PydanticAIDeps:
+    supabase: Client
+    openai_client: AsyncOpenAI
+    reasoner_output: str
+
+system_prompt = """
+[ROLE AND CONTEXT]
+You are a specialized AI agent engineer focused on building robust Pydantic AI agents. You have comprehensive access to the Pydantic AI documentation, including API references, usage guides, and implementation examples.
+
+[CORE RESPONSIBILITIES]
+1. Agent Development
+   - Create new agents from user requirements
+   - Complete partial agent implementations
+   - Optimize and debug existing agents
+   - Guide users through agent specification if needed
+
+2. Documentation Integration
+   - Systematically search documentation using RAG before any implementation
+   - Cross-reference multiple documentation pages for comprehensive understanding
+   - Validate all implementations against current best practices
+   - Notify users if documentation is insufficient for any requirement
+
+[CODE STRUCTURE AND DELIVERABLES]
+All new agents must include these files with complete, production-ready code:
+
+1. agent.py
+   - Primary agent definition and configuration
+   - Core agent logic and behaviors
+   - No tool implementations allowed here
+
+2. agent_tools.py
+   - All tool function implementations
+   - Tool configurations and setup
+   - External service integrations
+
+3. agent_prompts.py
+   - System prompts
+   - Task-specific prompts
+   - Conversation templates
+   - Instruction sets
+
+4. .env.example
+   - Required environment variables
+   - Clear setup instructions in a comment above the variable for how to do so
+   - API configuration templates
+
+5. requirements.txt
+   - Core dependencies without versions
+   - User-specified packages included
+
+[DOCUMENTATION WORKFLOW]
+1. Initial Research
+   - Begin with RAG search for relevant documentation
+   - List all documentation pages using list_documentation_pages
+   - Retrieve specific page content using get_page_content
+   - Cross-reference the weather agent example for best practices
+
+2. Implementation
+   - Provide complete, working code implementations
+   - Never leave placeholder functions
+   - Include all necessary error handling
+   - Implement proper logging and monitoring
+
+3. Quality Assurance
+   - Verify all tool implementations are complete
+   - Ensure proper separation of concerns
+   - Validate environment variable handling
+   - Test critical path functionality
+
+[INTERACTION GUIDELINES]
+- Take immediate action without asking for permission
+- Always verify documentation before implementation
+- Provide honest feedback about documentation gaps
+- Include specific enhancement suggestions
+- Request user feedback on implementations
+- Maintain code consistency across files
+
+[ERROR HANDLING]
+- Implement robust error handling in all tools
+- Provide clear error messages
+- Include recovery mechanisms
+- Log important state changes
+
+[BEST PRACTICES]
+- Follow Pydantic AI naming conventions
+- Implement proper type hints
+- Include comprehensive docstrings, the agent uses this to understand what tools are for.
+- Maintain clean code structure
+- Use consistent formatting
+"""
+
+pydantic_ai_coder = Agent(
+    model,
+    system_prompt=system_prompt,
+    deps_type=PydanticAIDeps,
+    retries=2
+)
+
+@pydantic_ai_coder.system_prompt  
+def add_reasoner_output(ctx: RunContext[str]) -> str:
+    return f"""
+    \n\nAdditional thoughts/instructions from the reasoner LLM. 
+    This scope includes documentation pages for you to search as well: 
+    {ctx.deps.reasoner_output}
+    """
+    
+    # Add this in to get some crazy tool calling:
+    # You must get ALL documentation pages listed in the scope.
+
+async def get_embedding(text: str, openai_client: AsyncOpenAI) -> List[float]:
+    """Get embedding vector from OpenAI."""
+    try:
+        response = await openai_client.embeddings.create(
+            model= embedding_model,
+            input=text
+        )
+        return response.data[0].embedding
+    except Exception as e:
+        print(f"Error getting embedding: {e}")
+        return [0] * 1536  # Return zero vector on error
+
+@pydantic_ai_coder.tool
+async def retrieve_relevant_documentation(ctx: RunContext[PydanticAIDeps], user_query: str) -> str:
+    """
+    Retrieve relevant documentation chunks based on the query with RAG.
+    
+    Args:
+        ctx: The context including the Supabase client and OpenAI client
+        user_query: The user's question or query
+        
+    Returns:
+        A formatted string containing the top 5 most relevant documentation chunks
+    """
+    try:
+        # Get the embedding for the query
+        query_embedding = await get_embedding(user_query, ctx.deps.openai_client)
+        
+        # Query Supabase for relevant documents
+        result = ctx.deps.supabase.rpc(
+            'match_site_pages',
+            {
+                'query_embedding': query_embedding,
+                'match_count': 5,
+                'filter': {'source': 'pydantic_ai_docs'}
+            }
+        ).execute()
+        
+        if not result.data:
+            return "No relevant documentation found."
+            
+        # Format the results
+        formatted_chunks = []
+        for doc in result.data:
+            chunk_text = f"""
+# {doc['title']}
+
+{doc['content']}
+"""
+            formatted_chunks.append(chunk_text)
+            
+        # Join all chunks with a separator
+        return "\n\n---\n\n".join(formatted_chunks)
+        
+    except Exception as e:
+        print(f"Error retrieving documentation: {e}")
+        return f"Error retrieving documentation: {str(e)}"
+
+async def list_documentation_pages_helper(supabase: Client) -> List[str]:
+    """
+    Function to retrieve a list of all available Pydantic AI documentation pages.
+    This is called by the list_documentation_pages tool and also externally
+    to fetch documentation pages for the reasoner LLM.
+    
+    Returns:
+        List[str]: List of unique URLs for all documentation pages
+    """
+    try:
+        # Query Supabase for unique URLs where source is pydantic_ai_docs
+        result = supabase.from_('site_pages') \
+            .select('url') \
+            .eq('metadata->>source', 'pydantic_ai_docs') \
+            .execute()
+        
+        if not result.data:
+            return []
+            
+        # Extract unique URLs
+        urls = sorted(set(doc['url'] for doc in result.data))
+        return urls
+        
+    except Exception as e:
+        print(f"Error retrieving documentation pages: {e}")
+        return []        
+
+@pydantic_ai_coder.tool
+async def list_documentation_pages(ctx: RunContext[PydanticAIDeps]) -> List[str]:
+    """
+    Retrieve a list of all available Pydantic AI documentation pages.
+    
+    Returns:
+        List[str]: List of unique URLs for all documentation pages
+    """
+    return await list_documentation_pages_helper(ctx.deps.supabase)
+
+@pydantic_ai_coder.tool
+async def get_page_content(ctx: RunContext[PydanticAIDeps], url: str) -> str:
+    """
+    Retrieve the full content of a specific documentation page by combining all its chunks.
+    
+    Args:
+        ctx: The context including the Supabase client
+        url: The URL of the page to retrieve
+        
+    Returns:
+        str: The complete page content with all chunks combined in order
+    """
+    try:
+        # Query Supabase for all chunks of this URL, ordered by chunk_number
+        result = ctx.deps.supabase.from_('site_pages') \
+            .select('title, content, chunk_number') \
+            .eq('url', url) \
+            .eq('metadata->>source', 'pydantic_ai_docs') \
+            .order('chunk_number') \
+            .execute()
+        
+        if not result.data:
+            return f"No content found for URL: {url}"
+            
+        # Format the page with its title and all chunks
+        page_title = result.data[0]['title'].split(' - ')[0]  # Get the main title
+        formatted_content = [f"# {page_title}\n"]
+        
+        # Add each chunk's content
+        for chunk in result.data:
+            formatted_content.append(chunk['content'])
+            
+        # Join everything together
+        return "\n\n".join(formatted_content)
+        
+    except Exception as e:
+        print(f"Error retrieving page content: {e}")
+        return f"Error retrieving page content: {str(e)}"

original_archon/iterations/v2-agentic-workflow/site_pages.sql

@@ -0,0 +1,72 @@
+-- Enable the pgvector extension
+create extension if not exists vector;
+
+-- Create the documentation chunks table
+create table site_pages (
+    id bigserial primary key,
+    url varchar not null,
+    chunk_number integer not null,
+    title varchar not null,
+    summary varchar not null,
+    content text not null,  -- Added content column
+    metadata jsonb not null default '{}'::jsonb,  -- Added metadata column
+    embedding vector(1536),  -- OpenAI embeddings are 1536 dimensions
+    created_at timestamp with time zone default timezone('utc'::text, now()) not null,
+    
+    -- Add a unique constraint to prevent duplicate chunks for the same URL
+    unique(url, chunk_number)
+);
+
+-- Create an index for better vector similarity search performance
+create index on site_pages using ivfflat (embedding vector_cosine_ops);
+
+-- Create an index on metadata for faster filtering
+create index idx_site_pages_metadata on site_pages using gin (metadata);
+
+-- Create a function to search for documentation chunks
+create function match_site_pages (
+  query_embedding vector(1536),
+  match_count int default 10,
+  filter jsonb DEFAULT '{}'::jsonb
+) returns table (
+  id bigint,
+  url varchar,
+  chunk_number integer,
+  title varchar,
+  summary varchar,
+  content text,
+  metadata jsonb,
+  similarity float
+)
+language plpgsql
+as $$
+#variable_conflict use_column
+begin
+  return query
+  select
+    id,
+    url,
+    chunk_number,
+    title,
+    summary,
+    content,
+    metadata,
+    1 - (site_pages.embedding <=> query_embedding) as similarity
+  from site_pages
+  where metadata @> filter
+  order by site_pages.embedding <=> query_embedding
+  limit match_count;
+end;
+$$;
+
+-- Everything above will work for any PostgreSQL database. The below commands are for Supabase security
+
+-- Enable RLS on the table
+alter table site_pages enable row level security;
+
+-- Create a policy that allows anyone to read
+create policy "Allow public read access"
+  on site_pages
+  for select
+  to public
+  using (true);

original_archon/iterations/v2-agentic-workflow/streamlit_ui.py

@@ -0,0 +1,124 @@
+from __future__ import annotations
+from typing import Literal, TypedDict
+from langgraph.types import Command
+from openai import AsyncOpenAI
+from supabase import Client
+import streamlit as st
+import logfire
+import asyncio
+import json
+import uuid
+import os
+
+# Import all the message part classes
+from pydantic_ai.messages import (
+    ModelMessage,
+    ModelRequest,
+    ModelResponse,
+    SystemPromptPart,
+    UserPromptPart,
+    TextPart,
+    ToolCallPart,
+    ToolReturnPart,
+    RetryPromptPart,
+    ModelMessagesTypeAdapter
+)
+
+from archon_graph import agentic_flow
+
+# Load environment variables
+from dotenv import load_dotenv
+load_dotenv()
+
+openai_client=None
+
+base_url = os.getenv('BASE_URL', 'https://api.openai.com/v1')
+api_key = os.getenv('LLM_API_KEY', 'no-llm-api-key-provided')
+is_ollama = "localhost" in base_url.lower()
+
+if is_ollama:
+    openai_client = AsyncOpenAI(base_url=base_url,api_key=api_key)
+else:
+    openai_client = AsyncOpenAI(api_key=os.getenv("OPENAI_API_KEY"))
+
+supabase: Client = Client(
+    os.getenv("SUPABASE_URL"),
+    os.getenv("SUPABASE_SERVICE_KEY")
+)
+
+# Configure logfire to suppress warnings (optional)
+logfire.configure(send_to_logfire='never')
+
+@st.cache_resource
+def get_thread_id():
+    return str(uuid.uuid4())
+
+thread_id = get_thread_id()
+
+async def run_agent_with_streaming(user_input: str):
+    """
+    Run the agent with streaming text for the user_input prompt,
+    while maintaining the entire conversation in `st.session_state.messages`.
+    """
+    config = {
+        "configurable": {
+            "thread_id": thread_id
+        }
+    }
+
+    # First message from user
+    if len(st.session_state.messages) == 1:
+        async for msg in agentic_flow.astream(
+                {"latest_user_message": user_input}, config, stream_mode="custom"
+            ):
+                yield msg
+    # Continue the conversation
+    else:
+        async for msg in agentic_flow.astream(
+            Command(resume=user_input), config, stream_mode="custom"
+        ):
+            yield msg
+
+
+async def main():
+    st.title("Archon - Agent Builder")
+    st.write("Describe to me an AI agent you want to build and I'll code it for you with Pydantic AI.")
+    st.write("Example: Build me an AI agent that can search the web with the Brave API.")
+
+    # Initialize chat history in session state if not present
+    if "messages" not in st.session_state:
+        st.session_state.messages = []
+
+    # Display chat messages from history on app rerun
+    for message in st.session_state.messages:
+        message_type = message["type"]
+        if message_type in ["human", "ai", "system"]:
+            with st.chat_message(message_type):
+                st.markdown(message["content"])    
+
+    # Chat input for the user
+    user_input = st.chat_input("What do you want to build today?")
+
+    if user_input:
+        # We append a new request to the conversation explicitly
+        st.session_state.messages.append({"type": "human", "content": user_input})
+        
+        # Display user prompt in the UI
+        with st.chat_message("user"):
+            st.markdown(user_input)
+
+        # Display assistant response in chat message container
+        response_content = ""
+        with st.chat_message("assistant"):
+            message_placeholder = st.empty()  # Placeholder for updating the message
+            # Run the async generator to fetch responses
+            async for chunk in run_agent_with_streaming(user_input):
+                response_content += chunk
+                # Update the placeholder with the current response content
+                message_placeholder.markdown(response_content)
+        
+        st.session_state.messages.append({"type": "ai", "content": response_content})
+
+
+if __name__ == "__main__":
+    asyncio.run(main())