Expand environment variables throughout mcp_settings.json configuration (#384)

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: samanhappy <2755122+samanhappy@users.noreply.github.com>
This commit is contained in:
Copilot
2025-10-26 19:25:53 +08:00
committed by GitHub
parent 5ca5e2ad47
commit f79028ed64
6 changed files with 728 additions and 21 deletions

View File

@@ -20,6 +20,7 @@ MCPHub makes it easy to manage and scale multiple MCP (Model Context Protocol) s
- **Group-Based Access Control**: Organize servers into customizable groups for streamlined permissions management.
- **Secure Authentication**: Built-in user management with role-based access powered by JWT and bcrypt.
- **OAuth 2.0 Support**: Full OAuth support for upstream MCP servers with proxy authorization capabilities.
- **Environment Variable Expansion**: Use environment variables anywhere in your configuration for secure credential management. See [Environment Variables Guide](docs/environment-variables.md).
- **Docker-Ready**: Deploy instantly with our containerized setup.
## 🔧 Quick Start

View File

@@ -0,0 +1,267 @@
# Environment Variable Expansion in mcp_settings.json
## Overview
MCPHub now supports comprehensive environment variable expansion throughout the entire `mcp_settings.json` configuration file. This allows you to externalize sensitive information and configuration values, making your setup more secure and flexible.
## Supported Formats
MCPHub supports two environment variable formats:
1. **${VAR}** - Standard format (recommended)
2. **$VAR** - Unix-style format (variable name must start with an uppercase letter or underscore, followed by uppercase letters, numbers, or underscores)
## What Can Be Expanded
Environment variables can now be used in **ANY** string value throughout your configuration:
- Server URLs
- Commands and arguments
- Headers
- Environment variables passed to child processes
- OpenAPI specifications and security configurations
- OAuth credentials
- System configuration values
- Any other string fields
## Examples
### 1. SSE/HTTP Server Configuration
```json
{
"mcpServers": {
"my-api-server": {
"type": "sse",
"url": "${MCP_SERVER_URL}",
"headers": {
"Authorization": "Bearer ${API_TOKEN}",
"X-Custom-Header": "${CUSTOM_VALUE}"
}
}
}
}
```
Environment variables:
```bash
export MCP_SERVER_URL="https://api.example.com/mcp"
export API_TOKEN="secret-token-123"
export CUSTOM_VALUE="my-custom-value"
```
### 2. Stdio Server Configuration
```json
{
"mcpServers": {
"my-python-server": {
"type": "stdio",
"command": "${PYTHON_PATH}",
"args": ["-m", "${MODULE_NAME}", "--api-key", "${API_KEY}"],
"env": {
"DATABASE_URL": "${DATABASE_URL}",
"DEBUG": "${DEBUG_MODE}"
}
}
}
}
```
Environment variables:
```bash
export PYTHON_PATH="/usr/bin/python3"
export MODULE_NAME="my_mcp_server"
export API_KEY="secret-api-key"
export DATABASE_URL="postgresql://localhost/mydb"
export DEBUG_MODE="true"
```
### 3. OpenAPI Server Configuration
```json
{
"mcpServers": {
"openapi-service": {
"type": "openapi",
"openapi": {
"url": "${OPENAPI_SPEC_URL}",
"security": {
"type": "apiKey",
"apiKey": {
"name": "X-API-Key",
"in": "header",
"value": "${OPENAPI_API_KEY}"
}
}
}
}
}
}
```
Environment variables:
```bash
export OPENAPI_SPEC_URL="https://api.example.com/openapi.json"
export OPENAPI_API_KEY="your-api-key-here"
```
### 4. OAuth Configuration
```json
{
"mcpServers": {
"oauth-server": {
"type": "sse",
"url": "${OAUTH_SERVER_URL}",
"oauth": {
"clientId": "${OAUTH_CLIENT_ID}",
"clientSecret": "${OAUTH_CLIENT_SECRET}",
"accessToken": "${OAUTH_ACCESS_TOKEN}"
}
}
}
}
```
Environment variables:
```bash
export OAUTH_SERVER_URL="https://oauth.example.com/mcp"
export OAUTH_CLIENT_ID="my-client-id"
export OAUTH_CLIENT_SECRET="my-client-secret"
export OAUTH_ACCESS_TOKEN="my-access-token"
```
### 5. System Configuration
```json
{
"systemConfig": {
"install": {
"pythonIndexUrl": "${PYTHON_INDEX_URL}",
"npmRegistry": "${NPM_REGISTRY}"
},
"mcpRouter": {
"apiKey": "${MCPROUTER_API_KEY}",
"referer": "${MCPROUTER_REFERER}"
}
}
}
```
Environment variables:
```bash
export PYTHON_INDEX_URL="https://pypi.tuna.tsinghua.edu.cn/simple"
export NPM_REGISTRY="https://registry.npmmirror.com"
export MCPROUTER_API_KEY="router-api-key"
export MCPROUTER_REFERER="https://myapp.com"
```
## Complete Example
See [examples/mcp_settings_with_env_vars.json](../examples/mcp_settings_with_env_vars.json) for a comprehensive example configuration using environment variables.
## Best Practices
### Security
1. **Never commit sensitive values to version control** - Use environment variables for all secrets
2. **Use .env files for local development** - MCPHub automatically loads `.env` files
3. **Use secure secret management in production** - Consider using Docker secrets, Kubernetes secrets, or cloud provider secret managers
### Organization
1. **Group related variables** - Use prefixes for related configuration (e.g., `API_`, `DB_`, `OAUTH_`)
2. **Document required variables** - Maintain a list of required environment variables in your README
3. **Provide example .env file** - Create a `.env.example` file with placeholder values
### Example .env File
```bash
# Server Configuration
MCP_SERVER_URL=https://api.example.com/mcp
API_TOKEN=your-api-token-here
# Python Server
PYTHON_PATH=/usr/bin/python3
MODULE_NAME=my_mcp_server
# Database
DATABASE_URL=postgresql://localhost/mydb
# OpenAPI
OPENAPI_SPEC_URL=https://api.example.com/openapi.json
OPENAPI_API_KEY=your-openapi-key
# OAuth
OAUTH_CLIENT_ID=your-client-id
OAUTH_CLIENT_SECRET=your-client-secret
OAUTH_ACCESS_TOKEN=your-access-token
```
## Docker Usage
When using Docker, pass environment variables using `-e` flag or `--env-file`:
```bash
# Using individual variables
docker run -e API_TOKEN=secret -e SERVER_URL=https://api.example.com mcphub
# Using env file
docker run --env-file .env mcphub
```
Or in docker-compose.yml:
```yaml
version: '3.8'
services:
mcphub:
image: mcphub
env_file:
- .env
environment:
- MCP_SERVER_URL=${MCP_SERVER_URL}
- API_TOKEN=${API_TOKEN}
```
## Troubleshooting
### Variable Not Expanding
If a variable is not expanding:
1. Check that the variable is set: `echo $VAR_NAME`
2. Verify the variable name matches exactly (case-sensitive)
3. Ensure the variable is exported: `export VAR_NAME=value`
4. Restart MCPHub after setting environment variables
### Empty Values
If an environment variable is not set, it will be replaced with an empty string. Make sure all required variables are set before starting MCPHub.
### Nested Variables
Environment variables in nested objects and arrays are fully supported:
```json
{
"nested": {
"deep": {
"value": "${MY_VAR}"
}
},
"array": ["${VAR1}", "${VAR2}"]
}
```
## Migration from Previous Version
If you were previously using environment variables only in headers, no changes are needed. The new implementation is backward compatible and simply extends support to all configuration fields.
## Technical Details
- Environment variables are expanded once when the configuration is loaded
- Expansion is recursive and handles nested objects and arrays
- Non-string values (booleans, numbers, null) are preserved as-is
- Empty string is used when an environment variable is not set

View File

@@ -0,0 +1,80 @@
{
"mcpServers": {
"example-sse-server": {
"type": "sse",
"url": "${MCP_SERVER_URL}",
"headers": {
"Authorization": "Bearer ${API_TOKEN}",
"X-Custom-Header": "${CUSTOM_HEADER_VALUE}"
},
"enabled": true
},
"example-streamable-http": {
"type": "streamable-http",
"url": "https://${SERVER_HOST}/mcp",
"headers": {
"API-Key": "${API_KEY}"
}
},
"example-stdio-server": {
"type": "stdio",
"command": "${PYTHON_PATH}",
"args": [
"-m",
"${MODULE_NAME}",
"--config",
"${CONFIG_PATH}"
],
"env": {
"API_KEY": "${MY_API_KEY}",
"DEBUG": "${DEBUG_MODE}",
"DATABASE_URL": "${DATABASE_URL}"
}
},
"example-openapi-server": {
"type": "openapi",
"openapi": {
"url": "${OPENAPI_SPEC_URL}",
"security": {
"type": "apiKey",
"apiKey": {
"name": "X-API-Key",
"in": "header",
"value": "${OPENAPI_API_KEY}"
}
}
},
"headers": {
"User-Agent": "MCPHub/${VERSION}"
}
},
"example-oauth-server": {
"type": "sse",
"url": "${OAUTH_SERVER_URL}",
"oauth": {
"clientId": "${OAUTH_CLIENT_ID}",
"clientSecret": "${OAUTH_CLIENT_SECRET}",
"accessToken": "${OAUTH_ACCESS_TOKEN}",
"scopes": ["read", "write"]
}
}
},
"users": [
{
"username": "admin",
"password": "${ADMIN_PASSWORD_HASH}",
"isAdmin": true
}
],
"systemConfig": {
"install": {
"pythonIndexUrl": "${PYTHON_INDEX_URL}",
"npmRegistry": "${NPM_REGISTRY}"
},
"mcpRouter": {
"apiKey": "${MCPROUTER_API_KEY}",
"referer": "${MCPROUTER_REFERER}",
"baseUrl": "${MCPROUTER_BASE_URL}"
}
}
}

View File

@@ -99,22 +99,33 @@ export function replaceEnvVars(input: string): string
export function replaceEnvVars(
input: Record<string, any> | string[] | string | undefined,
): Record<string, any> | string[] | string {
// Handle object input
// Handle object input - recursively expand all nested values
if (input && typeof input === 'object' && !Array.isArray(input)) {
const res: Record<string, string> = {}
const res: Record<string, any> = {}
for (const [key, value] of Object.entries(input)) {
if (typeof value === 'string') {
res[key] = expandEnvVars(value)
} else if (typeof value === 'object' && value !== null) {
// Recursively handle nested objects and arrays
res[key] = replaceEnvVars(value as any)
} else {
res[key] = String(value)
// Preserve non-string, non-object values (numbers, booleans, etc.)
res[key] = value
}
}
return res
}
// Handle array input
// Handle array input - recursively expand all elements
if (Array.isArray(input)) {
return input.map((item) => expandEnvVars(item))
return input.map((item) => {
if (typeof item === 'string') {
return expandEnvVars(item)
} else if (typeof item === 'object' && item !== null) {
return replaceEnvVars(item as any)
}
return item
})
}
// Handle string input

View File

@@ -235,6 +235,7 @@ export const createTransportFromConfig = async (name: string, conf: ServerConfig
env['npm_config_registry'] = settings.systemConfig.install.npmRegistry;
}
// Expand environment variables in command
transport = new StdioClientTransport({
cwd: os.homedir(),
command: conf.command,
@@ -379,12 +380,16 @@ export const initializeClientsFromSettings = async (
try {
for (const conf of allServers) {
const { name } = conf;
// Expand environment variables in all configuration values
const expandedConf = replaceEnvVars(conf as any) as ServerConfigWithName;
// Skip disabled servers
if (conf.enabled === false) {
if (expandedConf.enabled === false) {
console.log(`Skipping disabled server: ${name}`);
nextServerInfos.push({
name,
owner: conf.owner,
owner: expandedConf.owner,
status: 'disconnected',
error: null,
tools: [],
@@ -402,7 +407,7 @@ export const initializeClientsFromSettings = async (
if (existingServer && (!serverName || serverName !== name)) {
nextServerInfos.push({
...existingServer,
enabled: conf.enabled === undefined ? true : conf.enabled,
enabled: expandedConf.enabled === undefined ? true : expandedConf.enabled,
});
console.log(`Server '${name}' is already connected.`);
continue;
@@ -410,15 +415,15 @@ export const initializeClientsFromSettings = async (
let transport;
let openApiClient;
if (conf.type === 'openapi') {
if (expandedConf.type === 'openapi') {
// Handle OpenAPI type servers
if (!conf.openapi?.url && !conf.openapi?.schema) {
if (!expandedConf.openapi?.url && !expandedConf.openapi?.schema) {
console.warn(
`Skipping OpenAPI server '${name}': missing OpenAPI specification URL or schema`,
);
nextServerInfos.push({
name,
owner: conf.owner,
owner: expandedConf.owner,
status: 'disconnected',
error: 'Missing OpenAPI specification URL or schema',
tools: [],
@@ -431,20 +436,20 @@ export const initializeClientsFromSettings = async (
// Create server info first and keep reference to it
const serverInfo: ServerInfo = {
name,
owner: conf.owner,
owner: expandedConf.owner,
status: 'connecting',
error: null,
tools: [],
prompts: [],
createTime: Date.now(),
enabled: conf.enabled === undefined ? true : conf.enabled,
config: conf, // Store reference to original config for OpenAPI passthrough headers
enabled: expandedConf.enabled === undefined ? true : expandedConf.enabled,
config: expandedConf, // Store reference to expanded config for OpenAPI passthrough headers
};
nextServerInfos.push(serverInfo);
try {
// Create OpenAPI client instance
openApiClient = new OpenAPIClient(conf);
openApiClient = new OpenAPIClient(expandedConf);
console.log(`Initializing OpenAPI server: ${name}...`);
@@ -480,7 +485,7 @@ export const initializeClientsFromSettings = async (
continue;
}
} else {
transport = await createTransportFromConfig(name, conf);
transport = await createTransportFromConfig(name, expandedConf);
}
const client = new Client(
@@ -504,7 +509,7 @@ export const initializeClientsFromSettings = async (
: undefined;
// Get request options from server configuration, with fallbacks
const serverRequestOptions = conf.options || {};
const serverRequestOptions = expandedConf.options || {};
const requestOptions = {
timeout: serverRequestOptions.timeout || 60000,
resetTimeoutOnProgress: serverRequestOptions.resetTimeoutOnProgress || false,
@@ -514,7 +519,7 @@ export const initializeClientsFromSettings = async (
// Create server info first and keep reference to it
const serverInfo: ServerInfo = {
name,
owner: conf.owner,
owner: expandedConf.owner,
status: 'connecting',
error: null,
tools: [],
@@ -523,10 +528,10 @@ export const initializeClientsFromSettings = async (
transport,
options: requestOptions,
createTime: Date.now(),
config: conf, // Store reference to original config
config: expandedConf, // Store reference to expanded config
};
const pendingAuth = conf.oauth?.pendingAuthorization;
const pendingAuth = expandedConf.oauth?.pendingAuthorization;
if (pendingAuth) {
serverInfo.status = 'oauth_required';
serverInfo.error = null;
@@ -594,7 +599,7 @@ export const initializeClientsFromSettings = async (
serverInfo.error = null;
// Set up keep-alive ping for SSE connections
setupKeepAlive(serverInfo, conf);
setupKeepAlive(serverInfo, expandedConf);
} else {
serverInfo.status = 'disconnected';
serverInfo.error = `Failed to list data: ${dataError} `;

View File

@@ -0,0 +1,343 @@
import { replaceEnvVars, expandEnvVars } from '../../src/config/index.js';
describe('Environment Variable Expansion - Comprehensive Tests', () => {
const originalEnv = process.env;
beforeEach(() => {
// Reset process.env before each test
jest.resetModules();
process.env = { ...originalEnv };
});
afterAll(() => {
// Restore original environment
process.env = originalEnv;
});
describe('expandEnvVars - String expansion', () => {
it('should expand ${VAR} format', () => {
process.env.TEST_VAR = 'test-value';
expect(expandEnvVars('${TEST_VAR}')).toBe('test-value');
});
it('should expand $VAR format', () => {
process.env.TEST_VAR = 'test-value';
expect(expandEnvVars('$TEST_VAR')).toBe('test-value');
});
it('should expand multiple variables', () => {
process.env.HOST = 'localhost';
process.env.PORT = '3000';
expect(expandEnvVars('http://${HOST}:${PORT}')).toBe('http://localhost:3000');
});
it('should return empty string for undefined variables', () => {
expect(expandEnvVars('${UNDEFINED_VAR}')).toBe('');
});
it('should handle strings without variables', () => {
expect(expandEnvVars('plain-string')).toBe('plain-string');
});
it('should handle mixed variable formats', () => {
process.env.VAR1 = 'value1';
process.env.VAR2 = 'value2';
expect(expandEnvVars('$VAR1-${VAR2}')).toBe('value1-value2');
});
});
describe('replaceEnvVars - Recursive expansion', () => {
it('should expand environment variables in nested objects', () => {
process.env.API_KEY = 'secret123';
process.env.BASE_URL = 'https://api.example.com';
const config = {
url: '${BASE_URL}/endpoint',
headers: {
'X-API-Key': '${API_KEY}',
'Content-Type': 'application/json',
},
nested: {
value: '$API_KEY',
},
};
const result = replaceEnvVars(config);
expect(result).toEqual({
url: 'https://api.example.com/endpoint',
headers: {
'X-API-Key': 'secret123',
'Content-Type': 'application/json',
},
nested: {
value: 'secret123',
},
});
});
it('should expand environment variables in arrays', () => {
process.env.ARG1 = 'value1';
process.env.ARG2 = 'value2';
const args = ['--arg1', '${ARG1}', '--arg2', '${ARG2}'];
const result = replaceEnvVars(args);
expect(result).toEqual(['--arg1', 'value1', '--arg2', 'value2']);
});
it('should expand environment variables in nested arrays', () => {
process.env.ITEM = 'test-item';
const config = {
items: ['${ITEM}', 'static-item'],
};
const result = replaceEnvVars(config);
expect(result).toEqual({
items: ['test-item', 'static-item'],
});
});
it('should preserve non-string values', () => {
const config = {
enabled: true,
timeout: 3000,
ratio: 0.5,
nullable: null,
};
const result = replaceEnvVars(config);
expect(result).toEqual({
enabled: true,
timeout: 3000,
ratio: 0.5,
nullable: null,
});
});
it('should expand deeply nested structures', () => {
process.env.DEEP_VALUE = 'deep-secret';
const config = {
level1: {
level2: {
level3: {
value: '${DEEP_VALUE}',
},
},
},
};
const result = replaceEnvVars(config);
expect(result).toEqual({
level1: {
level2: {
level3: {
value: 'deep-secret',
},
},
},
});
});
it('should expand environment variables in mixed nested structures', () => {
process.env.VAR1 = 'value1';
process.env.VAR2 = 'value2';
const config = {
array: [
{
key: '${VAR1}',
},
{
key: '${VAR2}',
},
],
};
const result = replaceEnvVars(config);
expect(result).toEqual({
array: [
{
key: 'value1',
},
{
key: 'value2',
},
],
});
});
});
describe('ServerConfig scenarios', () => {
it('should expand URL with environment variables', () => {
process.env.SERVER_HOST = 'api.example.com';
process.env.SERVER_PORT = '8080';
const config = {
type: 'sse',
url: 'https://${SERVER_HOST}:${SERVER_PORT}/mcp',
};
const result = replaceEnvVars(config);
expect(result.url).toBe('https://api.example.com:8080/mcp');
});
it('should expand command with environment variables', () => {
process.env.PYTHON_PATH = '/usr/bin/python3';
const config = {
type: 'stdio',
command: '${PYTHON_PATH}',
args: ['-m', 'my_module'],
};
const result = replaceEnvVars(config);
expect(result.command).toBe('/usr/bin/python3');
});
it('should expand OpenAPI configuration', () => {
process.env.API_BASE_URL = 'https://api.example.com';
process.env.API_KEY = 'secret-key-123';
const config = {
type: 'openapi',
openapi: {
url: '${API_BASE_URL}/openapi.json',
security: {
type: 'apiKey',
apiKey: {
name: 'X-API-Key',
in: 'header',
value: '${API_KEY}',
},
},
},
};
const result = replaceEnvVars(config);
expect(result.openapi.url).toBe('https://api.example.com/openapi.json');
expect(result.openapi.security.apiKey.value).toBe('secret-key-123');
});
it('should expand OAuth configuration', () => {
process.env.CLIENT_ID = 'my-client-id';
process.env.CLIENT_SECRET = 'my-client-secret';
process.env.ACCESS_TOKEN = 'my-access-token';
const config = {
type: 'sse',
url: 'https://mcp.example.com',
oauth: {
clientId: '${CLIENT_ID}',
clientSecret: '${CLIENT_SECRET}',
accessToken: '${ACCESS_TOKEN}',
scopes: ['read', 'write'],
},
};
const result = replaceEnvVars(config);
expect(result.oauth.clientId).toBe('my-client-id');
expect(result.oauth.clientSecret).toBe('my-client-secret');
expect(result.oauth.accessToken).toBe('my-access-token');
expect(result.oauth.scopes).toEqual(['read', 'write']);
});
it('should expand environment variables in env object', () => {
process.env.API_KEY = 'my-api-key';
process.env.DEBUG = 'true';
const config = {
type: 'stdio',
command: 'node',
args: ['server.js'],
env: {
MY_API_KEY: '${API_KEY}',
DEBUG: '${DEBUG}',
},
};
const result = replaceEnvVars(config);
expect(result.env.MY_API_KEY).toBe('my-api-key');
expect(result.env.DEBUG).toBe('true');
});
it('should handle complete server configuration', () => {
process.env.SERVER_URL = 'https://mcp.example.com';
process.env.AUTH_TOKEN = 'bearer-token-123';
process.env.TIMEOUT = '60000';
const config = {
type: 'streamable-http',
url: '${SERVER_URL}/mcp',
headers: {
Authorization: 'Bearer ${AUTH_TOKEN}',
'User-Agent': 'MCPHub/1.0',
},
options: {
timeout: 30000,
},
enabled: true,
};
const result = replaceEnvVars(config);
expect(result.url).toBe('https://mcp.example.com/mcp');
expect(result.headers.Authorization).toBe('Bearer bearer-token-123');
expect(result.headers['User-Agent']).toBe('MCPHub/1.0');
expect(result.options.timeout).toBe(30000);
expect(result.enabled).toBe(true);
});
});
describe('Edge cases', () => {
it('should handle empty string values', () => {
const config = {
value: '',
};
const result = replaceEnvVars(config);
expect(result.value).toBe('');
});
it('should handle undefined values', () => {
const result = replaceEnvVars(undefined);
expect(result).toEqual([]);
});
it('should handle null values in objects', () => {
const config = {
value: null,
};
const result = replaceEnvVars(config);
expect(result.value).toBe(null);
});
it('should not break on circular references prevention', () => {
// Note: This test ensures we don't have infinite recursion issues
// by using a deeply nested structure
process.env.DEEP = 'value';
const config = {
a: { b: { c: { d: { e: { f: { g: { h: { i: { j: '${DEEP}' } } } } } } } } },
};
const result = replaceEnvVars(config);
expect(result.a.b.c.d.e.f.g.h.i.j).toBe('value');
});
});
});