Compare commits
1 Commits
af74841765
...
019-supers
| Author | SHA1 | Date | |
|---|---|---|---|
| 0cf0ef25f1 |
1250
.ai/MODULE_MAP.md
Normal file
1250
.ai/MODULE_MAP.md
Normal file
File diff suppressed because it is too large
Load Diff
@@ -3,7 +3,7 @@
|
||||
> Compressed view for AI Context. Generated automatically.
|
||||
|
||||
- 📦 **generate_semantic_map** (`Module`) `[CRITICAL]`
|
||||
- 📝 Scans the codebase to generate a Semantic Map and Compliance Report based on the System Standard.
|
||||
- 📝 Scans the codebase to generate a Semantic Map, Module Map, and Compliance Report based on the System Standard.
|
||||
- 🏗️ Layer: DevOps/Tooling
|
||||
- 🔒 Invariant: All DEF anchors must have matching closing anchors; TIER determines validation strictness.
|
||||
- ƒ **__init__** (`Function`) `[TRIVIAL]`
|
||||
@@ -71,35 +71,50 @@
|
||||
- 📝 Generates the token-optimized project map with enhanced Svelte details.
|
||||
- ƒ **_write_entity_md** (`Function`) `[CRITICAL]`
|
||||
- 📝 Recursive helper to write entity tree to Markdown with tier badges and enhanced details.
|
||||
- ƒ **_generate_module_map** (`Function`) `[CRITICAL]`
|
||||
- 📝 Generates a module-centric map grouping entities by directory structure.
|
||||
- ƒ **_get_module_path** (`Function`)
|
||||
- 📝 Extracts the module path from a file path.
|
||||
- ƒ **_collect_all_entities** (`Function`)
|
||||
- 📝 Flattens entity tree for easier grouping.
|
||||
- ƒ **to_dict** (`Function`) `[TRIVIAL]`
|
||||
- 📝 Auto-detected function (orphan)
|
||||
- 📦 **TransactionCore** (`Module`) `[CRITICAL]`
|
||||
- 📝 Core banking transaction processor with ACID guarantees.
|
||||
- 🏗️ Layer: Domain (Core)
|
||||
- 🔒 Invariant: Negative transfers are strictly forbidden.
|
||||
- 🔗 DEPENDS_ON -> `[DEF:Infra:PostgresDB]`
|
||||
- 🔗 DEPENDS_ON -> `[DEF:Infra:AuditLog]`
|
||||
- ƒ **execute_transfer** (`Function`)
|
||||
- 📝 Atomically move funds between accounts with audit trails.
|
||||
- 🔗 CALLS -> `atomic_transaction`
|
||||
- 📦 **PluginExampleShot** (`Module`)
|
||||
- 📝 Reference implementation of a plugin following GRACE standards.
|
||||
- 🔗 IMPLEMENTS -> `[DEF:Std:Plugin]`
|
||||
- 🏗️ Layer: Domain (Business Logic)
|
||||
- 🔒 Invariant: get_schema must return valid JSON Schema.
|
||||
- 🔗 INHERITS -> `PluginBase`
|
||||
- ƒ **get_schema** (`Function`)
|
||||
- 📝 Defines input validation schema.
|
||||
- ƒ **execute** (`Function`)
|
||||
- 📝 Core plugin logic with structured logging and progress reporting.
|
||||
- 📝 Core plugin logic with structured logging and scope isolation.
|
||||
- ƒ **id** (`Function`) `[TRIVIAL]`
|
||||
- 📝 Auto-detected function (orphan)
|
||||
- ƒ **name** (`Function`) `[TRIVIAL]`
|
||||
- 📝 Auto-detected function (orphan)
|
||||
- ƒ **description** (`Function`) `[TRIVIAL]`
|
||||
- 📝 Auto-detected function (orphan)
|
||||
- ƒ **version** (`Function`) `[TRIVIAL]`
|
||||
- 📝 Auto-detected function (orphan)
|
||||
- 📦 **BackendRouteShot** (`Module`)
|
||||
- 📝 Reference implementation of a task-based route using GRACE-Poly.
|
||||
- 🏗️ Layer: Interface (API)
|
||||
- 🔒 Invariant: TaskManager must be available in dependency graph.
|
||||
- 🔗 IMPLEMENTS -> `[DEF:Std:API_FastAPI]`
|
||||
- ƒ **create_task** (`Function`)
|
||||
- 📝 Create and start a new task using TaskManager. Non-blocking.
|
||||
- 🔗 CALLS -> `task_manager.create_task`
|
||||
- 🧩 **FrontendComponentShot** (`Component`)
|
||||
- 📝 Reference implementation of a task-spawning component using
|
||||
- 🏗️ Layer: UI
|
||||
- 🧩 **FrontendComponentShot** (`Component`) `[CRITICAL]`
|
||||
- 📝 Action button to spawn a new task with full UX feedback cycle.
|
||||
- 🏗️ Layer: UI (Presentation)
|
||||
- 🔒 Invariant: Must prevent double-submission while loading.
|
||||
- 📥 Props: plugin_id: any, params: any
|
||||
- ⬅️ READS_FROM `lib`
|
||||
- ⬅️ READS_FROM `t`
|
||||
- ƒ **spawnTask** (`Function`)
|
||||
- 📦 **DashboardTypes** (`Module`) `[TRIVIAL]`
|
||||
- 📝 TypeScript interfaces for Dashboard entities
|
||||
- 🏗️ Layer: Domain
|
||||
@@ -746,6 +761,7 @@
|
||||
- 🏗️ Layer: UI
|
||||
- ⚡ Events: cancel, resume
|
||||
- ➡️ WRITES_TO `props`
|
||||
- ➡️ WRITES_TO `state`
|
||||
- ⬅️ READS_FROM `effect`
|
||||
- ƒ **handleSubmit** (`Function`)
|
||||
- 📝 Validates and dispatches the passwords to resume the task.
|
||||
|
||||
@@ -30,6 +30,7 @@ Use these for code generation (Style Transfer).
|
||||
* Ref: `.ai/shots/critical_module.py` -> `[DEF:Shot:Critical_Module]`
|
||||
|
||||
## 3. DOMAIN MAP (Modules)
|
||||
* **Module Map:** `.ai/MODULE_MAP.md` -> `[DEF:Module_Map]`
|
||||
* **Project Map:** `.ai/PROJECT_MAP.md` -> `[DEF:Project_Map]`
|
||||
* **Backend Core:** `backend/src/core` -> `[DEF:Module:Backend_Core]`
|
||||
* **Backend API:** `backend/src/api` -> `[DEF:Module:Backend_API]`
|
||||
|
||||
27
.dockerignore
Normal file
27
.dockerignore
Normal file
@@ -0,0 +1,27 @@
|
||||
.git
|
||||
.gitignore
|
||||
.pytest_cache
|
||||
.ruff_cache
|
||||
.vscode
|
||||
.ai
|
||||
.specify
|
||||
.kilocode
|
||||
venv
|
||||
backend/.venv
|
||||
backend/.pytest_cache
|
||||
frontend/node_modules
|
||||
frontend/.svelte-kit
|
||||
frontend/.vite
|
||||
frontend/build
|
||||
backend/__pycache__
|
||||
backend/src/__pycache__
|
||||
backend/tests/__pycache__
|
||||
**/__pycache__
|
||||
*.pyc
|
||||
*.pyo
|
||||
*.pyd
|
||||
*.db
|
||||
*.log
|
||||
backups
|
||||
semantics
|
||||
specs
|
||||
36
Dockerfile
Normal file
36
Dockerfile
Normal file
@@ -0,0 +1,36 @@
|
||||
# Stage 1: Build frontend static assets
|
||||
FROM node:20-alpine AS frontend-build
|
||||
WORKDIR /app/frontend
|
||||
|
||||
COPY frontend/package*.json ./
|
||||
RUN npm ci
|
||||
|
||||
COPY frontend/ ./
|
||||
RUN npm run build
|
||||
|
||||
|
||||
# Stage 2: Runtime image for backend + static frontend
|
||||
FROM python:3.11-slim AS runtime
|
||||
|
||||
ENV PYTHONDONTWRITEBYTECODE=1
|
||||
ENV PYTHONUNBUFFERED=1
|
||||
ENV BACKEND_PORT=8000
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
curl \
|
||||
git \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
COPY backend/requirements.txt /app/backend/requirements.txt
|
||||
RUN pip install --no-cache-dir -r /app/backend/requirements.txt
|
||||
|
||||
COPY backend/ /app/backend/
|
||||
COPY --from=frontend-build /app/frontend/build /app/frontend/build
|
||||
|
||||
WORKDIR /app/backend
|
||||
|
||||
EXPOSE 8000
|
||||
|
||||
CMD ["python", "-m", "uvicorn", "src.app:app", "--host", "0.0.0.0", "--port", "8000"]
|
||||
69
README.md
69
README.md
@@ -32,7 +32,7 @@
|
||||
## Технологический стек
|
||||
- **Backend**: Python 3.9+, FastAPI, SQLAlchemy, APScheduler, Pydantic.
|
||||
- **Frontend**: Node.js 18+, SvelteKit, Tailwind CSS.
|
||||
- **Database**: SQLite (для хранения метаданных, задач и настроек доступа).
|
||||
- **Database**: PostgreSQL (для хранения метаданных, задач, логов и конфигурации).
|
||||
|
||||
## Структура проекта
|
||||
- `backend/` — Серверная часть, API и логика плагинов.
|
||||
@@ -58,11 +58,15 @@
|
||||
- `--skip-install`: Пропустить установку зависимостей.
|
||||
- `--help`: Показать справку.
|
||||
|
||||
Переменные окружения:
|
||||
- `BACKEND_PORT`: Порт API (по умолчанию 8000).
|
||||
- `FRONTEND_PORT`: Порт UI (по умолчанию 5173).
|
||||
Переменные окружения:
|
||||
- `BACKEND_PORT`: Порт API (по умолчанию 8000).
|
||||
- `FRONTEND_PORT`: Порт UI (по умолчанию 5173).
|
||||
- `POSTGRES_URL`: Базовый URL PostgreSQL по умолчанию для всех подсистем.
|
||||
- `DATABASE_URL`: URL основной БД (если не задан, используется `POSTGRES_URL`).
|
||||
- `TASKS_DATABASE_URL`: URL БД задач/логов (если не задан, используется `DATABASE_URL`).
|
||||
- `AUTH_DATABASE_URL`: URL БД авторизации (если не задан, используется PostgreSQL дефолт).
|
||||
|
||||
## Разработка
|
||||
## Разработка
|
||||
Проект следует строгим правилам разработки:
|
||||
1. **Semantic Code Generation**: Использование протокола `.ai/standards/semantics.md` для обеспечения надежности кода.
|
||||
2. **Design by Contract (DbC)**: Определение предусловий и постусловий для ключевых функций.
|
||||
@@ -71,7 +75,54 @@
|
||||
### Полезные команды
|
||||
- **Backend**: `cd backend && .venv/bin/python3 -m uvicorn src.app:app --reload`
|
||||
- **Frontend**: `cd frontend && npm run dev`
|
||||
- **Тесты**: `cd backend && .venv/bin/pytest`
|
||||
|
||||
## Контакты и вклад
|
||||
Для добавления новых функций или исправления ошибок, пожалуйста, ознакомьтесь с `docs/plugin_dev.md` и создайте соответствующую спецификацию в `specs/`.
|
||||
- **Тесты**: `cd backend && .venv/bin/pytest`
|
||||
|
||||
## Docker и CI/CD
|
||||
### Локальный запуск в Docker (приложение + PostgreSQL)
|
||||
```bash
|
||||
docker compose up --build
|
||||
```
|
||||
|
||||
После старта:
|
||||
- UI/API: `http://localhost:8000`
|
||||
- PostgreSQL: `localhost:5432` (`postgres/postgres`, DB `ss_tools`)
|
||||
|
||||
Остановить:
|
||||
```bash
|
||||
docker compose down
|
||||
```
|
||||
|
||||
Полная очистка тома БД:
|
||||
```bash
|
||||
docker compose down -v
|
||||
```
|
||||
|
||||
Если `postgres:16-alpine` не тянется из Docker Hub (TLS timeout), используйте fallback image:
|
||||
```bash
|
||||
POSTGRES_IMAGE=mirror.gcr.io/library/postgres:16-alpine docker compose up -d db
|
||||
```
|
||||
или:
|
||||
```bash
|
||||
POSTGRES_IMAGE=bitnami/postgresql:latest docker compose up -d db
|
||||
```
|
||||
Если на хосте уже занят `5432`, поднимайте Postgres на другом порту:
|
||||
```bash
|
||||
POSTGRES_HOST_PORT=5433 docker compose up -d db
|
||||
```
|
||||
|
||||
### Миграция legacy-данных в PostgreSQL
|
||||
Если нужно перенести старые данные из `tasks.db`/`config.json`:
|
||||
```bash
|
||||
cd backend
|
||||
PYTHONPATH=. .venv/bin/python src/scripts/migrate_sqlite_to_postgres.py --sqlite-path tasks.db
|
||||
```
|
||||
|
||||
### CI/CD
|
||||
Добавлен workflow: `.github/workflows/ci-cd.yml`
|
||||
- backend smoke tests
|
||||
- frontend build
|
||||
- docker build
|
||||
- push образа в GHCR на `main/master`
|
||||
|
||||
## Контакты и вклад
|
||||
Для добавления новых функций или исправления ошибок, пожалуйста, ознакомьтесь с `docs/plugin_dev.md` и создайте соответствующую спецификацию в `specs/`.
|
||||
|
||||
@@ -24,7 +24,10 @@ class AuthConfig(BaseSettings):
|
||||
REFRESH_TOKEN_EXPIRE_DAYS: int = 7
|
||||
|
||||
# Database Settings
|
||||
AUTH_DATABASE_URL: str = Field(default="sqlite:///./backend/auth.db", env="AUTH_DATABASE_URL")
|
||||
AUTH_DATABASE_URL: str = Field(
|
||||
default="postgresql+psycopg2://postgres:postgres@localhost:5432/ss_tools",
|
||||
env="AUTH_DATABASE_URL",
|
||||
)
|
||||
|
||||
# ADFS Settings
|
||||
ADFS_CLIENT_ID: str = Field(default="", env="ADFS_CLIENT_ID")
|
||||
@@ -41,4 +44,4 @@ class AuthConfig(BaseSettings):
|
||||
auth_config = AuthConfig()
|
||||
# [/DEF:auth_config:Variable]
|
||||
|
||||
# [/DEF:backend.src.core.auth.config:Module]
|
||||
# [/DEF:backend.src.core.auth.config:Module]
|
||||
|
||||
567
backend/src/core/config_manager.py
Executable file → Normal file
567
backend/src/core/config_manager.py
Executable file → Normal file
@@ -1,284 +1,283 @@
|
||||
# [DEF:ConfigManagerModule:Module]
|
||||
#
|
||||
# @SEMANTICS: config, manager, persistence, json
|
||||
# @PURPOSE: Manages application configuration, including loading/saving to JSON and CRUD for environments.
|
||||
# @LAYER: Core
|
||||
# @RELATION: DEPENDS_ON -> ConfigModels
|
||||
# @RELATION: CALLS -> logger
|
||||
# @RELATION: WRITES_TO -> config.json
|
||||
#
|
||||
# @INVARIANT: Configuration must always be valid according to AppConfig model.
|
||||
# @PUBLIC_API: ConfigManager
|
||||
|
||||
# [SECTION: IMPORTS]
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Optional, List
|
||||
from .config_models import AppConfig, Environment, GlobalSettings, StorageConfig
|
||||
from .logger import logger, configure_logger, belief_scope
|
||||
# [/SECTION]
|
||||
|
||||
# [DEF:ConfigManager:Class]
|
||||
# @PURPOSE: A class to handle application configuration persistence and management.
|
||||
# @RELATION: WRITES_TO -> config.json
|
||||
class ConfigManager:
|
||||
|
||||
# [DEF:__init__:Function]
|
||||
# @PURPOSE: Initializes the ConfigManager.
|
||||
# @PRE: isinstance(config_path, str) and len(config_path) > 0
|
||||
# @POST: self.config is an instance of AppConfig
|
||||
# @PARAM: config_path (str) - Path to the configuration file.
|
||||
def __init__(self, config_path: str = "config.json"):
|
||||
with belief_scope("__init__"):
|
||||
# 1. Runtime check of @PRE
|
||||
assert isinstance(config_path, str) and config_path, "config_path must be a non-empty string"
|
||||
|
||||
logger.info(f"[ConfigManager][Entry] Initializing with {config_path}")
|
||||
|
||||
# 2. Logic implementation
|
||||
self.config_path = Path(config_path)
|
||||
self.config: AppConfig = self._load_config()
|
||||
|
||||
# Configure logger with loaded settings
|
||||
configure_logger(self.config.settings.logging)
|
||||
|
||||
# 3. Runtime check of @POST
|
||||
assert isinstance(self.config, AppConfig), "self.config must be an instance of AppConfig"
|
||||
|
||||
logger.info("[ConfigManager][Exit] Initialized")
|
||||
# [/DEF:__init__:Function]
|
||||
|
||||
# [DEF:_load_config:Function]
|
||||
# @PURPOSE: Loads the configuration from disk or creates a default one.
|
||||
# @PRE: self.config_path is set.
|
||||
# @POST: isinstance(return, AppConfig)
|
||||
# @RETURN: AppConfig - The loaded or default configuration.
|
||||
def _load_config(self) -> AppConfig:
|
||||
with belief_scope("_load_config"):
|
||||
logger.debug(f"[_load_config][Entry] Loading from {self.config_path}")
|
||||
|
||||
if not self.config_path.exists():
|
||||
logger.info("[_load_config][Action] Config file not found. Creating default.")
|
||||
default_config = AppConfig(
|
||||
environments=[],
|
||||
settings=GlobalSettings()
|
||||
)
|
||||
self._save_config_to_disk(default_config)
|
||||
return default_config
|
||||
try:
|
||||
with open(self.config_path, "r") as f:
|
||||
data = json.load(f)
|
||||
|
||||
# Check for deprecated field
|
||||
if "settings" in data and "backup_path" in data["settings"]:
|
||||
del data["settings"]["backup_path"]
|
||||
|
||||
config = AppConfig(**data)
|
||||
logger.info("[_load_config][Coherence:OK] Configuration loaded")
|
||||
return config
|
||||
except Exception as e:
|
||||
logger.error(f"[_load_config][Coherence:Failed] Error loading config: {e}")
|
||||
# Fallback but try to preserve existing settings if possible?
|
||||
# For now, return default to be safe, but log the error prominently.
|
||||
return AppConfig(
|
||||
environments=[],
|
||||
settings=GlobalSettings(storage=StorageConfig())
|
||||
)
|
||||
# [/DEF:_load_config:Function]
|
||||
|
||||
# [DEF:_save_config_to_disk:Function]
|
||||
# @PURPOSE: Saves the provided configuration object to disk.
|
||||
# @PRE: isinstance(config, AppConfig)
|
||||
# @POST: Configuration saved to disk.
|
||||
# @PARAM: config (AppConfig) - The configuration to save.
|
||||
def _save_config_to_disk(self, config: AppConfig):
|
||||
with belief_scope("_save_config_to_disk"):
|
||||
logger.debug(f"[_save_config_to_disk][Entry] Saving to {self.config_path}")
|
||||
|
||||
# 1. Runtime check of @PRE
|
||||
assert isinstance(config, AppConfig), "config must be an instance of AppConfig"
|
||||
|
||||
# 2. Logic implementation
|
||||
try:
|
||||
with open(self.config_path, "w") as f:
|
||||
json.dump(config.dict(), f, indent=4)
|
||||
logger.info("[_save_config_to_disk][Action] Configuration saved")
|
||||
except Exception as e:
|
||||
logger.error(f"[_save_config_to_disk][Coherence:Failed] Failed to save: {e}")
|
||||
# [/DEF:_save_config_to_disk:Function]
|
||||
|
||||
# [DEF:save:Function]
|
||||
# @PURPOSE: Saves the current configuration state to disk.
|
||||
# @PRE: self.config is set.
|
||||
# @POST: self._save_config_to_disk called.
|
||||
def save(self):
|
||||
with belief_scope("save"):
|
||||
self._save_config_to_disk(self.config)
|
||||
# [/DEF:save:Function]
|
||||
|
||||
# [DEF:get_config:Function]
|
||||
# @PURPOSE: Returns the current configuration.
|
||||
# @PRE: self.config is set.
|
||||
# @POST: Returns self.config.
|
||||
# @RETURN: AppConfig - The current configuration.
|
||||
def get_config(self) -> AppConfig:
|
||||
with belief_scope("get_config"):
|
||||
return self.config
|
||||
# [/DEF:get_config:Function]
|
||||
|
||||
# [DEF:update_global_settings:Function]
|
||||
# @PURPOSE: Updates the global settings and persists the change.
|
||||
# @PRE: isinstance(settings, GlobalSettings)
|
||||
# @POST: self.config.settings updated and saved.
|
||||
# @PARAM: settings (GlobalSettings) - The new global settings.
|
||||
def update_global_settings(self, settings: GlobalSettings):
|
||||
with belief_scope("update_global_settings"):
|
||||
logger.info("[update_global_settings][Entry] Updating settings")
|
||||
|
||||
# 1. Runtime check of @PRE
|
||||
assert isinstance(settings, GlobalSettings), "settings must be an instance of GlobalSettings"
|
||||
|
||||
# 2. Logic implementation
|
||||
self.config.settings = settings
|
||||
self.save()
|
||||
|
||||
# Reconfigure logger with new settings
|
||||
configure_logger(settings.logging)
|
||||
|
||||
logger.info("[update_global_settings][Exit] Settings updated")
|
||||
# [/DEF:update_global_settings:Function]
|
||||
|
||||
# [DEF:validate_path:Function]
|
||||
# @PURPOSE: Validates if a path exists and is writable.
|
||||
# @PRE: path is a string.
|
||||
# @POST: Returns (bool, str) status.
|
||||
# @PARAM: path (str) - The path to validate.
|
||||
# @RETURN: tuple (bool, str) - (is_valid, message)
|
||||
def validate_path(self, path: str) -> tuple[bool, str]:
|
||||
with belief_scope("validate_path"):
|
||||
p = os.path.abspath(path)
|
||||
if not os.path.exists(p):
|
||||
try:
|
||||
os.makedirs(p, exist_ok=True)
|
||||
except Exception as e:
|
||||
return False, f"Path does not exist and could not be created: {e}"
|
||||
|
||||
if not os.access(p, os.W_OK):
|
||||
return False, "Path is not writable"
|
||||
|
||||
return True, "Path is valid and writable"
|
||||
# [/DEF:validate_path:Function]
|
||||
|
||||
# [DEF:get_environments:Function]
|
||||
# @PURPOSE: Returns the list of configured environments.
|
||||
# @PRE: self.config is set.
|
||||
# @POST: Returns list of environments.
|
||||
# @RETURN: List[Environment] - List of environments.
|
||||
def get_environments(self) -> List[Environment]:
|
||||
with belief_scope("get_environments"):
|
||||
return self.config.environments
|
||||
# [/DEF:get_environments:Function]
|
||||
|
||||
# [DEF:has_environments:Function]
|
||||
# @PURPOSE: Checks if at least one environment is configured.
|
||||
# @PRE: self.config is set.
|
||||
# @POST: Returns boolean indicating if environments exist.
|
||||
# @RETURN: bool - True if at least one environment exists.
|
||||
def has_environments(self) -> bool:
|
||||
with belief_scope("has_environments"):
|
||||
return len(self.config.environments) > 0
|
||||
# [/DEF:has_environments:Function]
|
||||
|
||||
# [DEF:get_environment:Function]
|
||||
# @PURPOSE: Returns a single environment by ID.
|
||||
# @PRE: self.config is set and isinstance(env_id, str) and len(env_id) > 0.
|
||||
# @POST: Returns Environment object if found, None otherwise.
|
||||
# @PARAM: env_id (str) - The ID of the environment to retrieve.
|
||||
# @RETURN: Optional[Environment] - The environment with the given ID, or None.
|
||||
def get_environment(self, env_id: str) -> Optional[Environment]:
|
||||
with belief_scope("get_environment"):
|
||||
for env in self.config.environments:
|
||||
if env.id == env_id:
|
||||
return env
|
||||
return None
|
||||
# [/DEF:get_environment:Function]
|
||||
|
||||
# [DEF:add_environment:Function]
|
||||
# @PURPOSE: Adds a new environment to the configuration.
|
||||
# @PRE: isinstance(env, Environment)
|
||||
# @POST: Environment added or updated in self.config.environments.
|
||||
# @PARAM: env (Environment) - The environment to add.
|
||||
def add_environment(self, env: Environment):
|
||||
with belief_scope("add_environment"):
|
||||
logger.info(f"[add_environment][Entry] Adding environment {env.id}")
|
||||
|
||||
# 1. Runtime check of @PRE
|
||||
assert isinstance(env, Environment), "env must be an instance of Environment"
|
||||
|
||||
# 2. Logic implementation
|
||||
# Check for duplicate ID and remove if exists
|
||||
self.config.environments = [e for e in self.config.environments if e.id != env.id]
|
||||
self.config.environments.append(env)
|
||||
self.save()
|
||||
|
||||
logger.info("[add_environment][Exit] Environment added")
|
||||
# [/DEF:add_environment:Function]
|
||||
|
||||
# [DEF:update_environment:Function]
|
||||
# @PURPOSE: Updates an existing environment.
|
||||
# @PRE: isinstance(env_id, str) and len(env_id) > 0 and isinstance(updated_env, Environment)
|
||||
# @POST: Returns True if environment was found and updated.
|
||||
# @PARAM: env_id (str) - The ID of the environment to update.
|
||||
# @PARAM: updated_env (Environment) - The updated environment data.
|
||||
# @RETURN: bool - True if updated, False otherwise.
|
||||
def update_environment(self, env_id: str, updated_env: Environment) -> bool:
|
||||
with belief_scope("update_environment"):
|
||||
logger.info(f"[update_environment][Entry] Updating {env_id}")
|
||||
|
||||
# 1. Runtime check of @PRE
|
||||
assert env_id and isinstance(env_id, str), "env_id must be a non-empty string"
|
||||
assert isinstance(updated_env, Environment), "updated_env must be an instance of Environment"
|
||||
|
||||
# 2. Logic implementation
|
||||
for i, env in enumerate(self.config.environments):
|
||||
if env.id == env_id:
|
||||
# If password is masked, keep the old one
|
||||
if updated_env.password == "********":
|
||||
updated_env.password = env.password
|
||||
|
||||
self.config.environments[i] = updated_env
|
||||
self.save()
|
||||
logger.info(f"[update_environment][Coherence:OK] Updated {env_id}")
|
||||
return True
|
||||
|
||||
logger.warning(f"[update_environment][Coherence:Failed] Environment {env_id} not found")
|
||||
return False
|
||||
# [/DEF:update_environment:Function]
|
||||
|
||||
# [DEF:delete_environment:Function]
|
||||
# @PURPOSE: Deletes an environment by ID.
|
||||
# @PRE: isinstance(env_id, str) and len(env_id) > 0
|
||||
# @POST: Environment removed from self.config.environments if it existed.
|
||||
# @PARAM: env_id (str) - The ID of the environment to delete.
|
||||
def delete_environment(self, env_id: str):
|
||||
with belief_scope("delete_environment"):
|
||||
logger.info(f"[delete_environment][Entry] Deleting {env_id}")
|
||||
|
||||
# 1. Runtime check of @PRE
|
||||
assert env_id and isinstance(env_id, str), "env_id must be a non-empty string"
|
||||
|
||||
# 2. Logic implementation
|
||||
original_count = len(self.config.environments)
|
||||
self.config.environments = [e for e in self.config.environments if e.id != env_id]
|
||||
|
||||
if len(self.config.environments) < original_count:
|
||||
self.save()
|
||||
logger.info(f"[delete_environment][Action] Deleted {env_id}")
|
||||
else:
|
||||
logger.warning(f"[delete_environment][Coherence:Failed] Environment {env_id} not found")
|
||||
# [/DEF:delete_environment:Function]
|
||||
|
||||
# [/DEF:ConfigManager:Class]
|
||||
|
||||
# [/DEF:ConfigManagerModule:Module]
|
||||
# [DEF:ConfigManagerModule:Module]
|
||||
#
|
||||
# @SEMANTICS: config, manager, persistence, postgresql
|
||||
# @PURPOSE: Manages application configuration persisted in database with one-time migration from JSON.
|
||||
# @LAYER: Core
|
||||
# @RELATION: DEPENDS_ON -> ConfigModels
|
||||
# @RELATION: DEPENDS_ON -> AppConfigRecord
|
||||
# @RELATION: CALLS -> logger
|
||||
#
|
||||
# @INVARIANT: Configuration must always be valid according to AppConfig model.
|
||||
# @PUBLIC_API: ConfigManager
|
||||
|
||||
# [SECTION: IMPORTS]
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Optional, List
|
||||
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from .config_models import AppConfig, Environment, GlobalSettings, StorageConfig
|
||||
from .database import SessionLocal
|
||||
from ..models.config import AppConfigRecord
|
||||
from .logger import logger, configure_logger, belief_scope
|
||||
# [/SECTION]
|
||||
|
||||
|
||||
# [DEF:ConfigManager:Class]
|
||||
# @PURPOSE: A class to handle application configuration persistence and management.
|
||||
class ConfigManager:
|
||||
# [DEF:__init__:Function]
|
||||
# @PURPOSE: Initializes the ConfigManager.
|
||||
# @PRE: isinstance(config_path, str) and len(config_path) > 0
|
||||
# @POST: self.config is an instance of AppConfig
|
||||
# @PARAM: config_path (str) - Path to legacy JSON config (used only for initial migration fallback).
|
||||
def __init__(self, config_path: str = "config.json"):
|
||||
with belief_scope("__init__"):
|
||||
assert isinstance(config_path, str) and config_path, "config_path must be a non-empty string"
|
||||
|
||||
logger.info(f"[ConfigManager][Entry] Initializing with legacy path {config_path}")
|
||||
|
||||
self.config_path = Path(config_path)
|
||||
self.config: AppConfig = self._load_config()
|
||||
|
||||
configure_logger(self.config.settings.logging)
|
||||
assert isinstance(self.config, AppConfig), "self.config must be an instance of AppConfig"
|
||||
|
||||
logger.info("[ConfigManager][Exit] Initialized")
|
||||
# [/DEF:__init__:Function]
|
||||
|
||||
# [DEF:_default_config:Function]
|
||||
# @PURPOSE: Returns default application configuration.
|
||||
# @RETURN: AppConfig - Default configuration.
|
||||
def _default_config(self) -> AppConfig:
|
||||
return AppConfig(
|
||||
environments=[],
|
||||
settings=GlobalSettings(storage=StorageConfig()),
|
||||
)
|
||||
# [/DEF:_default_config:Function]
|
||||
|
||||
# [DEF:_load_from_legacy_file:Function]
|
||||
# @PURPOSE: Loads legacy configuration from config.json for migration fallback.
|
||||
# @RETURN: AppConfig - Loaded or default configuration.
|
||||
def _load_from_legacy_file(self) -> AppConfig:
|
||||
with belief_scope("_load_from_legacy_file"):
|
||||
if not self.config_path.exists():
|
||||
logger.info("[_load_from_legacy_file][Action] Legacy config file not found, using defaults")
|
||||
return self._default_config()
|
||||
|
||||
try:
|
||||
with open(self.config_path, "r", encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
logger.info("[_load_from_legacy_file][Coherence:OK] Legacy configuration loaded")
|
||||
return AppConfig(**data)
|
||||
except Exception as e:
|
||||
logger.error(f"[_load_from_legacy_file][Coherence:Failed] Error loading legacy config: {e}")
|
||||
return self._default_config()
|
||||
# [/DEF:_load_from_legacy_file:Function]
|
||||
|
||||
# [DEF:_get_record:Function]
|
||||
# @PURPOSE: Loads config record from DB.
|
||||
# @PARAM: session (Session) - DB session.
|
||||
# @RETURN: Optional[AppConfigRecord] - Existing record or None.
|
||||
def _get_record(self, session: Session) -> Optional[AppConfigRecord]:
|
||||
return session.query(AppConfigRecord).filter(AppConfigRecord.id == "global").first()
|
||||
# [/DEF:_get_record:Function]
|
||||
|
||||
# [DEF:_load_config:Function]
|
||||
# @PURPOSE: Loads the configuration from DB or performs one-time migration from JSON file.
|
||||
# @PRE: DB session factory is available.
|
||||
# @POST: isinstance(return, AppConfig)
|
||||
# @RETURN: AppConfig - Loaded configuration.
|
||||
def _load_config(self) -> AppConfig:
|
||||
with belief_scope("_load_config"):
|
||||
session: Session = SessionLocal()
|
||||
try:
|
||||
record = self._get_record(session)
|
||||
if record and record.payload:
|
||||
logger.info("[_load_config][Coherence:OK] Configuration loaded from database")
|
||||
return AppConfig(**record.payload)
|
||||
|
||||
logger.info("[_load_config][Action] No database config found, migrating legacy config")
|
||||
config = self._load_from_legacy_file()
|
||||
self._save_config_to_db(config, session=session)
|
||||
return config
|
||||
except Exception as e:
|
||||
logger.error(f"[_load_config][Coherence:Failed] Error loading config from DB: {e}")
|
||||
return self._default_config()
|
||||
finally:
|
||||
session.close()
|
||||
# [/DEF:_load_config:Function]
|
||||
|
||||
# [DEF:_save_config_to_db:Function]
|
||||
# @PURPOSE: Saves the provided configuration object to DB.
|
||||
# @PRE: isinstance(config, AppConfig)
|
||||
# @POST: Configuration saved to database.
|
||||
# @PARAM: config (AppConfig) - The configuration to save.
|
||||
# @PARAM: session (Optional[Session]) - Existing DB session for transactional reuse.
|
||||
def _save_config_to_db(self, config: AppConfig, session: Optional[Session] = None):
|
||||
with belief_scope("_save_config_to_db"):
|
||||
assert isinstance(config, AppConfig), "config must be an instance of AppConfig"
|
||||
|
||||
owns_session = session is None
|
||||
db = session or SessionLocal()
|
||||
try:
|
||||
record = self._get_record(db)
|
||||
payload = config.model_dump()
|
||||
if record is None:
|
||||
record = AppConfigRecord(id="global", payload=payload)
|
||||
db.add(record)
|
||||
else:
|
||||
record.payload = payload
|
||||
db.commit()
|
||||
logger.info("[_save_config_to_db][Action] Configuration saved to database")
|
||||
except Exception as e:
|
||||
db.rollback()
|
||||
logger.error(f"[_save_config_to_db][Coherence:Failed] Failed to save: {e}")
|
||||
raise
|
||||
finally:
|
||||
if owns_session:
|
||||
db.close()
|
||||
# [/DEF:_save_config_to_db:Function]
|
||||
|
||||
# [DEF:save:Function]
|
||||
# @PURPOSE: Saves the current configuration state to DB.
|
||||
# @PRE: self.config is set.
|
||||
# @POST: self._save_config_to_db called.
|
||||
def save(self):
|
||||
with belief_scope("save"):
|
||||
self._save_config_to_db(self.config)
|
||||
# [/DEF:save:Function]
|
||||
|
||||
# [DEF:get_config:Function]
|
||||
# @PURPOSE: Returns the current configuration.
|
||||
# @RETURN: AppConfig - The current configuration.
|
||||
def get_config(self) -> AppConfig:
|
||||
with belief_scope("get_config"):
|
||||
return self.config
|
||||
# [/DEF:get_config:Function]
|
||||
|
||||
# [DEF:update_global_settings:Function]
|
||||
# @PURPOSE: Updates the global settings and persists the change.
|
||||
# @PRE: isinstance(settings, GlobalSettings)
|
||||
# @POST: self.config.settings updated and saved.
|
||||
# @PARAM: settings (GlobalSettings) - The new global settings.
|
||||
def update_global_settings(self, settings: GlobalSettings):
|
||||
with belief_scope("update_global_settings"):
|
||||
logger.info("[update_global_settings][Entry] Updating settings")
|
||||
|
||||
assert isinstance(settings, GlobalSettings), "settings must be an instance of GlobalSettings"
|
||||
self.config.settings = settings
|
||||
self.save()
|
||||
configure_logger(settings.logging)
|
||||
logger.info("[update_global_settings][Exit] Settings updated")
|
||||
# [/DEF:update_global_settings:Function]
|
||||
|
||||
# [DEF:validate_path:Function]
|
||||
# @PURPOSE: Validates if a path exists and is writable.
|
||||
# @PARAM: path (str) - The path to validate.
|
||||
# @RETURN: tuple (bool, str) - (is_valid, message)
|
||||
def validate_path(self, path: str) -> tuple[bool, str]:
|
||||
with belief_scope("validate_path"):
|
||||
p = os.path.abspath(path)
|
||||
if not os.path.exists(p):
|
||||
try:
|
||||
os.makedirs(p, exist_ok=True)
|
||||
except Exception as e:
|
||||
return False, f"Path does not exist and could not be created: {e}"
|
||||
|
||||
if not os.access(p, os.W_OK):
|
||||
return False, "Path is not writable"
|
||||
|
||||
return True, "Path is valid and writable"
|
||||
# [/DEF:validate_path:Function]
|
||||
|
||||
# [DEF:get_environments:Function]
|
||||
# @PURPOSE: Returns the list of configured environments.
|
||||
# @RETURN: List[Environment] - List of environments.
|
||||
def get_environments(self) -> List[Environment]:
|
||||
with belief_scope("get_environments"):
|
||||
return self.config.environments
|
||||
# [/DEF:get_environments:Function]
|
||||
|
||||
# [DEF:has_environments:Function]
|
||||
# @PURPOSE: Checks if at least one environment is configured.
|
||||
# @RETURN: bool - True if at least one environment exists.
|
||||
def has_environments(self) -> bool:
|
||||
with belief_scope("has_environments"):
|
||||
return len(self.config.environments) > 0
|
||||
# [/DEF:has_environments:Function]
|
||||
|
||||
# [DEF:get_environment:Function]
|
||||
# @PURPOSE: Returns a single environment by ID.
|
||||
# @PARAM: env_id (str) - The ID of the environment to retrieve.
|
||||
# @RETURN: Optional[Environment] - The environment with the given ID, or None.
|
||||
def get_environment(self, env_id: str) -> Optional[Environment]:
|
||||
with belief_scope("get_environment"):
|
||||
for env in self.config.environments:
|
||||
if env.id == env_id:
|
||||
return env
|
||||
return None
|
||||
# [/DEF:get_environment:Function]
|
||||
|
||||
# [DEF:add_environment:Function]
|
||||
# @PURPOSE: Adds a new environment to the configuration.
|
||||
# @PARAM: env (Environment) - The environment to add.
|
||||
def add_environment(self, env: Environment):
|
||||
with belief_scope("add_environment"):
|
||||
logger.info(f"[add_environment][Entry] Adding environment {env.id}")
|
||||
assert isinstance(env, Environment), "env must be an instance of Environment"
|
||||
|
||||
self.config.environments = [e for e in self.config.environments if e.id != env.id]
|
||||
self.config.environments.append(env)
|
||||
self.save()
|
||||
logger.info("[add_environment][Exit] Environment added")
|
||||
# [/DEF:add_environment:Function]
|
||||
|
||||
# [DEF:update_environment:Function]
|
||||
# @PURPOSE: Updates an existing environment.
|
||||
# @PARAM: env_id (str) - The ID of the environment to update.
|
||||
# @PARAM: updated_env (Environment) - The updated environment data.
|
||||
# @RETURN: bool - True if updated, False otherwise.
|
||||
def update_environment(self, env_id: str, updated_env: Environment) -> bool:
|
||||
with belief_scope("update_environment"):
|
||||
logger.info(f"[update_environment][Entry] Updating {env_id}")
|
||||
assert env_id and isinstance(env_id, str), "env_id must be a non-empty string"
|
||||
assert isinstance(updated_env, Environment), "updated_env must be an instance of Environment"
|
||||
|
||||
for i, env in enumerate(self.config.environments):
|
||||
if env.id == env_id:
|
||||
if updated_env.password == "********":
|
||||
updated_env.password = env.password
|
||||
|
||||
self.config.environments[i] = updated_env
|
||||
self.save()
|
||||
logger.info(f"[update_environment][Coherence:OK] Updated {env_id}")
|
||||
return True
|
||||
|
||||
logger.warning(f"[update_environment][Coherence:Failed] Environment {env_id} not found")
|
||||
return False
|
||||
# [/DEF:update_environment:Function]
|
||||
|
||||
# [DEF:delete_environment:Function]
|
||||
# @PURPOSE: Deletes an environment by ID.
|
||||
# @PARAM: env_id (str) - The ID of the environment to delete.
|
||||
def delete_environment(self, env_id: str):
|
||||
with belief_scope("delete_environment"):
|
||||
logger.info(f"[delete_environment][Entry] Deleting {env_id}")
|
||||
assert env_id and isinstance(env_id, str), "env_id must be a non-empty string"
|
||||
|
||||
original_count = len(self.config.environments)
|
||||
self.config.environments = [e for e in self.config.environments if e.id != env_id]
|
||||
|
||||
if len(self.config.environments) < original_count:
|
||||
self.save()
|
||||
logger.info(f"[delete_environment][Action] Deleted {env_id}")
|
||||
else:
|
||||
logger.warning(f"[delete_environment][Coherence:Failed] Environment {env_id} not found")
|
||||
# [/DEF:delete_environment:Function]
|
||||
|
||||
|
||||
# [/DEF:ConfigManager:Class]
|
||||
# [/DEF:ConfigManagerModule:Module]
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
# @SEMANTICS: config, models, pydantic
|
||||
# @PURPOSE: Defines the data models for application configuration using Pydantic.
|
||||
# @LAYER: Core
|
||||
# @RELATION: READS_FROM -> config.json
|
||||
# @RELATION: READS_FROM -> app_configurations (database)
|
||||
# @RELATION: USED_BY -> ConfigManager
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
@@ -33,10 +33,10 @@ class Environment(BaseModel):
|
||||
|
||||
# [DEF:LoggingConfig:DataClass]
|
||||
# @PURPOSE: Defines the configuration for the application's logging system.
|
||||
class LoggingConfig(BaseModel):
|
||||
level: str = "INFO"
|
||||
task_log_level: str = "INFO" # Minimum level for task-specific logs (DEBUG, INFO, WARNING, ERROR)
|
||||
file_path: Optional[str] = "logs/app.log"
|
||||
class LoggingConfig(BaseModel):
|
||||
level: str = "INFO"
|
||||
task_log_level: str = "INFO" # Minimum level for task-specific logs (DEBUG, INFO, WARNING, ERROR)
|
||||
file_path: Optional[str] = None
|
||||
max_bytes: int = 10 * 1024 * 1024
|
||||
backup_count: int = 5
|
||||
enable_belief_state: bool = True
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# [DEF:backend.src.core.database:Module]
|
||||
#
|
||||
# @SEMANTICS: database, sqlite, sqlalchemy, session, persistence
|
||||
# @PURPOSE: Configures the SQLite database connection and session management.
|
||||
# @SEMANTICS: database, postgresql, sqlalchemy, session, persistence
|
||||
# @PURPOSE: Configures database connection and session management (PostgreSQL-first).
|
||||
# @LAYER: Core
|
||||
# @RELATION: DEPENDS_ON -> sqlalchemy
|
||||
# @RELATION: USES -> backend.src.models.mapping
|
||||
@@ -14,6 +14,9 @@ from sqlalchemy import create_engine
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
from ..models.mapping import Base
|
||||
# Import models to ensure they're registered with Base
|
||||
from ..models import task as _task_models # noqa: F401
|
||||
from ..models import auth as _auth_models # noqa: F401
|
||||
from ..models import config as _config_models # noqa: F401
|
||||
from .logger import belief_scope
|
||||
from .auth.config import auth_config
|
||||
import os
|
||||
@@ -21,44 +24,50 @@ from pathlib import Path
|
||||
# [/SECTION]
|
||||
|
||||
# [DEF:BASE_DIR:Variable]
|
||||
# @PURPOSE: Base directory for the backend (where .db files should reside).
|
||||
# @PURPOSE: Base directory for the backend.
|
||||
BASE_DIR = Path(__file__).resolve().parent.parent.parent
|
||||
# [/DEF:BASE_DIR:Variable]
|
||||
|
||||
# [DEF:DATABASE_URL:Constant]
|
||||
# @PURPOSE: URL for the main mappings database.
|
||||
DATABASE_URL = os.getenv("DATABASE_URL", f"sqlite:///{BASE_DIR}/mappings.db")
|
||||
# @PURPOSE: URL for the main application database.
|
||||
DEFAULT_POSTGRES_URL = os.getenv(
|
||||
"POSTGRES_URL",
|
||||
"postgresql+psycopg2://postgres:postgres@localhost:5432/ss_tools",
|
||||
)
|
||||
DATABASE_URL = os.getenv("DATABASE_URL", DEFAULT_POSTGRES_URL)
|
||||
# [/DEF:DATABASE_URL:Constant]
|
||||
|
||||
# [DEF:TASKS_DATABASE_URL:Constant]
|
||||
# @PURPOSE: URL for the tasks execution database.
|
||||
TASKS_DATABASE_URL = os.getenv("TASKS_DATABASE_URL", f"sqlite:///{BASE_DIR}/tasks.db")
|
||||
# Defaults to DATABASE_URL to keep task logs in the same PostgreSQL instance.
|
||||
TASKS_DATABASE_URL = os.getenv("TASKS_DATABASE_URL", DATABASE_URL)
|
||||
# [/DEF:TASKS_DATABASE_URL:Constant]
|
||||
|
||||
# [DEF:AUTH_DATABASE_URL:Constant]
|
||||
# @PURPOSE: URL for the authentication database.
|
||||
AUTH_DATABASE_URL = os.getenv("AUTH_DATABASE_URL", auth_config.AUTH_DATABASE_URL)
|
||||
# If it's a relative sqlite path starting with ./backend/, fix it to be absolute or relative to BASE_DIR
|
||||
if AUTH_DATABASE_URL.startswith("sqlite:///./backend/"):
|
||||
AUTH_DATABASE_URL = AUTH_DATABASE_URL.replace("sqlite:///./backend/", f"sqlite:///{BASE_DIR}/")
|
||||
elif AUTH_DATABASE_URL.startswith("sqlite:///./") and not AUTH_DATABASE_URL.startswith("sqlite:///./backend/"):
|
||||
# If it's just ./ but we are in backend, it's fine, but let's make it absolute for robustness
|
||||
AUTH_DATABASE_URL = AUTH_DATABASE_URL.replace("sqlite:///./", f"sqlite:///{BASE_DIR}/")
|
||||
# [/DEF:AUTH_DATABASE_URL:Constant]
|
||||
|
||||
# [DEF:engine:Variable]
|
||||
def _build_engine(db_url: str):
|
||||
with belief_scope("_build_engine"):
|
||||
if db_url.startswith("sqlite"):
|
||||
return create_engine(db_url, connect_args={"check_same_thread": False})
|
||||
return create_engine(db_url, pool_pre_ping=True)
|
||||
|
||||
|
||||
# @PURPOSE: SQLAlchemy engine for mappings database.
|
||||
engine = create_engine(DATABASE_URL, connect_args={"check_same_thread": False})
|
||||
engine = _build_engine(DATABASE_URL)
|
||||
# [/DEF:engine:Variable]
|
||||
|
||||
# [DEF:tasks_engine:Variable]
|
||||
# @PURPOSE: SQLAlchemy engine for tasks database.
|
||||
tasks_engine = create_engine(TASKS_DATABASE_URL, connect_args={"check_same_thread": False})
|
||||
tasks_engine = _build_engine(TASKS_DATABASE_URL)
|
||||
# [/DEF:tasks_engine:Variable]
|
||||
|
||||
# [DEF:auth_engine:Variable]
|
||||
# @PURPOSE: SQLAlchemy engine for authentication database.
|
||||
auth_engine = create_engine(AUTH_DATABASE_URL, connect_args={"check_same_thread": False})
|
||||
auth_engine = _build_engine(AUTH_DATABASE_URL)
|
||||
# [/DEF:auth_engine:Variable]
|
||||
|
||||
# [DEF:SessionLocal:Class]
|
||||
|
||||
@@ -20,14 +20,14 @@ from .core.auth.jwt import decode_token
|
||||
from .core.auth.repository import AuthRepository
|
||||
from .models.auth import User
|
||||
|
||||
# Initialize singletons
|
||||
# Use absolute path relative to this file to ensure plugins are found regardless of CWD
|
||||
project_root = Path(__file__).parent.parent.parent
|
||||
config_path = project_root / "config.json"
|
||||
config_manager = ConfigManager(config_path=str(config_path))
|
||||
|
||||
# Initialize database before any other services that might use it
|
||||
init_db()
|
||||
# Initialize singletons
|
||||
# Use absolute path relative to this file to ensure plugins are found regardless of CWD
|
||||
project_root = Path(__file__).parent.parent.parent
|
||||
config_path = project_root / "config.json"
|
||||
|
||||
# Initialize database before services that use persisted configuration.
|
||||
init_db()
|
||||
config_manager = ConfigManager(config_path=str(config_path))
|
||||
|
||||
# [DEF:get_config_manager:Function]
|
||||
# @PURPOSE: Dependency injector for ConfigManager.
|
||||
|
||||
26
backend/src/models/config.py
Normal file
26
backend/src/models/config.py
Normal file
@@ -0,0 +1,26 @@
|
||||
# [DEF:backend.src.models.config:Module]
|
||||
#
|
||||
# @TIER: STANDARD
|
||||
# @SEMANTICS: database, config, settings, sqlalchemy
|
||||
# @PURPOSE: Defines database schema for persisted application configuration.
|
||||
# @LAYER: Domain
|
||||
# @RELATION: DEPENDS_ON -> sqlalchemy
|
||||
|
||||
from sqlalchemy import Column, String, DateTime, JSON
|
||||
from sqlalchemy.sql import func
|
||||
|
||||
from .mapping import Base
|
||||
|
||||
|
||||
# [DEF:AppConfigRecord:Class]
|
||||
# @PURPOSE: Stores the single source of truth for application configuration.
|
||||
class AppConfigRecord(Base):
|
||||
__tablename__ = "app_configurations"
|
||||
|
||||
id = Column(String, primary_key=True)
|
||||
payload = Column(JSON, nullable=False)
|
||||
updated_at = Column(DateTime(timezone=True), server_default=func.now(), onupdate=func.now())
|
||||
|
||||
|
||||
# [/DEF:AppConfigRecord:Class]
|
||||
# [/DEF:backend.src.models.config:Module]
|
||||
@@ -22,6 +22,8 @@ class FileCategory(str, Enum):
|
||||
# @PURPOSE: Configuration model for the storage system, defining paths and naming patterns.
|
||||
class StorageConfig(BaseModel):
|
||||
root_path: str = Field(default="backups", description="Absolute path to the storage root directory.")
|
||||
backup_path: str = Field(default="backups", description="Subpath for backups.")
|
||||
repo_path: str = Field(default="repositorys", description="Subpath for repositories.")
|
||||
backup_structure_pattern: str = Field(default="{category}/", description="Pattern for backup directory structure.")
|
||||
repo_structure_pattern: str = Field(default="{category}/", description="Pattern for repository directory structure.")
|
||||
filename_pattern: str = Field(default="{name}_{timestamp}", description="Pattern for filenames.")
|
||||
|
||||
350
backend/src/scripts/migrate_sqlite_to_postgres.py
Normal file
350
backend/src/scripts/migrate_sqlite_to_postgres.py
Normal file
@@ -0,0 +1,350 @@
|
||||
# [DEF:backend.src.scripts.migrate_sqlite_to_postgres:Module]
|
||||
#
|
||||
# @SEMANTICS: migration, sqlite, postgresql, config, task_logs, task_records
|
||||
# @PURPOSE: Migrates legacy config and task history from SQLite/file storage to PostgreSQL.
|
||||
# @LAYER: Scripts
|
||||
# @RELATION: READS_FROM -> backend/tasks.db
|
||||
# @RELATION: READS_FROM -> backend/config.json
|
||||
# @RELATION: WRITES_TO -> postgresql.task_records
|
||||
# @RELATION: WRITES_TO -> postgresql.task_logs
|
||||
# @RELATION: WRITES_TO -> postgresql.app_configurations
|
||||
#
|
||||
# @INVARIANT: Script is idempotent for task_records and app_configurations.
|
||||
|
||||
# [SECTION: IMPORTS]
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sqlite3
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, Iterable, Optional
|
||||
|
||||
from sqlalchemy import create_engine, text
|
||||
from sqlalchemy.exc import SQLAlchemyError
|
||||
|
||||
from src.core.logger import belief_scope, logger
|
||||
# [/SECTION]
|
||||
|
||||
|
||||
# [DEF:Constants:Section]
|
||||
DEFAULT_TARGET_URL = os.getenv(
|
||||
"DATABASE_URL",
|
||||
os.getenv("POSTGRES_URL", "postgresql+psycopg2://postgres:postgres@localhost:5432/ss_tools"),
|
||||
)
|
||||
# [/DEF:Constants:Section]
|
||||
|
||||
|
||||
# [DEF:_json_load_if_needed:Function]
|
||||
# @PURPOSE: Parses JSON-like values from SQLite TEXT/JSON columns to Python objects.
|
||||
def _json_load_if_needed(value: Any) -> Any:
|
||||
with belief_scope("_json_load_if_needed"):
|
||||
if value is None:
|
||||
return None
|
||||
if isinstance(value, (dict, list)):
|
||||
return value
|
||||
if isinstance(value, str):
|
||||
raw = value.strip()
|
||||
if not raw:
|
||||
return None
|
||||
if raw[0] in "{[":
|
||||
try:
|
||||
return json.loads(raw)
|
||||
except json.JSONDecodeError:
|
||||
return value
|
||||
return value
|
||||
|
||||
|
||||
# [DEF:_find_legacy_config_path:Function]
|
||||
# @PURPOSE: Resolves the existing legacy config.json path from candidates.
|
||||
def _find_legacy_config_path(explicit_path: Optional[str]) -> Optional[Path]:
|
||||
with belief_scope("_find_legacy_config_path"):
|
||||
if explicit_path:
|
||||
p = Path(explicit_path)
|
||||
return p if p.exists() else None
|
||||
|
||||
candidates = [
|
||||
Path("backend/config.json"),
|
||||
Path("config.json"),
|
||||
]
|
||||
for candidate in candidates:
|
||||
if candidate.exists():
|
||||
return candidate
|
||||
return None
|
||||
|
||||
|
||||
# [DEF:_connect_sqlite:Function]
|
||||
# @PURPOSE: Opens a SQLite connection with row factory.
|
||||
def _connect_sqlite(path: Path) -> sqlite3.Connection:
|
||||
with belief_scope("_connect_sqlite"):
|
||||
conn = sqlite3.connect(str(path))
|
||||
conn.row_factory = sqlite3.Row
|
||||
return conn
|
||||
|
||||
|
||||
# [DEF:_ensure_target_schema:Function]
|
||||
# @PURPOSE: Ensures required PostgreSQL tables exist before migration.
|
||||
def _ensure_target_schema(engine) -> None:
|
||||
with belief_scope("_ensure_target_schema"):
|
||||
stmts: Iterable[str] = (
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS app_configurations (
|
||||
id TEXT PRIMARY KEY,
|
||||
payload JSONB NOT NULL,
|
||||
updated_at TIMESTAMPTZ DEFAULT NOW()
|
||||
)
|
||||
""",
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS task_records (
|
||||
id TEXT PRIMARY KEY,
|
||||
type TEXT NOT NULL,
|
||||
status TEXT NOT NULL,
|
||||
environment_id TEXT NULL,
|
||||
started_at TIMESTAMPTZ NULL,
|
||||
finished_at TIMESTAMPTZ NULL,
|
||||
logs JSONB NULL,
|
||||
error TEXT NULL,
|
||||
result JSONB NULL,
|
||||
created_at TIMESTAMPTZ DEFAULT NOW(),
|
||||
params JSONB NULL
|
||||
)
|
||||
""",
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS task_logs (
|
||||
id INTEGER PRIMARY KEY,
|
||||
task_id TEXT NOT NULL,
|
||||
timestamp TIMESTAMPTZ NOT NULL,
|
||||
level VARCHAR(16) NOT NULL,
|
||||
source VARCHAR(64) NOT NULL DEFAULT 'system',
|
||||
message TEXT NOT NULL,
|
||||
metadata_json TEXT NULL,
|
||||
CONSTRAINT fk_task_logs_task
|
||||
FOREIGN KEY(task_id)
|
||||
REFERENCES task_records(id)
|
||||
ON DELETE CASCADE
|
||||
)
|
||||
""",
|
||||
"CREATE INDEX IF NOT EXISTS ix_task_logs_task_timestamp ON task_logs (task_id, timestamp)",
|
||||
"CREATE INDEX IF NOT EXISTS ix_task_logs_task_level ON task_logs (task_id, level)",
|
||||
"CREATE INDEX IF NOT EXISTS ix_task_logs_task_source ON task_logs (task_id, source)",
|
||||
"""
|
||||
DO $$
|
||||
BEGIN
|
||||
IF EXISTS (
|
||||
SELECT 1 FROM pg_class WHERE relkind = 'S' AND relname = 'task_logs_id_seq'
|
||||
) THEN
|
||||
PERFORM 1;
|
||||
ELSE
|
||||
CREATE SEQUENCE task_logs_id_seq OWNED BY task_logs.id;
|
||||
END IF;
|
||||
END $$;
|
||||
""",
|
||||
"ALTER TABLE task_logs ALTER COLUMN id SET DEFAULT nextval('task_logs_id_seq')",
|
||||
)
|
||||
with engine.begin() as conn:
|
||||
for stmt in stmts:
|
||||
conn.execute(text(stmt))
|
||||
|
||||
|
||||
# [DEF:_migrate_config:Function]
|
||||
# @PURPOSE: Migrates legacy config.json into app_configurations(global).
|
||||
def _migrate_config(engine, legacy_config_path: Optional[Path]) -> int:
|
||||
with belief_scope("_migrate_config"):
|
||||
if legacy_config_path is None:
|
||||
logger.info("[_migrate_config][Action] No legacy config.json found, skipping")
|
||||
return 0
|
||||
|
||||
payload = json.loads(legacy_config_path.read_text(encoding="utf-8"))
|
||||
with engine.begin() as conn:
|
||||
conn.execute(
|
||||
text(
|
||||
"""
|
||||
INSERT INTO app_configurations (id, payload, updated_at)
|
||||
VALUES ('global', CAST(:payload AS JSONB), NOW())
|
||||
ON CONFLICT (id)
|
||||
DO UPDATE SET payload = EXCLUDED.payload, updated_at = NOW()
|
||||
"""
|
||||
),
|
||||
{"payload": json.dumps(payload, ensure_ascii=True)},
|
||||
)
|
||||
logger.info("[_migrate_config][Coherence:OK] Config migrated from %s", legacy_config_path)
|
||||
return 1
|
||||
|
||||
|
||||
# [DEF:_migrate_tasks_and_logs:Function]
|
||||
# @PURPOSE: Migrates task_records and task_logs from SQLite into PostgreSQL.
|
||||
def _migrate_tasks_and_logs(engine, sqlite_conn: sqlite3.Connection) -> Dict[str, int]:
|
||||
with belief_scope("_migrate_tasks_and_logs"):
|
||||
stats = {"task_records_total": 0, "task_records_inserted": 0, "task_logs_total": 0, "task_logs_inserted": 0}
|
||||
|
||||
rows = sqlite_conn.execute(
|
||||
"""
|
||||
SELECT id, type, status, environment_id, started_at, finished_at, logs, error, result, created_at, params
|
||||
FROM task_records
|
||||
ORDER BY created_at ASC
|
||||
"""
|
||||
).fetchall()
|
||||
stats["task_records_total"] = len(rows)
|
||||
|
||||
with engine.begin() as conn:
|
||||
existing_env_ids = {
|
||||
row[0]
|
||||
for row in conn.execute(text("SELECT id FROM environments")).fetchall()
|
||||
}
|
||||
for row in rows:
|
||||
params_obj = _json_load_if_needed(row["params"])
|
||||
result_obj = _json_load_if_needed(row["result"])
|
||||
logs_obj = _json_load_if_needed(row["logs"])
|
||||
environment_id = row["environment_id"]
|
||||
if environment_id and environment_id not in existing_env_ids:
|
||||
# Legacy task may reference environments that were not migrated; keep task row and drop FK value.
|
||||
environment_id = None
|
||||
|
||||
res = conn.execute(
|
||||
text(
|
||||
"""
|
||||
INSERT INTO task_records (
|
||||
id, type, status, environment_id, started_at, finished_at,
|
||||
logs, error, result, created_at, params
|
||||
) VALUES (
|
||||
:id, :type, :status, :environment_id, :started_at, :finished_at,
|
||||
CAST(:logs AS JSONB), :error, CAST(:result AS JSONB), :created_at, CAST(:params AS JSONB)
|
||||
)
|
||||
ON CONFLICT (id) DO NOTHING
|
||||
"""
|
||||
),
|
||||
{
|
||||
"id": row["id"],
|
||||
"type": row["type"],
|
||||
"status": row["status"],
|
||||
"environment_id": environment_id,
|
||||
"started_at": row["started_at"],
|
||||
"finished_at": row["finished_at"],
|
||||
"logs": json.dumps(logs_obj, ensure_ascii=True) if logs_obj is not None else None,
|
||||
"error": row["error"],
|
||||
"result": json.dumps(result_obj, ensure_ascii=True) if result_obj is not None else None,
|
||||
"created_at": row["created_at"],
|
||||
"params": json.dumps(params_obj, ensure_ascii=True) if params_obj is not None else None,
|
||||
},
|
||||
)
|
||||
if res.rowcount and res.rowcount > 0:
|
||||
stats["task_records_inserted"] += int(res.rowcount)
|
||||
|
||||
log_rows = sqlite_conn.execute(
|
||||
"""
|
||||
SELECT id, task_id, timestamp, level, source, message, metadata_json
|
||||
FROM task_logs
|
||||
ORDER BY id ASC
|
||||
"""
|
||||
).fetchall()
|
||||
stats["task_logs_total"] = len(log_rows)
|
||||
|
||||
with engine.begin() as conn:
|
||||
for row in log_rows:
|
||||
# Preserve original IDs to keep migration idempotent.
|
||||
res = conn.execute(
|
||||
text(
|
||||
"""
|
||||
INSERT INTO task_logs (id, task_id, timestamp, level, source, message, metadata_json)
|
||||
VALUES (:id, :task_id, :timestamp, :level, :source, :message, :metadata_json)
|
||||
ON CONFLICT (id) DO NOTHING
|
||||
"""
|
||||
),
|
||||
{
|
||||
"id": row["id"],
|
||||
"task_id": row["task_id"],
|
||||
"timestamp": row["timestamp"],
|
||||
"level": row["level"],
|
||||
"source": row["source"] or "system",
|
||||
"message": row["message"],
|
||||
"metadata_json": row["metadata_json"],
|
||||
},
|
||||
)
|
||||
if res.rowcount and res.rowcount > 0:
|
||||
stats["task_logs_inserted"] += int(res.rowcount)
|
||||
|
||||
# Ensure sequence is aligned after explicit id inserts.
|
||||
conn.execute(
|
||||
text(
|
||||
"""
|
||||
SELECT setval(
|
||||
'task_logs_id_seq',
|
||||
COALESCE((SELECT MAX(id) FROM task_logs), 1),
|
||||
TRUE
|
||||
)
|
||||
"""
|
||||
)
|
||||
)
|
||||
|
||||
logger.info(
|
||||
"[_migrate_tasks_and_logs][Coherence:OK] task_records=%s/%s task_logs=%s/%s",
|
||||
stats["task_records_inserted"],
|
||||
stats["task_records_total"],
|
||||
stats["task_logs_inserted"],
|
||||
stats["task_logs_total"],
|
||||
)
|
||||
return stats
|
||||
|
||||
|
||||
# [DEF:run_migration:Function]
|
||||
# @PURPOSE: Orchestrates migration from SQLite/file to PostgreSQL.
|
||||
def run_migration(sqlite_path: Path, target_url: str, legacy_config_path: Optional[Path]) -> Dict[str, int]:
|
||||
with belief_scope("run_migration"):
|
||||
logger.info("[run_migration][Entry] sqlite=%s target=%s", sqlite_path, target_url)
|
||||
if not sqlite_path.exists():
|
||||
raise FileNotFoundError(f"SQLite source not found: {sqlite_path}")
|
||||
|
||||
sqlite_conn = _connect_sqlite(sqlite_path)
|
||||
engine = create_engine(target_url, pool_pre_ping=True)
|
||||
try:
|
||||
_ensure_target_schema(engine)
|
||||
config_upserted = _migrate_config(engine, legacy_config_path)
|
||||
stats = _migrate_tasks_and_logs(engine, sqlite_conn)
|
||||
stats["config_upserted"] = config_upserted
|
||||
return stats
|
||||
finally:
|
||||
sqlite_conn.close()
|
||||
|
||||
|
||||
# [DEF:main:Function]
|
||||
# @PURPOSE: CLI entrypoint.
|
||||
def main() -> int:
|
||||
with belief_scope("main"):
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Migrate legacy config.json and task logs from SQLite to PostgreSQL.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--sqlite-path",
|
||||
default="backend/tasks.db",
|
||||
help="Path to source SQLite DB with task_records/task_logs (default: backend/tasks.db).",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--target-url",
|
||||
default=DEFAULT_TARGET_URL,
|
||||
help="Target PostgreSQL SQLAlchemy URL (default: DATABASE_URL/POSTGRES_URL env).",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--config-path",
|
||||
default=None,
|
||||
help="Optional path to legacy config.json (auto-detected when omitted).",
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
sqlite_path = Path(args.sqlite_path)
|
||||
legacy_config_path = _find_legacy_config_path(args.config_path)
|
||||
try:
|
||||
stats = run_migration(sqlite_path=sqlite_path, target_url=args.target_url, legacy_config_path=legacy_config_path)
|
||||
print("Migration completed.")
|
||||
print(json.dumps(stats, indent=2))
|
||||
return 0
|
||||
except (SQLAlchemyError, OSError, sqlite3.Error, ValueError) as e:
|
||||
logger.error("[main][Coherence:Failed] Migration failed: %s", e)
|
||||
print(f"Migration failed: {e}")
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
# [/DEF:main:Function]
|
||||
|
||||
# [/DEF:backend.src.scripts.migrate_sqlite_to_postgres:Module]
|
||||
42
docker-compose.yml
Normal file
42
docker-compose.yml
Normal file
@@ -0,0 +1,42 @@
|
||||
services:
|
||||
db:
|
||||
image: ${POSTGRES_IMAGE:-postgres:16-alpine}
|
||||
container_name: ss_tools_db
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
POSTGRES_DB: ss_tools
|
||||
POSTGRES_USER: postgres
|
||||
POSTGRES_PASSWORD: postgres
|
||||
ports:
|
||||
- "${POSTGRES_HOST_PORT:-5432}:5432"
|
||||
volumes:
|
||||
- postgres_data:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U postgres -d ss_tools"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 10
|
||||
|
||||
app:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile
|
||||
container_name: ss_tools_app
|
||||
restart: unless-stopped
|
||||
depends_on:
|
||||
db:
|
||||
condition: service_healthy
|
||||
environment:
|
||||
POSTGRES_URL: postgresql+psycopg2://postgres:postgres@db:5432/ss_tools
|
||||
DATABASE_URL: postgresql+psycopg2://postgres:postgres@db:5432/ss_tools
|
||||
TASKS_DATABASE_URL: postgresql+psycopg2://postgres:postgres@db:5432/ss_tools
|
||||
AUTH_DATABASE_URL: postgresql+psycopg2://postgres:postgres@db:5432/ss_tools
|
||||
BACKEND_PORT: 8000
|
||||
ports:
|
||||
- "8000:8000"
|
||||
volumes:
|
||||
- ./backups:/app/backups
|
||||
- ./backend/git_repos:/app/backend/git_repos
|
||||
|
||||
volumes:
|
||||
postgres_data:
|
||||
@@ -1,13 +1,14 @@
|
||||
# [DEF:generate_semantic_map:Module]
|
||||
#
|
||||
# @TIER: CRITICAL
|
||||
# @SEMANTICS: semantic_analysis, parser, map_generator, compliance_checker, tier_validation, svelte_props, data_flow
|
||||
# @PURPOSE: Scans the codebase to generate a Semantic Map and Compliance Report based on the System Standard.
|
||||
# @SEMANTICS: semantic_analysis, parser, map_generator, compliance_checker, tier_validation, svelte_props, data_flow, module_map
|
||||
# @PURPOSE: Scans the codebase to generate a Semantic Map, Module Map, and Compliance Report based on the System Standard.
|
||||
# @LAYER: DevOps/Tooling
|
||||
# @INVARIANT: All DEF anchors must have matching closing anchors; TIER determines validation strictness.
|
||||
# @RELATION: READS -> FileSystem
|
||||
# @RELATION: PRODUCES -> semantics/semantic_map.json
|
||||
# @RELATION: PRODUCES -> .ai/PROJECT_MAP.md
|
||||
# @RELATION: PRODUCES -> .ai/MODULE_MAP.md
|
||||
# @RELATION: PRODUCES -> semantics/reports/semantic_report_*.md
|
||||
|
||||
# [SECTION: IMPORTS]
|
||||
@@ -83,6 +84,7 @@ IGNORE_FILES = {
|
||||
}
|
||||
OUTPUT_JSON = "semantics/semantic_map.json"
|
||||
OUTPUT_COMPRESSED_MD = ".ai/PROJECT_MAP.md"
|
||||
OUTPUT_MODULE_MAP_MD = ".ai/MODULE_MAP.md"
|
||||
REPORTS_DIR = "semantics/reports"
|
||||
|
||||
# Tier-based mandatory tags
|
||||
@@ -830,6 +832,7 @@ class SemanticMapGenerator:
|
||||
|
||||
self._generate_report()
|
||||
self._generate_compressed_map()
|
||||
self._generate_module_map()
|
||||
# [/DEF:_generate_artifacts:Function]
|
||||
|
||||
# [DEF:_generate_report:Function]
|
||||
@@ -990,6 +993,163 @@ class SemanticMapGenerator:
|
||||
self._write_entity_md(f, child, level + 1)
|
||||
# [/DEF:_write_entity_md:Function]
|
||||
|
||||
# [DEF:_generate_module_map:Function]
|
||||
# @TIER: CRITICAL
|
||||
# @PURPOSE: Generates a module-centric map grouping entities by directory structure.
|
||||
# @PRE: Entities have been processed.
|
||||
# @POST: Markdown module map is written to .ai/MODULE_MAP.md.
|
||||
def _generate_module_map(self):
|
||||
with belief_scope("_generate_module_map"):
|
||||
os.makedirs(os.path.dirname(OUTPUT_MODULE_MAP_MD), exist_ok=True)
|
||||
|
||||
# Group entities by directory/module
|
||||
modules: Dict[str, Dict[str, Any]] = {}
|
||||
|
||||
# [DEF:_get_module_path:Function]
|
||||
# @TIER: STANDARD
|
||||
# @PURPOSE: Extracts the module path from a file path.
|
||||
# @PRE: file_path is a valid relative path.
|
||||
# @POST: Returns a module path string.
|
||||
def _get_module_path(file_path: str) -> str:
|
||||
# Convert file path to module-like path
|
||||
parts = file_path.replace(os.sep, '/').split('/')
|
||||
# Remove filename
|
||||
if len(parts) > 1:
|
||||
return '/'.join(parts[:-1])
|
||||
return 'root'
|
||||
# [/DEF:_get_module_path:Function]
|
||||
|
||||
# [DEF:_collect_all_entities:Function]
|
||||
# @TIER: STANDARD
|
||||
# @PURPOSE: Flattens entity tree for easier grouping.
|
||||
# @PRE: entity list is valid.
|
||||
# @POST: Returns flat list of all entities with their hierarchy.
|
||||
def _collect_all_entities(entities: List[SemanticEntity], result: List[Tuple[str, SemanticEntity]]):
|
||||
for e in entities:
|
||||
result.append((_get_module_path(e.file_path), e))
|
||||
_collect_all_entities(e.children, result)
|
||||
# [/DEF:_collect_all_entities:Function]
|
||||
|
||||
# Collect all entities
|
||||
all_entities: List[Tuple[str, SemanticEntity]] = []
|
||||
_collect_all_entities(self.entities, all_entities)
|
||||
|
||||
# Group by module path
|
||||
for module_path, entity in all_entities:
|
||||
if module_path not in modules:
|
||||
modules[module_path] = {
|
||||
'entities': [],
|
||||
'files': set(),
|
||||
'layers': set(),
|
||||
'tiers': {'CRITICAL': 0, 'STANDARD': 0, 'TRIVIAL': 0},
|
||||
'relations': []
|
||||
}
|
||||
modules[module_path]['entities'].append(entity)
|
||||
modules[module_path]['files'].add(entity.file_path)
|
||||
if entity.tags.get('LAYER'):
|
||||
modules[module_path]['layers'].add(entity.tags.get('LAYER'))
|
||||
tier = entity.get_tier().value
|
||||
modules[module_path]['tiers'][tier] = modules[module_path]['tiers'].get(tier, 0) + 1
|
||||
for rel in entity.relations:
|
||||
modules[module_path]['relations'].append(rel)
|
||||
|
||||
# Write module map
|
||||
with open(OUTPUT_MODULE_MAP_MD, 'w', encoding='utf-8') as f:
|
||||
f.write("# Module Map\n\n")
|
||||
f.write("> High-level module structure for AI Context. Generated automatically.\n\n")
|
||||
f.write(f"**Generated:** {datetime.datetime.now().isoformat()}\n\n")
|
||||
|
||||
# Summary statistics
|
||||
total_modules = len(modules)
|
||||
total_entities = len(all_entities)
|
||||
f.write("## Summary\n\n")
|
||||
f.write(f"- **Total Modules:** {total_modules}\n")
|
||||
f.write(f"- **Total Entities:** {total_entities}\n\n")
|
||||
|
||||
# Module hierarchy
|
||||
f.write("## Module Hierarchy\n\n")
|
||||
|
||||
# Sort modules by path for consistent output
|
||||
sorted_modules = sorted(modules.items(), key=lambda x: x[0])
|
||||
|
||||
for module_path, data in sorted_modules:
|
||||
# Calculate module depth for indentation
|
||||
depth = module_path.count('/')
|
||||
indent = " " * depth
|
||||
|
||||
# Module header
|
||||
module_name = module_path.split('/')[-1] if module_path != 'root' else 'root'
|
||||
f.write(f"{indent}### 📁 `{module_name}/`\n\n")
|
||||
|
||||
# Module metadata
|
||||
if data['layers']:
|
||||
layers_str = ", ".join(sorted(data['layers']))
|
||||
f.write(f"{indent}- 🏗️ **Layers:** {layers_str}\n")
|
||||
|
||||
tiers_summary = []
|
||||
for tier_name, count in data['tiers'].items():
|
||||
if count > 0:
|
||||
tiers_summary.append(f"{tier_name}: {count}")
|
||||
if tiers_summary:
|
||||
f.write(f"{indent}- 📊 **Tiers:** {', '.join(tiers_summary)}\n")
|
||||
|
||||
f.write(f"{indent}- 📄 **Files:** {len(data['files'])}\n")
|
||||
f.write(f"{indent}- 📦 **Entities:** {len(data['entities'])}\n")
|
||||
|
||||
# List key entities (Modules, Classes, Components only)
|
||||
key_entities = [e for e in data['entities'] if e.type in ['Module', 'Class', 'Component', 'Store']]
|
||||
if key_entities:
|
||||
f.write(f"\n{indent}**Key Entities:**\n\n")
|
||||
for entity in sorted(key_entities, key=lambda x: (x.type, x.name))[:10]:
|
||||
icon = "📦" if entity.type == "Module" else "ℂ" if entity.type == "Class" else "🧩" if entity.type == "Component" else "🗄️"
|
||||
tier_badge = ""
|
||||
if entity.get_tier() == Tier.CRITICAL:
|
||||
tier_badge = " `[CRITICAL]`"
|
||||
elif entity.get_tier() == Tier.TRIVIAL:
|
||||
tier_badge = " `[TRIVIAL]`"
|
||||
purpose = entity.tags.get('PURPOSE', '')[:60] + "..." if entity.tags.get('PURPOSE') and len(entity.tags.get('PURPOSE', '')) > 60 else entity.tags.get('PURPOSE', '')
|
||||
f.write(f"{indent} - {icon} **{entity.name}** ({entity.type}){tier_badge}\n")
|
||||
if purpose:
|
||||
f.write(f"{indent} - {purpose}\n")
|
||||
|
||||
# External relations
|
||||
external_relations = [r for r in data['relations'] if r['type'] in ['DEPENDS_ON', 'IMPLEMENTS', 'INHERITS']]
|
||||
if external_relations:
|
||||
unique_deps = {}
|
||||
for rel in external_relations:
|
||||
key = f"{rel['type']} -> {rel['target']}"
|
||||
unique_deps[key] = rel
|
||||
f.write(f"\n{indent}**Dependencies:**\n\n")
|
||||
for rel_str in sorted(unique_deps.keys())[:5]:
|
||||
f.write(f"{indent} - 🔗 {rel_str}\n")
|
||||
|
||||
f.write("\n")
|
||||
|
||||
# Cross-module dependency graph
|
||||
f.write("## Cross-Module Dependencies\n\n")
|
||||
f.write("```mermaid\n")
|
||||
f.write("graph TD\n")
|
||||
|
||||
# Find inter-module dependencies
|
||||
for module_path, data in sorted_modules:
|
||||
module_name = module_path.split('/')[-1] if module_path != 'root' else 'root'
|
||||
safe_name = module_name.replace('-', '_').replace('.', '_')
|
||||
|
||||
for rel in data['relations']:
|
||||
target = rel.get('target', '')
|
||||
# Check if target references another module
|
||||
for other_module in modules:
|
||||
if other_module != module_path and other_module in target:
|
||||
other_name = other_module.split('/')[-1]
|
||||
safe_other = other_name.replace('-', '_').replace('.', '_')
|
||||
f.write(f" {safe_name}-->|{rel['type']}|{safe_other}\n")
|
||||
break
|
||||
|
||||
f.write("```\n")
|
||||
|
||||
print(f"Generated {OUTPUT_MODULE_MAP_MD}")
|
||||
# [/DEF:_generate_module_map:Function]
|
||||
|
||||
# [/DEF:SemanticMapGenerator:Class]
|
||||
|
||||
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
15
ut
Normal file
15
ut
Normal file
@@ -0,0 +1,15 @@
|
||||
Prepended http:// to './RealiTLScanner'
|
||||
--2026-02-20 11:14:59-- http://./RealiTLScanner
|
||||
Распознаётся . (.)… ошибка: С именем узла не связано ни одного адреса.
|
||||
wget: не удаётся разрешить адрес ‘.’
|
||||
Prepended http:// to 'www.microsoft.com'
|
||||
--2026-02-20 11:14:59-- http://www.microsoft.com/
|
||||
Распознаётся www.microsoft.com (www.microsoft.com)… 95.100.178.81
|
||||
Подключение к www.microsoft.com (www.microsoft.com)|95.100.178.81|:80... соединение установлено.
|
||||
HTTP-запрос отправлен. Ожидание ответа… 403 Forbidden
|
||||
2026-02-20 11:15:00 ОШИБКА 403: Forbidden.
|
||||
|
||||
Prepended http:// to 'file.csv'
|
||||
--2026-02-20 11:15:00-- http://file.csv/
|
||||
Распознаётся file.csv (file.csv)… ошибка: Неизвестное имя или служба.
|
||||
wget: не удаётся разрешить адрес ‘file.csv’
|
||||
Reference in New Issue
Block a user