Compare commits
5 Commits
013-unify-
...
edf9286071
| Author | SHA1 | Date | |
|---|---|---|---|
| edf9286071 | |||
| a542e7d2df | |||
| a863807cf2 | |||
| e2bc68683f | |||
| 43cb82697b |
1
.gitignore
vendored
1
.gitignore
vendored
@@ -69,3 +69,4 @@ backend/tasks.db
|
||||
|
||||
# Git Integration repositories
|
||||
backend/git_repos/
|
||||
backend/backend/git_repos
|
||||
|
||||
@@ -27,6 +27,8 @@ Auto-generated from all feature plans. Last updated: 2025-12-19
|
||||
- SQLite (for config/history), Filesystem (local Git repositories) (011-git-integration-dashboard)
|
||||
- Node.js 18+ (Frontend Build), Svelte 5.x + SvelteKit, Tailwind CSS, `date-fns` (existing) (013-unify-frontend-css)
|
||||
- LocalStorage (for language preference) (013-unify-frontend-css)
|
||||
- Python 3.9+ (Backend), Node.js 18+ (Frontend) + FastAPI (Backend), SvelteKit (Frontend) (014-file-storage-ui)
|
||||
- Local Filesystem (for artifacts), Config (for storage path) (014-file-storage-ui)
|
||||
|
||||
- Python 3.9+ (Backend), Node.js 18+ (Frontend Build) (001-plugin-arch-svelte-ui)
|
||||
|
||||
@@ -47,9 +49,9 @@ cd src; pytest; ruff check .
|
||||
Python 3.9+ (Backend), Node.js 18+ (Frontend Build): Follow standard conventions
|
||||
|
||||
## Recent Changes
|
||||
- 014-file-storage-ui: Added Python 3.9+ (Backend), Node.js 18+ (Frontend) + FastAPI (Backend), SvelteKit (Frontend)
|
||||
- 013-unify-frontend-css: Added Node.js 18+ (Frontend Build), Svelte 5.x + SvelteKit, Tailwind CSS, `date-fns` (existing)
|
||||
- 011-git-integration-dashboard: Added Python 3.9+ (Backend), Node.js 18+ (Frontend) + FastAPI, SvelteKit, GitPython (or CLI git), Pydantic, SQLAlchemy, Superset API
|
||||
- 011-git-integration-dashboard: Added Python 3.9+ (Backend), Node.js 18+ (Frontend) + FastAPI, SvelteKit, GitPython (or CLI git), Pydantic, SQLAlchemy, Superset API
|
||||
|
||||
|
||||
<!-- MANUAL ADDITIONS START -->
|
||||
|
||||
Submodule backend/backend/git_repos/12 updated: d592fa7ed5...f46772443a
@@ -1,269 +0,0 @@
|
||||
2025-12-20 19:55:11,325 - INFO - [BackupPlugin][Entry] Starting backup for superset.
|
||||
2025-12-20 19:55:11,325 - INFO - [setup_clients][Enter] Starting Superset clients initialization.
|
||||
2025-12-20 19:55:11,327 - CRITICAL - [setup_clients][Failure] Critical error during client initialization: 1 validation error for SupersetConfig
|
||||
base_url
|
||||
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
|
||||
For further information visit https://errors.pydantic.dev/2.12/v/value_error
|
||||
Traceback (most recent call last):
|
||||
File "/home/user/ss-tools/superset_tool/utils/init_clients.py", line 43, in setup_clients
|
||||
config = SupersetConfig(
|
||||
^^^^^^^^^^^^^^^
|
||||
File "/home/user/ss-tools/backend/venv/lib/python3.12/site-packages/pydantic/main.py", line 250, in __init__
|
||||
validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
pydantic_core._pydantic_core.ValidationError: 1 validation error for SupersetConfig
|
||||
base_url
|
||||
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
|
||||
For further information visit https://errors.pydantic.dev/2.12/v/value_error
|
||||
2025-12-20 21:01:49,905 - INFO - [BackupPlugin][Entry] Starting backup for superset.
|
||||
2025-12-20 21:01:49,906 - INFO - [setup_clients][Enter] Starting Superset clients initialization.
|
||||
2025-12-20 21:01:49,988 - INFO - [setup_clients][Action] Loading environments from ConfigManager
|
||||
2025-12-20 21:01:49,990 - CRITICAL - [setup_clients][Failure] Critical error during client initialization: 1 validation error for SupersetConfig
|
||||
base_url
|
||||
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
|
||||
For further information visit https://errors.pydantic.dev/2.12/v/value_error
|
||||
Traceback (most recent call last):
|
||||
File "/home/user/ss-tools/superset_tool/utils/init_clients.py", line 66, in setup_clients
|
||||
config = SupersetConfig(
|
||||
^^^^^^^^^^^^^^^
|
||||
File "/home/user/ss-tools/venv/lib/python3.12/site-packages/pydantic/main.py", line 250, in __init__
|
||||
validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
pydantic_core._pydantic_core.ValidationError: 1 validation error for SupersetConfig
|
||||
base_url
|
||||
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
|
||||
For further information visit https://errors.pydantic.dev/2.12/v/value_error
|
||||
2025-12-20 22:42:32,538 - INFO - [BackupPlugin][Entry] Starting backup for superset.
|
||||
2025-12-20 22:42:32,538 - INFO - [setup_clients][Enter] Starting Superset clients initialization.
|
||||
2025-12-20 22:42:32,583 - INFO - [setup_clients][Action] Loading environments from ConfigManager
|
||||
2025-12-20 22:42:32,587 - CRITICAL - [setup_clients][Failure] Critical error during client initialization: 1 validation error for SupersetConfig
|
||||
base_url
|
||||
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
|
||||
For further information visit https://errors.pydantic.dev/2.12/v/value_error
|
||||
Traceback (most recent call last):
|
||||
File "/home/user/ss-tools/superset_tool/utils/init_clients.py", line 66, in setup_clients
|
||||
config = SupersetConfig(
|
||||
^^^^^^^^^^^^^^^
|
||||
File "/home/user/ss-tools/backend/.venv/lib/python3.12/site-packages/pydantic/main.py", line 250, in __init__
|
||||
validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
pydantic_core._pydantic_core.ValidationError: 1 validation error for SupersetConfig
|
||||
base_url
|
||||
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
|
||||
For further information visit https://errors.pydantic.dev/2.12/v/value_error
|
||||
2025-12-20 22:54:29,770 - INFO - [BackupPlugin][Entry] Starting backup for .
|
||||
2025-12-20 22:54:29,771 - INFO - [setup_clients][Enter] Starting Superset clients initialization.
|
||||
2025-12-20 22:54:29,831 - INFO - [setup_clients][Action] Loading environments from ConfigManager
|
||||
2025-12-20 22:54:29,833 - CRITICAL - [setup_clients][Failure] Critical error during client initialization: 1 validation error for SupersetConfig
|
||||
base_url
|
||||
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
|
||||
For further information visit https://errors.pydantic.dev/2.12/v/value_error
|
||||
Traceback (most recent call last):
|
||||
File "/home/user/ss-tools/superset_tool/utils/init_clients.py", line 66, in setup_clients
|
||||
config = SupersetConfig(
|
||||
^^^^^^^^^^^^^^^
|
||||
File "/home/user/ss-tools/backend/.venv/lib/python3.12/site-packages/pydantic/main.py", line 250, in __init__
|
||||
validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
pydantic_core._pydantic_core.ValidationError: 1 validation error for SupersetConfig
|
||||
base_url
|
||||
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
|
||||
For further information visit https://errors.pydantic.dev/2.12/v/value_error
|
||||
2025-12-20 22:54:34,078 - INFO - [BackupPlugin][Entry] Starting backup for superset.
|
||||
2025-12-20 22:54:34,078 - INFO - [setup_clients][Enter] Starting Superset clients initialization.
|
||||
2025-12-20 22:54:34,079 - INFO - [setup_clients][Action] Loading environments from ConfigManager
|
||||
2025-12-20 22:54:34,079 - CRITICAL - [setup_clients][Failure] Critical error during client initialization: 1 validation error for SupersetConfig
|
||||
base_url
|
||||
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
|
||||
For further information visit https://errors.pydantic.dev/2.12/v/value_error
|
||||
Traceback (most recent call last):
|
||||
File "/home/user/ss-tools/superset_tool/utils/init_clients.py", line 66, in setup_clients
|
||||
config = SupersetConfig(
|
||||
^^^^^^^^^^^^^^^
|
||||
File "/home/user/ss-tools/backend/.venv/lib/python3.12/site-packages/pydantic/main.py", line 250, in __init__
|
||||
validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
pydantic_core._pydantic_core.ValidationError: 1 validation error for SupersetConfig
|
||||
base_url
|
||||
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
|
||||
For further information visit https://errors.pydantic.dev/2.12/v/value_error
|
||||
2025-12-20 22:59:25,060 - INFO - [BackupPlugin][Entry] Starting backup for superset.
|
||||
2025-12-20 22:59:25,060 - INFO - [setup_clients][Enter] Starting Superset clients initialization.
|
||||
2025-12-20 22:59:25,114 - INFO - [setup_clients][Action] Loading environments from ConfigManager
|
||||
2025-12-20 22:59:25,117 - CRITICAL - [setup_clients][Failure] Critical error during client initialization: 1 validation error for SupersetConfig
|
||||
base_url
|
||||
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
|
||||
For further information visit https://errors.pydantic.dev/2.12/v/value_error
|
||||
Traceback (most recent call last):
|
||||
File "/home/user/ss-tools/superset_tool/utils/init_clients.py", line 66, in setup_clients
|
||||
config = SupersetConfig(
|
||||
^^^^^^^^^^^^^^^
|
||||
File "/home/user/ss-tools/backend/.venv/lib/python3.12/site-packages/pydantic/main.py", line 250, in __init__
|
||||
validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
pydantic_core._pydantic_core.ValidationError: 1 validation error for SupersetConfig
|
||||
base_url
|
||||
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
|
||||
For further information visit https://errors.pydantic.dev/2.12/v/value_error
|
||||
2025-12-20 23:00:31,156 - INFO - [BackupPlugin][Entry] Starting backup for superset.
|
||||
2025-12-20 23:00:31,156 - INFO - [setup_clients][Enter] Starting Superset clients initialization.
|
||||
2025-12-20 23:00:31,157 - INFO - [setup_clients][Action] Loading environments from ConfigManager
|
||||
2025-12-20 23:00:31,162 - CRITICAL - [setup_clients][Failure] Critical error during client initialization: 1 validation error for SupersetConfig
|
||||
base_url
|
||||
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
|
||||
For further information visit https://errors.pydantic.dev/2.12/v/value_error
|
||||
Traceback (most recent call last):
|
||||
File "/home/user/ss-tools/superset_tool/utils/init_clients.py", line 66, in setup_clients
|
||||
config = SupersetConfig(
|
||||
^^^^^^^^^^^^^^^
|
||||
File "/home/user/ss-tools/backend/.venv/lib/python3.12/site-packages/pydantic/main.py", line 250, in __init__
|
||||
validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
pydantic_core._pydantic_core.ValidationError: 1 validation error for SupersetConfig
|
||||
base_url
|
||||
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
|
||||
For further information visit https://errors.pydantic.dev/2.12/v/value_error
|
||||
2025-12-20 23:00:34,710 - INFO - [BackupPlugin][Entry] Starting backup for superset.
|
||||
2025-12-20 23:00:34,710 - INFO - [setup_clients][Enter] Starting Superset clients initialization.
|
||||
2025-12-20 23:00:34,710 - INFO - [setup_clients][Action] Loading environments from ConfigManager
|
||||
2025-12-20 23:00:34,711 - CRITICAL - [setup_clients][Failure] Critical error during client initialization: 1 validation error for SupersetConfig
|
||||
base_url
|
||||
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
|
||||
For further information visit https://errors.pydantic.dev/2.12/v/value_error
|
||||
Traceback (most recent call last):
|
||||
File "/home/user/ss-tools/superset_tool/utils/init_clients.py", line 66, in setup_clients
|
||||
config = SupersetConfig(
|
||||
^^^^^^^^^^^^^^^
|
||||
File "/home/user/ss-tools/backend/.venv/lib/python3.12/site-packages/pydantic/main.py", line 250, in __init__
|
||||
validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
pydantic_core._pydantic_core.ValidationError: 1 validation error for SupersetConfig
|
||||
base_url
|
||||
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
|
||||
For further information visit https://errors.pydantic.dev/2.12/v/value_error
|
||||
2025-12-20 23:01:43,894 - INFO - [BackupPlugin][Entry] Starting backup for superset.
|
||||
2025-12-20 23:01:43,894 - INFO - [setup_clients][Enter] Starting Superset clients initialization.
|
||||
2025-12-20 23:01:43,895 - INFO - [setup_clients][Action] Loading environments from ConfigManager
|
||||
2025-12-20 23:01:43,895 - CRITICAL - [setup_clients][Failure] Critical error during client initialization: 1 validation error for SupersetConfig
|
||||
base_url
|
||||
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
|
||||
For further information visit https://errors.pydantic.dev/2.12/v/value_error
|
||||
Traceback (most recent call last):
|
||||
File "/home/user/ss-tools/superset_tool/utils/init_clients.py", line 66, in setup_clients
|
||||
config = SupersetConfig(
|
||||
^^^^^^^^^^^^^^^
|
||||
File "/home/user/ss-tools/backend/.venv/lib/python3.12/site-packages/pydantic/main.py", line 250, in __init__
|
||||
validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
pydantic_core._pydantic_core.ValidationError: 1 validation error for SupersetConfig
|
||||
base_url
|
||||
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
|
||||
For further information visit https://errors.pydantic.dev/2.12/v/value_error
|
||||
2025-12-20 23:04:07,731 - INFO - [BackupPlugin][Entry] Starting backup for superset.
|
||||
2025-12-20 23:04:07,731 - INFO - [setup_clients][Enter] Starting Superset clients initialization.
|
||||
2025-12-20 23:04:07,732 - INFO - [setup_clients][Action] Loading environments from ConfigManager
|
||||
2025-12-20 23:04:07,732 - CRITICAL - [setup_clients][Failure] Critical error during client initialization: 1 validation error for SupersetConfig
|
||||
base_url
|
||||
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
|
||||
For further information visit https://errors.pydantic.dev/2.12/v/value_error
|
||||
Traceback (most recent call last):
|
||||
File "/home/user/ss-tools/superset_tool/utils/init_clients.py", line 66, in setup_clients
|
||||
config = SupersetConfig(
|
||||
^^^^^^^^^^^^^^^
|
||||
File "/home/user/ss-tools/backend/.venv/lib/python3.12/site-packages/pydantic/main.py", line 250, in __init__
|
||||
validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
pydantic_core._pydantic_core.ValidationError: 1 validation error for SupersetConfig
|
||||
base_url
|
||||
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
|
||||
For further information visit https://errors.pydantic.dev/2.12/v/value_error
|
||||
2025-12-20 23:06:39,641 - INFO - [BackupPlugin][Entry] Starting backup for superset.
|
||||
2025-12-20 23:06:39,642 - INFO - [setup_clients][Enter] Starting Superset clients initialization.
|
||||
2025-12-20 23:06:39,687 - INFO - [setup_clients][Action] Loading environments from ConfigManager
|
||||
2025-12-20 23:06:39,689 - CRITICAL - [setup_clients][Failure] Critical error during client initialization: 1 validation error for SupersetConfig
|
||||
base_url
|
||||
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
|
||||
For further information visit https://errors.pydantic.dev/2.12/v/value_error
|
||||
Traceback (most recent call last):
|
||||
File "/home/user/ss-tools/superset_tool/utils/init_clients.py", line 66, in setup_clients
|
||||
config = SupersetConfig(
|
||||
^^^^^^^^^^^^^^^
|
||||
File "/home/user/ss-tools/backend/.venv/lib/python3.12/site-packages/pydantic/main.py", line 250, in __init__
|
||||
validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
pydantic_core._pydantic_core.ValidationError: 1 validation error for SupersetConfig
|
||||
base_url
|
||||
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
|
||||
For further information visit https://errors.pydantic.dev/2.12/v/value_error
|
||||
2025-12-20 23:30:36,090 - INFO - [BackupPlugin][Entry] Starting backup for superset.
|
||||
2025-12-20 23:30:36,093 - INFO - [setup_clients][Enter] Starting Superset clients initialization.
|
||||
2025-12-20 23:30:36,128 - INFO - [setup_clients][Action] Loading environments from ConfigManager
|
||||
2025-12-20 23:30:36,129 - INFO - [SupersetClient.__init__][Enter] Initializing SupersetClient.
|
||||
2025-12-20 23:30:36,129 - INFO - [APIClient.__init__][Entry] Initializing APIClient.
|
||||
2025-12-20 23:30:36,130 - WARNING - [_init_session][State] SSL verification disabled.
|
||||
2025-12-20 23:30:36,130 - INFO - [APIClient.__init__][Exit] APIClient initialized.
|
||||
2025-12-20 23:30:36,130 - INFO - [SupersetClient.__init__][Exit] SupersetClient initialized.
|
||||
2025-12-20 23:30:36,130 - INFO - [get_dashboards][Enter] Fetching dashboards.
|
||||
2025-12-20 23:30:36,131 - INFO - [authenticate][Enter] Authenticating to https://superset.bebesh.ru/api/v1
|
||||
2025-12-20 23:30:36,897 - INFO - [authenticate][Exit] Authenticated successfully.
|
||||
2025-12-20 23:30:37,527 - INFO - [get_dashboards][Exit] Found 11 dashboards.
|
||||
2025-12-20 23:30:37,527 - INFO - [BackupPlugin][Progress] Found 11 dashboards to export in superset.
|
||||
2025-12-20 23:30:37,529 - INFO - [export_dashboard][Enter] Exporting dashboard 11.
|
||||
2025-12-20 23:30:38,224 - INFO - [export_dashboard][Exit] Exported dashboard 11 to dashboard_export_20251220T203037.zip.
|
||||
2025-12-20 23:30:38,225 - INFO - [save_and_unpack_dashboard][Enter] Processing dashboard. Unpack: False
|
||||
2025-12-20 23:30:38,226 - INFO - [save_and_unpack_dashboard][State] Dashboard saved to: backups/SUPERSET/FCC New Coder Survey 2018/dashboard_export_20251220T203037.zip
|
||||
2025-12-20 23:30:38,227 - INFO - [archive_exports][Enter] Managing archive in backups/SUPERSET/FCC New Coder Survey 2018
|
||||
2025-12-20 23:30:38,230 - INFO - [export_dashboard][Enter] Exporting dashboard 10.
|
||||
2025-12-20 23:30:38,438 - INFO - [export_dashboard][Exit] Exported dashboard 10 to dashboard_export_20251220T203038.zip.
|
||||
2025-12-20 23:30:38,438 - INFO - [save_and_unpack_dashboard][Enter] Processing dashboard. Unpack: False
|
||||
2025-12-20 23:30:38,439 - INFO - [save_and_unpack_dashboard][State] Dashboard saved to: backups/SUPERSET/COVID Vaccine Dashboard/dashboard_export_20251220T203038.zip
|
||||
2025-12-20 23:30:38,439 - INFO - [archive_exports][Enter] Managing archive in backups/SUPERSET/COVID Vaccine Dashboard
|
||||
2025-12-20 23:30:38,440 - INFO - [export_dashboard][Enter] Exporting dashboard 9.
|
||||
2025-12-20 23:30:38,853 - INFO - [export_dashboard][Exit] Exported dashboard 9 to dashboard_export_20251220T203038.zip.
|
||||
2025-12-20 23:30:38,853 - INFO - [save_and_unpack_dashboard][Enter] Processing dashboard. Unpack: False
|
||||
2025-12-20 23:30:38,856 - INFO - [save_and_unpack_dashboard][State] Dashboard saved to: backups/SUPERSET/Sales Dashboard/dashboard_export_20251220T203038.zip
|
||||
2025-12-20 23:30:38,856 - INFO - [archive_exports][Enter] Managing archive in backups/SUPERSET/Sales Dashboard
|
||||
2025-12-20 23:30:38,858 - INFO - [export_dashboard][Enter] Exporting dashboard 8.
|
||||
2025-12-20 23:30:38,939 - INFO - [export_dashboard][Exit] Exported dashboard 8 to dashboard_export_20251220T203038.zip.
|
||||
2025-12-20 23:30:38,940 - INFO - [save_and_unpack_dashboard][Enter] Processing dashboard. Unpack: False
|
||||
2025-12-20 23:30:38,941 - INFO - [save_and_unpack_dashboard][State] Dashboard saved to: backups/SUPERSET/Unicode Test/dashboard_export_20251220T203038.zip
|
||||
2025-12-20 23:30:38,941 - INFO - [archive_exports][Enter] Managing archive in backups/SUPERSET/Unicode Test
|
||||
2025-12-20 23:30:38,942 - INFO - [export_dashboard][Enter] Exporting dashboard 7.
|
||||
2025-12-20 23:30:39,148 - INFO - [export_dashboard][Exit] Exported dashboard 7 to dashboard_export_20251220T203038.zip.
|
||||
2025-12-20 23:30:39,148 - INFO - [save_and_unpack_dashboard][Enter] Processing dashboard. Unpack: False
|
||||
2025-12-20 23:30:39,149 - INFO - [save_and_unpack_dashboard][State] Dashboard saved to: backups/SUPERSET/Video Game Sales/dashboard_export_20251220T203038.zip
|
||||
2025-12-20 23:30:39,149 - INFO - [archive_exports][Enter] Managing archive in backups/SUPERSET/Video Game Sales
|
||||
2025-12-20 23:30:39,150 - INFO - [export_dashboard][Enter] Exporting dashboard 6.
|
||||
2025-12-20 23:30:39,689 - INFO - [export_dashboard][Exit] Exported dashboard 6 to dashboard_export_20251220T203039.zip.
|
||||
2025-12-20 23:30:39,689 - INFO - [save_and_unpack_dashboard][Enter] Processing dashboard. Unpack: False
|
||||
2025-12-20 23:30:39,690 - INFO - [save_and_unpack_dashboard][State] Dashboard saved to: backups/SUPERSET/Featured Charts/dashboard_export_20251220T203039.zip
|
||||
2025-12-20 23:30:39,691 - INFO - [archive_exports][Enter] Managing archive in backups/SUPERSET/Featured Charts
|
||||
2025-12-20 23:30:39,692 - INFO - [export_dashboard][Enter] Exporting dashboard 5.
|
||||
2025-12-20 23:30:39,960 - INFO - [export_dashboard][Exit] Exported dashboard 5 to dashboard_export_20251220T203039.zip.
|
||||
2025-12-20 23:30:39,960 - INFO - [save_and_unpack_dashboard][Enter] Processing dashboard. Unpack: False
|
||||
2025-12-20 23:30:39,961 - INFO - [save_and_unpack_dashboard][State] Dashboard saved to: backups/SUPERSET/Slack Dashboard/dashboard_export_20251220T203039.zip
|
||||
2025-12-20 23:30:39,961 - INFO - [archive_exports][Enter] Managing archive in backups/SUPERSET/Slack Dashboard
|
||||
2025-12-20 23:30:39,962 - INFO - [export_dashboard][Enter] Exporting dashboard 4.
|
||||
2025-12-20 23:30:40,196 - INFO - [export_dashboard][Exit] Exported dashboard 4 to dashboard_export_20251220T203039.zip.
|
||||
2025-12-20 23:30:40,196 - INFO - [save_and_unpack_dashboard][Enter] Processing dashboard. Unpack: False
|
||||
2025-12-20 23:30:40,197 - INFO - [save_and_unpack_dashboard][State] Dashboard saved to: backups/SUPERSET/deck.gl Demo/dashboard_export_20251220T203039.zip
|
||||
2025-12-20 23:30:40,197 - INFO - [archive_exports][Enter] Managing archive in backups/SUPERSET/deck.gl Demo
|
||||
2025-12-20 23:30:40,198 - INFO - [export_dashboard][Enter] Exporting dashboard 3.
|
||||
2025-12-20 23:30:40,745 - INFO - [export_dashboard][Exit] Exported dashboard 3 to dashboard_export_20251220T203040.zip.
|
||||
2025-12-20 23:30:40,746 - INFO - [save_and_unpack_dashboard][Enter] Processing dashboard. Unpack: False
|
||||
2025-12-20 23:30:40,760 - INFO - [save_and_unpack_dashboard][State] Dashboard saved to: backups/SUPERSET/Misc Charts/dashboard_export_20251220T203040.zip
|
||||
2025-12-20 23:30:40,761 - INFO - [archive_exports][Enter] Managing archive in backups/SUPERSET/Misc Charts
|
||||
2025-12-20 23:30:40,762 - INFO - [export_dashboard][Enter] Exporting dashboard 2.
|
||||
2025-12-20 23:30:40,928 - INFO - [export_dashboard][Exit] Exported dashboard 2 to dashboard_export_20251220T203040.zip.
|
||||
2025-12-20 23:30:40,929 - INFO - [save_and_unpack_dashboard][Enter] Processing dashboard. Unpack: False
|
||||
2025-12-20 23:30:40,930 - INFO - [save_and_unpack_dashboard][State] Dashboard saved to: backups/SUPERSET/USA Births Names/dashboard_export_20251220T203040.zip
|
||||
2025-12-20 23:30:40,931 - INFO - [archive_exports][Enter] Managing archive in backups/SUPERSET/USA Births Names
|
||||
2025-12-20 23:30:40,932 - INFO - [export_dashboard][Enter] Exporting dashboard 1.
|
||||
2025-12-20 23:30:41,582 - INFO - [export_dashboard][Exit] Exported dashboard 1 to dashboard_export_20251220T203040.zip.
|
||||
2025-12-20 23:30:41,582 - INFO - [save_and_unpack_dashboard][Enter] Processing dashboard. Unpack: False
|
||||
2025-12-20 23:30:41,749 - INFO - [save_and_unpack_dashboard][State] Dashboard saved to: backups/SUPERSET/World Bank's Data/dashboard_export_20251220T203040.zip
|
||||
2025-12-20 23:30:41,750 - INFO - [archive_exports][Enter] Managing archive in backups/SUPERSET/World Bank's Data
|
||||
2025-12-20 23:30:41,752 - INFO - [consolidate_archive_folders][Enter] Consolidating archives in backups/SUPERSET
|
||||
2025-12-20 23:30:41,753 - INFO - [remove_empty_directories][Enter] Starting cleanup of empty directories in backups/SUPERSET
|
||||
2025-12-20 23:30:41,758 - INFO - [remove_empty_directories][Exit] Removed 0 empty directories.
|
||||
2025-12-20 23:30:41,758 - INFO - [BackupPlugin][CoherenceCheck:Passed] Backup logic completed for superset.
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -1 +1 @@
|
||||
from . import plugins, tasks, settings, connections, environments, mappings, migration, git
|
||||
from . import plugins, tasks, settings, connections, environments, mappings, migration, git, storage
|
||||
|
||||
@@ -13,6 +13,7 @@
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from typing import List
|
||||
from ...core.config_models import AppConfig, Environment, GlobalSettings
|
||||
from ...models.storage import StorageConfig
|
||||
from ...dependencies import get_config_manager
|
||||
from ...core.config_manager import ConfigManager
|
||||
from ...core.logger import logger, belief_scope
|
||||
@@ -52,10 +53,38 @@ async def update_global_settings(
|
||||
):
|
||||
with belief_scope("update_global_settings"):
|
||||
logger.info("[update_global_settings][Entry] Updating global settings")
|
||||
|
||||
config_manager.update_global_settings(settings)
|
||||
return settings
|
||||
# [/DEF:update_global_settings:Function]
|
||||
|
||||
# [DEF:get_storage_settings:Function]
|
||||
# @PURPOSE: Retrieves storage-specific settings.
|
||||
# @RETURN: StorageConfig - The storage configuration.
|
||||
@router.get("/storage", response_model=StorageConfig)
|
||||
async def get_storage_settings(config_manager: ConfigManager = Depends(get_config_manager)):
|
||||
with belief_scope("get_storage_settings"):
|
||||
return config_manager.get_config().settings.storage
|
||||
# [/DEF:get_storage_settings:Function]
|
||||
|
||||
# [DEF:update_storage_settings:Function]
|
||||
# @PURPOSE: Updates storage-specific settings.
|
||||
# @PARAM: storage (StorageConfig) - The new storage settings.
|
||||
# @POST: Storage settings are updated and saved.
|
||||
# @RETURN: StorageConfig - The updated storage settings.
|
||||
@router.put("/storage", response_model=StorageConfig)
|
||||
async def update_storage_settings(storage: StorageConfig, config_manager: ConfigManager = Depends(get_config_manager)):
|
||||
with belief_scope("update_storage_settings"):
|
||||
is_valid, message = config_manager.validate_path(storage.root_path)
|
||||
if not is_valid:
|
||||
raise HTTPException(status_code=400, detail=message)
|
||||
|
||||
settings = config_manager.get_config().settings
|
||||
settings.storage = storage
|
||||
config_manager.update_global_settings(settings)
|
||||
return config_manager.get_config().settings.storage
|
||||
# [/DEF:update_storage_settings:Function]
|
||||
|
||||
# [DEF:get_environments:Function]
|
||||
# @PURPOSE: Lists all configured Superset environments.
|
||||
# @PRE: Config manager is available.
|
||||
@@ -179,30 +208,5 @@ async def test_environment_connection(
|
||||
return {"status": "error", "message": str(e)}
|
||||
# [/DEF:test_environment_connection:Function]
|
||||
|
||||
# [DEF:validate_backup_path:Function]
|
||||
# @PURPOSE: Validates if a backup path exists and is writable.
|
||||
# @PRE: Path is provided in path_data.
|
||||
# @POST: Returns success or error status.
|
||||
# @PARAM: path (str) - The path to validate.
|
||||
# @RETURN: dict - Validation result.
|
||||
@router.post("/validate-path")
|
||||
async def validate_backup_path(
|
||||
path_data: dict,
|
||||
config_manager: ConfigManager = Depends(get_config_manager)
|
||||
):
|
||||
with belief_scope("validate_backup_path"):
|
||||
path = path_data.get("path")
|
||||
if not path:
|
||||
raise HTTPException(status_code=400, detail="Path is required")
|
||||
|
||||
logger.info(f"[validate_backup_path][Entry] Validating path: {path}")
|
||||
|
||||
valid, message = config_manager.validate_path(path)
|
||||
|
||||
if not valid:
|
||||
return {"status": "error", "message": message}
|
||||
|
||||
return {"status": "success", "message": message}
|
||||
# [/DEF:validate_backup_path:Function]
|
||||
|
||||
# [/DEF:SettingsRouter:Module]
|
||||
|
||||
132
backend/src/api/routes/storage.py
Normal file
132
backend/src/api/routes/storage.py
Normal file
@@ -0,0 +1,132 @@
|
||||
# [DEF:storage_routes:Module]
|
||||
#
|
||||
# @SEMANTICS: storage, files, upload, download, backup, repository
|
||||
# @PURPOSE: API endpoints for file storage management (backups and repositories).
|
||||
# @LAYER: API
|
||||
# @RELATION: DEPENDS_ON -> backend.src.models.storage
|
||||
#
|
||||
# @INVARIANT: All paths must be validated against path traversal.
|
||||
|
||||
# [SECTION: IMPORTS]
|
||||
from fastapi import APIRouter, Depends, UploadFile, File, Form, HTTPException
|
||||
from fastapi.responses import FileResponse
|
||||
from typing import List, Optional
|
||||
from ...models.storage import StoredFile, FileCategory
|
||||
from ...dependencies import get_plugin_loader
|
||||
from ...plugins.storage.plugin import StoragePlugin
|
||||
from ...core.logger import belief_scope
|
||||
# [/SECTION]
|
||||
|
||||
router = APIRouter(tags=["storage"])
|
||||
|
||||
# [DEF:list_files:Function]
|
||||
# @PURPOSE: List all files and directories in the storage system.
|
||||
#
|
||||
# @PRE: None.
|
||||
# @POST: Returns a list of StoredFile objects.
|
||||
#
|
||||
# @PARAM: category (Optional[FileCategory]) - Filter by category.
|
||||
# @PARAM: path (Optional[str]) - Subpath within the category.
|
||||
# @RETURN: List[StoredFile] - List of files/directories.
|
||||
#
|
||||
# @RELATION: CALLS -> StoragePlugin.list_files
|
||||
@router.get("/files", response_model=List[StoredFile])
|
||||
async def list_files(
|
||||
category: Optional[FileCategory] = None,
|
||||
path: Optional[str] = None,
|
||||
plugin_loader=Depends(get_plugin_loader)
|
||||
):
|
||||
with belief_scope("list_files"):
|
||||
storage_plugin: StoragePlugin = plugin_loader.get_plugin("storage-manager")
|
||||
if not storage_plugin:
|
||||
raise HTTPException(status_code=500, detail="Storage plugin not loaded")
|
||||
return storage_plugin.list_files(category, path)
|
||||
# [/DEF:list_files:Function]
|
||||
|
||||
# [DEF:upload_file:Function]
|
||||
# @PURPOSE: Upload a file to the storage system.
|
||||
#
|
||||
# @PRE: category must be a valid FileCategory.
|
||||
# @PRE: file must be a valid UploadFile.
|
||||
# @POST: Returns the StoredFile object of the uploaded file.
|
||||
#
|
||||
# @PARAM: category (FileCategory) - Target category.
|
||||
# @PARAM: path (Optional[str]) - Target subpath.
|
||||
# @PARAM: file (UploadFile) - The file content.
|
||||
# @RETURN: StoredFile - Metadata of the uploaded file.
|
||||
#
|
||||
# @SIDE_EFFECT: Writes file to the filesystem.
|
||||
#
|
||||
# @RELATION: CALLS -> StoragePlugin.save_file
|
||||
@router.post("/upload", response_model=StoredFile, status_code=201)
|
||||
async def upload_file(
|
||||
category: FileCategory = Form(...),
|
||||
path: Optional[str] = Form(None),
|
||||
file: UploadFile = File(...),
|
||||
plugin_loader=Depends(get_plugin_loader)
|
||||
):
|
||||
with belief_scope("upload_file"):
|
||||
storage_plugin: StoragePlugin = plugin_loader.get_plugin("storage-manager")
|
||||
if not storage_plugin:
|
||||
raise HTTPException(status_code=500, detail="Storage plugin not loaded")
|
||||
try:
|
||||
return await storage_plugin.save_file(file, category, path)
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
# [/DEF:upload_file:Function]
|
||||
|
||||
# [DEF:delete_file:Function]
|
||||
# @PURPOSE: Delete a specific file or directory.
|
||||
#
|
||||
# @PRE: category must be a valid FileCategory.
|
||||
# @POST: Item is removed from storage.
|
||||
#
|
||||
# @PARAM: category (FileCategory) - File category.
|
||||
# @PARAM: path (str) - Relative path of the item.
|
||||
# @RETURN: None
|
||||
#
|
||||
# @SIDE_EFFECT: Deletes item from the filesystem.
|
||||
#
|
||||
# @RELATION: CALLS -> StoragePlugin.delete_file
|
||||
@router.delete("/files/{category}/{path:path}", status_code=204)
|
||||
async def delete_file(category: FileCategory, path: str, plugin_loader=Depends(get_plugin_loader)):
|
||||
with belief_scope("delete_file"):
|
||||
storage_plugin: StoragePlugin = plugin_loader.get_plugin("storage-manager")
|
||||
if not storage_plugin:
|
||||
raise HTTPException(status_code=500, detail="Storage plugin not loaded")
|
||||
try:
|
||||
storage_plugin.delete_file(category, path)
|
||||
except FileNotFoundError:
|
||||
raise HTTPException(status_code=404, detail="File not found")
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
# [/DEF:delete_file:Function]
|
||||
|
||||
# [DEF:download_file:Function]
|
||||
# @PURPOSE: Retrieve a file for download.
|
||||
#
|
||||
# @PRE: category must be a valid FileCategory.
|
||||
# @POST: Returns a FileResponse.
|
||||
#
|
||||
# @PARAM: category (FileCategory) - File category.
|
||||
# @PARAM: path (str) - Relative path of the file.
|
||||
# @RETURN: FileResponse - The file content.
|
||||
#
|
||||
# @RELATION: CALLS -> StoragePlugin.get_file_path
|
||||
@router.get("/download/{category}/{path:path}")
|
||||
async def download_file(category: FileCategory, path: str, plugin_loader=Depends(get_plugin_loader)):
|
||||
with belief_scope("download_file"):
|
||||
storage_plugin: StoragePlugin = plugin_loader.get_plugin("storage-manager")
|
||||
if not storage_plugin:
|
||||
raise HTTPException(status_code=500, detail="Storage plugin not loaded")
|
||||
try:
|
||||
abs_path = storage_plugin.get_file_path(category, path)
|
||||
filename = Path(path).name
|
||||
return FileResponse(path=abs_path, filename=filename)
|
||||
except FileNotFoundError:
|
||||
raise HTTPException(status_code=404, detail="File not found")
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
# [/DEF:download_file:Function]
|
||||
|
||||
# [/DEF:storage_routes:Module]
|
||||
@@ -18,7 +18,7 @@ import os
|
||||
|
||||
from .dependencies import get_task_manager, get_scheduler_service
|
||||
from .core.logger import logger, belief_scope
|
||||
from .api.routes import plugins, tasks, settings, environments, mappings, migration, connections, git
|
||||
from .api.routes import plugins, tasks, settings, environments, mappings, migration, connections, git, storage
|
||||
from .core.database import init_db
|
||||
|
||||
# [DEF:App:Global]
|
||||
@@ -89,6 +89,7 @@ app.include_router(environments.router, prefix="/api/environments", tags=["Envir
|
||||
app.include_router(mappings.router)
|
||||
app.include_router(migration.router)
|
||||
app.include_router(git.router)
|
||||
app.include_router(storage.router, prefix="/api/storage", tags=["Storage"])
|
||||
|
||||
# [DEF:websocket_endpoint:Function]
|
||||
# @PURPOSE: Provides a WebSocket endpoint for real-time log streaming of a task.
|
||||
|
||||
@@ -62,14 +62,18 @@ class ConfigManager:
|
||||
logger.info(f"[_load_config][Action] Config file not found. Creating default.")
|
||||
default_config = AppConfig(
|
||||
environments=[],
|
||||
settings=GlobalSettings(backup_path="backups")
|
||||
settings=GlobalSettings()
|
||||
)
|
||||
self._save_config_to_disk(default_config)
|
||||
return default_config
|
||||
|
||||
try:
|
||||
with open(self.config_path, "r") as f:
|
||||
data = json.load(f)
|
||||
|
||||
# Check for deprecated field
|
||||
if "settings" in data and "backup_path" in data["settings"]:
|
||||
del data["settings"]["backup_path"]
|
||||
|
||||
config = AppConfig(**data)
|
||||
logger.info(f"[_load_config][Coherence:OK] Configuration loaded")
|
||||
return config
|
||||
@@ -79,7 +83,7 @@ class ConfigManager:
|
||||
# For now, return default to be safe, but log the error prominently.
|
||||
return AppConfig(
|
||||
environments=[],
|
||||
settings=GlobalSettings(backup_path="backups")
|
||||
settings=GlobalSettings(storage=StorageConfig())
|
||||
)
|
||||
# [/DEF:_load_config:Function]
|
||||
|
||||
|
||||
@@ -7,6 +7,7 @@
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
from typing import List, Optional
|
||||
from ..models.storage import StorageConfig
|
||||
|
||||
# [DEF:Schedule:DataClass]
|
||||
# @PURPOSE: Represents a backup schedule configuration.
|
||||
@@ -42,7 +43,7 @@ class LoggingConfig(BaseModel):
|
||||
# [DEF:GlobalSettings:DataClass]
|
||||
# @PURPOSE: Represents global application settings.
|
||||
class GlobalSettings(BaseModel):
|
||||
backup_path: str
|
||||
storage: StorageConfig = Field(default_factory=StorageConfig)
|
||||
default_environment_id: Optional[str] = None
|
||||
logging: LoggingConfig = Field(default_factory=LoggingConfig)
|
||||
|
||||
|
||||
@@ -50,9 +50,18 @@ class PluginLoader:
|
||||
sys.path.insert(0, plugin_parent_dir)
|
||||
|
||||
for filename in os.listdir(self.plugin_dir):
|
||||
file_path = os.path.join(self.plugin_dir, filename)
|
||||
|
||||
# Handle directory-based plugins (packages)
|
||||
if os.path.isdir(file_path):
|
||||
init_file = os.path.join(file_path, "__init__.py")
|
||||
if os.path.exists(init_file):
|
||||
self._load_module(filename, init_file)
|
||||
continue
|
||||
|
||||
# Handle single-file plugins
|
||||
if filename.endswith(".py") and filename != "__init__.py":
|
||||
module_name = filename[:-3]
|
||||
file_path = os.path.join(self.plugin_dir, filename)
|
||||
self._load_module(module_name, file_path)
|
||||
# [/DEF:_load_plugins:Function]
|
||||
|
||||
|
||||
28
backend/src/models/storage.py
Normal file
28
backend/src/models/storage.py
Normal file
@@ -0,0 +1,28 @@
|
||||
from datetime import datetime
|
||||
from enum import Enum
|
||||
from typing import Optional
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
# [DEF:FileCategory:Class]
|
||||
class FileCategory(str, Enum):
|
||||
BACKUP = "backups"
|
||||
REPOSITORY = "repositorys"
|
||||
# [/DEF:FileCategory:Class]
|
||||
|
||||
# [DEF:StorageConfig:Class]
|
||||
class StorageConfig(BaseModel):
|
||||
root_path: str = Field(default="backups", description="Absolute path to the storage root directory.")
|
||||
backup_structure_pattern: str = Field(default="{category}/", description="Pattern for backup directory structure.")
|
||||
repo_structure_pattern: str = Field(default="{category}/", description="Pattern for repository directory structure.")
|
||||
filename_pattern: str = Field(default="{name}_{timestamp}", description="Pattern for filenames.")
|
||||
# [/DEF:StorageConfig:Class]
|
||||
|
||||
# [DEF:StoredFile:Class]
|
||||
class StoredFile(BaseModel):
|
||||
name: str = Field(..., description="Name of the file (including extension).")
|
||||
path: str = Field(..., description="Relative path from storage root.")
|
||||
size: int = Field(..., ge=0, description="Size of the file in bytes.")
|
||||
created_at: datetime = Field(..., description="Creation timestamp.")
|
||||
category: FileCategory = Field(..., description="Category of the file.")
|
||||
mime_type: Optional[str] = Field(None, description="MIME type of the file.")
|
||||
# [/DEF:StoredFile:Class]
|
||||
@@ -84,7 +84,7 @@ class BackupPlugin(PluginBase):
|
||||
with belief_scope("get_schema"):
|
||||
config_manager = get_config_manager()
|
||||
envs = [e.name for e in config_manager.get_environments()]
|
||||
default_path = config_manager.get_config().settings.backup_path
|
||||
default_path = config_manager.get_config().settings.storage.root_path
|
||||
|
||||
return {
|
||||
"type": "object",
|
||||
@@ -95,14 +95,8 @@ class BackupPlugin(PluginBase):
|
||||
"description": "The Superset environment to back up.",
|
||||
"enum": envs if envs else [],
|
||||
},
|
||||
"backup_path": {
|
||||
"type": "string",
|
||||
"title": "Backup Path",
|
||||
"description": "The root directory to save backups to.",
|
||||
"default": default_path
|
||||
}
|
||||
},
|
||||
"required": ["env", "backup_path"],
|
||||
"required": ["env"],
|
||||
}
|
||||
# [/DEF:get_schema:Function]
|
||||
|
||||
@@ -126,8 +120,9 @@ class BackupPlugin(PluginBase):
|
||||
if not env:
|
||||
raise KeyError("env")
|
||||
|
||||
backup_path_str = params.get("backup_path") or config_manager.get_config().settings.backup_path
|
||||
backup_path = Path(backup_path_str)
|
||||
storage_settings = config_manager.get_config().settings.storage
|
||||
# Use 'backups' subfolder within the storage root
|
||||
backup_path = Path(storage_settings.root_path) / "backups"
|
||||
|
||||
from ..core.logger import logger as app_logger
|
||||
app_logger.info(f"[BackupPlugin][Entry] Starting backup for {env}.")
|
||||
|
||||
3
backend/src/plugins/storage/__init__.py
Normal file
3
backend/src/plugins/storage/__init__.py
Normal file
@@ -0,0 +1,3 @@
|
||||
from .plugin import StoragePlugin
|
||||
|
||||
__all__ = ["StoragePlugin"]
|
||||
292
backend/src/plugins/storage/plugin.py
Normal file
292
backend/src/plugins/storage/plugin.py
Normal file
@@ -0,0 +1,292 @@
|
||||
# [DEF:StoragePlugin:Module]
|
||||
#
|
||||
# @SEMANTICS: storage, files, filesystem, plugin
|
||||
# @PURPOSE: Provides core filesystem operations for managing backups and repositories.
|
||||
# @LAYER: App
|
||||
# @RELATION: IMPLEMENTS -> PluginBase
|
||||
# @RELATION: DEPENDS_ON -> backend.src.models.storage
|
||||
#
|
||||
# @INVARIANT: All file operations must be restricted to the configured storage root.
|
||||
|
||||
# [SECTION: IMPORTS]
|
||||
import os
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
from datetime import datetime
|
||||
from typing import Dict, Any, List, Optional
|
||||
from fastapi import UploadFile
|
||||
|
||||
from ...core.plugin_base import PluginBase
|
||||
from ...core.logger import belief_scope, logger
|
||||
from ...models.storage import StoredFile, FileCategory, StorageConfig
|
||||
from ...dependencies import get_config_manager
|
||||
# [/SECTION]
|
||||
|
||||
# [DEF:StoragePlugin:Class]
|
||||
# @PURPOSE: Implementation of the storage management plugin.
|
||||
class StoragePlugin(PluginBase):
|
||||
"""
|
||||
Plugin for managing local file storage for backups and repositories.
|
||||
"""
|
||||
|
||||
# [DEF:__init__:Function]
|
||||
# @PURPOSE: Initializes the StoragePlugin and ensures required directories exist.
|
||||
def __init__(self):
|
||||
with belief_scope("StoragePlugin:init"):
|
||||
self.ensure_directories()
|
||||
# [/DEF:__init__:Function]
|
||||
|
||||
@property
|
||||
# [DEF:id:Function]
|
||||
# @PURPOSE: Returns the unique identifier for the storage plugin.
|
||||
# @RETURN: str - "storage-manager"
|
||||
def id(self) -> str:
|
||||
return "storage-manager"
|
||||
# [/DEF:id:Function]
|
||||
|
||||
@property
|
||||
# [DEF:name:Function]
|
||||
# @PURPOSE: Returns the human-readable name of the storage plugin.
|
||||
# @RETURN: str - "Storage Manager"
|
||||
def name(self) -> str:
|
||||
return "Storage Manager"
|
||||
# [/DEF:name:Function]
|
||||
|
||||
@property
|
||||
# [DEF:description:Function]
|
||||
# @PURPOSE: Returns a description of the storage plugin.
|
||||
# @RETURN: str - Plugin description.
|
||||
def description(self) -> str:
|
||||
return "Manages local file storage for backups and repositories."
|
||||
# [/DEF:description:Function]
|
||||
|
||||
@property
|
||||
# [DEF:version:Function]
|
||||
# @PURPOSE: Returns the version of the storage plugin.
|
||||
# @RETURN: str - "1.0.0"
|
||||
def version(self) -> str:
|
||||
return "1.0.0"
|
||||
# [/DEF:version:Function]
|
||||
|
||||
# [DEF:get_schema:Function]
|
||||
# @PURPOSE: Returns the JSON schema for storage plugin parameters.
|
||||
# @RETURN: Dict[str, Any] - JSON schema.
|
||||
def get_schema(self) -> Dict[str, Any]:
|
||||
return {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"category": {
|
||||
"type": "string",
|
||||
"enum": [c.value for c in FileCategory],
|
||||
"title": "Category"
|
||||
}
|
||||
},
|
||||
"required": ["category"]
|
||||
}
|
||||
# [/DEF:get_schema:Function]
|
||||
|
||||
# [DEF:execute:Function]
|
||||
# @PURPOSE: Executes storage-related tasks (placeholder for PluginBase compliance).
|
||||
async def execute(self, params: Dict[str, Any]):
|
||||
with belief_scope("StoragePlugin:execute"):
|
||||
logger.info(f"[StoragePlugin][Action] Executing with params: {params}")
|
||||
# [/DEF:execute:Function]
|
||||
|
||||
# [DEF:get_storage_root:Function]
|
||||
# @PURPOSE: Resolves the absolute path to the storage root.
|
||||
# @POST: Returns a Path object representing the storage root.
|
||||
def get_storage_root(self) -> Path:
|
||||
with belief_scope("StoragePlugin:get_storage_root"):
|
||||
config_manager = get_config_manager()
|
||||
global_settings = config_manager.get_config().settings
|
||||
|
||||
# Use storage.root_path as the source of truth for storage UI
|
||||
root = Path(global_settings.storage.root_path)
|
||||
|
||||
if not root.is_absolute():
|
||||
# Resolve relative to the backend directory
|
||||
# Path(__file__) is backend/src/plugins/storage/plugin.py
|
||||
# parents[3] is the project root (ss-tools)
|
||||
# We need to ensure it's relative to where backend/ is
|
||||
project_root = Path(__file__).parents[3]
|
||||
root = (project_root / root).resolve()
|
||||
return root
|
||||
# [/DEF:get_storage_root:Function]
|
||||
|
||||
# [DEF:resolve_path:Function]
|
||||
# @PURPOSE: Resolves a dynamic path pattern using provided variables.
|
||||
# @PARAM: pattern (str) - The path pattern to resolve.
|
||||
# @PARAM: variables (Dict[str, str]) - Variables to substitute in the pattern.
|
||||
# @RETURN: str - The resolved path.
|
||||
def resolve_path(self, pattern: str, variables: Dict[str, str]) -> str:
|
||||
with belief_scope("StoragePlugin:resolve_path"):
|
||||
# Add common variables
|
||||
vars_with_defaults = {
|
||||
"timestamp": datetime.now().strftime("%Y%m%dT%H%M%S"),
|
||||
**variables
|
||||
}
|
||||
try:
|
||||
resolved = pattern.format(**vars_with_defaults)
|
||||
# Clean up any double slashes or leading/trailing slashes for relative path
|
||||
return os.path.normpath(resolved).strip("/")
|
||||
except KeyError as e:
|
||||
logger.warning(f"[StoragePlugin][Coherence:Failed] Missing variable for path resolution: {e}")
|
||||
# Fallback to literal pattern if formatting fails partially (or handle as needed)
|
||||
return pattern.replace("{", "").replace("}", "")
|
||||
# [/DEF:resolve_path:Function]
|
||||
|
||||
# [DEF:ensure_directories:Function]
|
||||
# @PURPOSE: Creates the storage root and category subdirectories if they don't exist.
|
||||
# @SIDE_EFFECT: Creates directories on the filesystem.
|
||||
def ensure_directories(self):
|
||||
with belief_scope("StoragePlugin:ensure_directories"):
|
||||
root = self.get_storage_root()
|
||||
for category in FileCategory:
|
||||
# Use singular name for consistency with BackupPlugin and GitService
|
||||
path = root / category.value
|
||||
path.mkdir(parents=True, exist_ok=True)
|
||||
logger.debug(f"[StoragePlugin][Action] Ensured directory: {path}")
|
||||
# [/DEF:ensure_directories:Function]
|
||||
|
||||
# [DEF:validate_path:Function]
|
||||
# @PURPOSE: Prevents path traversal attacks by ensuring the path is within the storage root.
|
||||
# @PRE: path must be a Path object.
|
||||
# @POST: Returns the resolved absolute path if valid, otherwise raises ValueError.
|
||||
def validate_path(self, path: Path) -> Path:
|
||||
with belief_scope("StoragePlugin:validate_path"):
|
||||
root = self.get_storage_root().resolve()
|
||||
resolved = path.resolve()
|
||||
try:
|
||||
resolved.relative_to(root)
|
||||
except ValueError:
|
||||
logger.error(f"[StoragePlugin][Coherence:Failed] Path traversal detected: {resolved} is not under {root}")
|
||||
raise ValueError("Access denied: Path is outside of storage root.")
|
||||
return resolved
|
||||
# [/DEF:validate_path:Function]
|
||||
|
||||
# [DEF:list_files:Function]
|
||||
# @PURPOSE: Lists all files and directories in a specific category and subpath.
|
||||
# @PARAM: category (Optional[FileCategory]) - The category to list.
|
||||
# @PARAM: subpath (Optional[str]) - Nested path within the category.
|
||||
# @RETURN: List[StoredFile] - List of file and directory metadata objects.
|
||||
def list_files(self, category: Optional[FileCategory] = None, subpath: Optional[str] = None) -> List[StoredFile]:
|
||||
with belief_scope("StoragePlugin:list_files"):
|
||||
root = self.get_storage_root()
|
||||
logger.info(f"[StoragePlugin][Action] Listing files in root: {root}, category: {category}, subpath: {subpath}")
|
||||
files = []
|
||||
|
||||
categories = [category] if category else list(FileCategory)
|
||||
|
||||
for cat in categories:
|
||||
# Scan the category subfolder + optional subpath
|
||||
base_dir = root / cat.value
|
||||
if subpath:
|
||||
target_dir = self.validate_path(base_dir / subpath)
|
||||
else:
|
||||
target_dir = base_dir
|
||||
|
||||
if not target_dir.exists():
|
||||
continue
|
||||
|
||||
logger.debug(f"[StoragePlugin][Action] Scanning directory: {target_dir}")
|
||||
|
||||
# Use os.scandir for better performance and to distinguish files vs dirs
|
||||
with os.scandir(target_dir) as it:
|
||||
for entry in it:
|
||||
# Skip logs
|
||||
if "Logs" in entry.path:
|
||||
continue
|
||||
|
||||
stat = entry.stat()
|
||||
is_dir = entry.is_dir()
|
||||
|
||||
files.append(StoredFile(
|
||||
name=entry.name,
|
||||
path=str(Path(entry.path).relative_to(root)),
|
||||
size=stat.st_size if not is_dir else 0,
|
||||
created_at=datetime.fromtimestamp(stat.st_ctime),
|
||||
category=cat,
|
||||
mime_type="directory" if is_dir else None
|
||||
))
|
||||
|
||||
# Sort: directories first, then by name
|
||||
return sorted(files, key=lambda x: (x.mime_type != "directory", x.name))
|
||||
# [/DEF:list_files:Function]
|
||||
|
||||
# [DEF:save_file:Function]
|
||||
# @PURPOSE: Saves an uploaded file to the specified category and optional subpath.
|
||||
# @PARAM: file (UploadFile) - The uploaded file.
|
||||
# @PARAM: category (FileCategory) - The target category.
|
||||
# @PARAM: subpath (Optional[str]) - The target subpath.
|
||||
# @RETURN: StoredFile - Metadata of the saved file.
|
||||
# @SIDE_EFFECT: Writes file to disk.
|
||||
async def save_file(self, file: UploadFile, category: FileCategory, subpath: Optional[str] = None) -> StoredFile:
|
||||
with belief_scope("StoragePlugin:save_file"):
|
||||
root = self.get_storage_root()
|
||||
dest_dir = root / category.value
|
||||
if subpath:
|
||||
dest_dir = dest_dir / subpath
|
||||
|
||||
dest_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
dest_path = self.validate_path(dest_dir / file.filename)
|
||||
|
||||
with dest_path.open("wb") as buffer:
|
||||
shutil.copyfileobj(file.file, buffer)
|
||||
|
||||
stat = dest_path.stat()
|
||||
return StoredFile(
|
||||
name=dest_path.name,
|
||||
path=str(dest_path.relative_to(root)),
|
||||
size=stat.st_size,
|
||||
created_at=datetime.fromtimestamp(stat.st_ctime),
|
||||
category=category,
|
||||
mime_type=file.content_type
|
||||
)
|
||||
# [/DEF:save_file:Function]
|
||||
|
||||
# [DEF:delete_file:Function]
|
||||
# @PURPOSE: Deletes a file or directory from the specified category and path.
|
||||
# @PARAM: category (FileCategory) - The category.
|
||||
# @PARAM: path (str) - The relative path of the file or directory.
|
||||
# @SIDE_EFFECT: Removes item from disk.
|
||||
def delete_file(self, category: FileCategory, path: str):
|
||||
with belief_scope("StoragePlugin:delete_file"):
|
||||
root = self.get_storage_root()
|
||||
# path is relative to root, but we ensure it starts with category
|
||||
full_path = self.validate_path(root / path)
|
||||
|
||||
if not str(Path(path)).startswith(category.value):
|
||||
raise ValueError(f"Path {path} does not belong to category {category}")
|
||||
|
||||
if full_path.exists():
|
||||
if full_path.is_dir():
|
||||
shutil.rmtree(full_path)
|
||||
else:
|
||||
full_path.unlink()
|
||||
logger.info(f"[StoragePlugin][Action] Deleted: {full_path}")
|
||||
else:
|
||||
raise FileNotFoundError(f"Item {path} not found")
|
||||
# [/DEF:delete_file:Function]
|
||||
|
||||
# [DEF:get_file_path:Function]
|
||||
# @PURPOSE: Returns the absolute path of a file for download.
|
||||
# @PARAM: category (FileCategory) - The category.
|
||||
# @PARAM: path (str) - The relative path of the file.
|
||||
# @RETURN: Path - Absolute path to the file.
|
||||
def get_file_path(self, category: FileCategory, path: str) -> Path:
|
||||
with belief_scope("StoragePlugin:get_file_path"):
|
||||
root = self.get_storage_root()
|
||||
file_path = self.validate_path(root / path)
|
||||
|
||||
if not str(Path(path)).startswith(category.value):
|
||||
raise ValueError(f"Path {path} does not belong to category {category}")
|
||||
|
||||
if not file_path.exists() or file_path.is_dir():
|
||||
raise FileNotFoundError(f"File {path} not found")
|
||||
|
||||
return file_path
|
||||
# [/DEF:get_file_path:Function]
|
||||
|
||||
# [/DEF:StoragePlugin:Class]
|
||||
# [/DEF:StoragePlugin:Module]
|
||||
@@ -31,9 +31,15 @@ class GitService:
|
||||
# @PARAM: base_path (str) - Root directory for all Git clones.
|
||||
# @PRE: base_path is a valid string path.
|
||||
# @POST: GitService is initialized; base_path directory exists.
|
||||
def __init__(self, base_path: str = "backend/git_repos"):
|
||||
def __init__(self, base_path: str = "git_repos"):
|
||||
with belief_scope("GitService.__init__"):
|
||||
self.base_path = base_path
|
||||
# Resolve relative to the backend directory
|
||||
# Path(__file__) is backend/src/services/git_service.py
|
||||
# parents[2] is backend/
|
||||
from pathlib import Path
|
||||
backend_root = Path(__file__).parents[2]
|
||||
|
||||
self.base_path = str((backend_root / base_path).resolve())
|
||||
if not os.path.exists(self.base_path):
|
||||
os.makedirs(self.base_path)
|
||||
# [/DEF:__init__:Function]
|
||||
|
||||
BIN
backend/tasks.db
BIN
backend/tasks.db
Binary file not shown.
@@ -13,7 +13,7 @@ The settings mechanism allows users to configure multiple Superset environments
|
||||
Configuration is structured using Pydantic models in `backend/src/core/config_models.py`:
|
||||
|
||||
- `Environment`: Represents a Superset instance (URL, credentials). The `base_url` is automatically normalized to include the `/api/v1` suffix if missing.
|
||||
- `GlobalSettings`: Global application parameters (e.g., `backup_path`).
|
||||
- `GlobalSettings`: Global application parameters (e.g., `storage.root_path`).
|
||||
- `AppConfig`: The root configuration object.
|
||||
|
||||
### Configuration Manager
|
||||
@@ -43,4 +43,4 @@ The settings page is located at `frontend/src/pages/Settings.svelte`. It provide
|
||||
|
||||
Existing plugins and utilities use the `ConfigManager` to fetch configuration:
|
||||
- `superset_tool/utils/init_clients.py`: Dynamically initializes Superset clients from the configured environments.
|
||||
- `BackupPlugin`: Uses the configured `backup_path` as the default storage location.
|
||||
- `BackupPlugin`: Uses the configured `storage.root_path` as the default storage location.
|
||||
|
||||
@@ -51,6 +51,7 @@
|
||||
<a href="/tools/search" class="block px-4 py-2 text-sm text-gray-700 hover:bg-blue-50 hover:text-blue-600">{$t.nav.tools_search}</a>
|
||||
<a href="/tools/mapper" class="block px-4 py-2 text-sm text-gray-700 hover:bg-blue-50 hover:text-blue-600">{$t.nav.tools_mapper}</a>
|
||||
<a href="/tools/debug" class="block px-4 py-2 text-sm text-gray-700 hover:bg-blue-50 hover:text-blue-600">{$t.nav.tools_debug}</a>
|
||||
<a href="/tools/storage" class="block px-4 py-2 text-sm text-gray-700 hover:bg-blue-50 hover:text-blue-600">{$t.nav.tools_storage}</a>
|
||||
</div>
|
||||
</div>
|
||||
<div class="relative inline-block group">
|
||||
|
||||
113
frontend/src/components/storage/FileList.svelte
Normal file
113
frontend/src/components/storage/FileList.svelte
Normal file
@@ -0,0 +1,113 @@
|
||||
<!-- [DEF:FileList:Component] -->
|
||||
<!--
|
||||
@SEMANTICS: storage, files, list, table
|
||||
@PURPOSE: Displays a table of files with metadata and actions.
|
||||
@LAYER: Component
|
||||
@RELATION: DEPENDS_ON -> storageService
|
||||
|
||||
@PROPS: files (Array) - List of StoredFile objects.
|
||||
@EVENTS: delete (filename) - Dispatched when a file is deleted.
|
||||
-->
|
||||
|
||||
<script lang="ts">
|
||||
// [SECTION: IMPORTS]
|
||||
import { createEventDispatcher } from 'svelte';
|
||||
import { downloadFileUrl } from '../../services/storageService';
|
||||
import { t } from '../../lib/i18n';
|
||||
// [/SECTION: IMPORTS]
|
||||
|
||||
export let files = [];
|
||||
const dispatch = createEventDispatcher();
|
||||
|
||||
function isDirectory(file) {
|
||||
return file.mime_type === 'directory';
|
||||
}
|
||||
|
||||
function formatSize(bytes) {
|
||||
if (bytes === 0) return '0 B';
|
||||
const k = 1024;
|
||||
const sizes = ['B', 'KB', 'MB', 'GB', 'TB'];
|
||||
const i = Math.floor(Math.log(bytes) / Math.log(k));
|
||||
return parseFloat((bytes / Math.pow(k, i)).toFixed(2)) + ' ' + sizes[i];
|
||||
}
|
||||
|
||||
function formatDate(dateStr) {
|
||||
return new Date(dateStr).toLocaleString();
|
||||
}
|
||||
</script>
|
||||
|
||||
<!-- [SECTION: TEMPLATE] -->
|
||||
<div class="overflow-x-auto">
|
||||
<table class="min-w-full bg-white border border-gray-200">
|
||||
<thead class="bg-gray-50">
|
||||
<tr>
|
||||
<th class="px-6 py-3 text-left text-xs font-medium text-gray-500 uppercase tracking-wider">{$t.storage.table.name}</th>
|
||||
<th class="px-6 py-3 text-left text-xs font-medium text-gray-500 uppercase tracking-wider">{$t.storage.table.category}</th>
|
||||
<th class="px-6 py-3 text-left text-xs font-medium text-gray-500 uppercase tracking-wider">{$t.storage.table.size}</th>
|
||||
<th class="px-6 py-3 text-left text-xs font-medium text-gray-500 uppercase tracking-wider">{$t.storage.table.created_at}</th>
|
||||
<th class="px-6 py-3 text-right text-xs font-medium text-gray-500 uppercase tracking-wider">{$t.storage.table.actions}</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody class="divide-y divide-gray-200">
|
||||
{#each files as file}
|
||||
<tr class="hover:bg-gray-50">
|
||||
<td class="px-6 py-4 whitespace-nowrap text-sm font-medium text-gray-900">
|
||||
{#if isDirectory(file)}
|
||||
<button
|
||||
on:click={() => dispatch('navigate', file.path)}
|
||||
class="flex items-center text-indigo-600 hover:text-indigo-900"
|
||||
>
|
||||
<svg class="h-5 w-5 mr-2 text-yellow-400" fill="currentColor" viewBox="0 0 20 20">
|
||||
<path d="M2 6a2 2 0 012-2h5l2 2h5a2 2 0 012 2v6a2 2 0 01-2 2H4a2 2 0 01-2-2V6z" />
|
||||
</svg>
|
||||
{file.name}
|
||||
</button>
|
||||
{:else}
|
||||
<div class="flex items-center">
|
||||
<svg class="h-5 w-5 mr-2 text-gray-400" fill="none" stroke="currentColor" viewBox="0 0 24 24">
|
||||
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M7 21h10a2 2 0 002-2V9.414a1 1 0 00-.293-.707l-5.414-5.414A1 1 0 0012.586 3H7a2 2 0 00-2 2v14a2 2 0 002 2z" />
|
||||
</svg>
|
||||
{file.name}
|
||||
</div>
|
||||
{/if}
|
||||
</td>
|
||||
<td class="px-6 py-4 whitespace-nowrap text-sm text-gray-500 capitalize">{file.category}</td>
|
||||
<td class="px-6 py-4 whitespace-nowrap text-sm text-gray-500">
|
||||
{isDirectory(file) ? '--' : formatSize(file.size)}
|
||||
</td>
|
||||
<td class="px-6 py-4 whitespace-nowrap text-sm text-gray-500">{formatDate(file.created_at)}</td>
|
||||
<td class="px-6 py-4 whitespace-nowrap text-right text-sm font-medium">
|
||||
{#if !isDirectory(file)}
|
||||
<a
|
||||
href={downloadFileUrl(file.category, file.path)}
|
||||
download={file.name}
|
||||
class="text-indigo-600 hover:text-indigo-900 mr-4"
|
||||
>
|
||||
{$t.storage.table.download}
|
||||
</a>
|
||||
{/if}
|
||||
<button
|
||||
on:click={() => dispatch('delete', { category: file.category, path: file.path, name: file.name })}
|
||||
class="text-red-600 hover:text-red-900"
|
||||
>
|
||||
{$t.storage.table.delete}
|
||||
</button>
|
||||
</td>
|
||||
</tr>
|
||||
{:else}
|
||||
<tr>
|
||||
<td colspan="5" class="px-6 py-10 text-center text-sm text-gray-500">
|
||||
{$t.storage.no_files}
|
||||
</td>
|
||||
</tr>
|
||||
{/each}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
<!-- [/SECTION: TEMPLATE] -->
|
||||
|
||||
<style>
|
||||
/* ... */
|
||||
</style>
|
||||
|
||||
<!-- [/DEF:FileList:Component] -->
|
||||
134
frontend/src/components/storage/FileUpload.svelte
Normal file
134
frontend/src/components/storage/FileUpload.svelte
Normal file
@@ -0,0 +1,134 @@
|
||||
<!-- [DEF:FileUpload:Component] -->
|
||||
<!--
|
||||
@SEMANTICS: storage, upload, files
|
||||
@PURPOSE: Provides a form for uploading files to a specific category.
|
||||
@LAYER: Component
|
||||
@RELATION: DEPENDS_ON -> storageService
|
||||
|
||||
@PROPS: None
|
||||
@EVENTS: uploaded - Dispatched when a file is successfully uploaded.
|
||||
-->
|
||||
|
||||
<script lang="ts">
|
||||
// [SECTION: IMPORTS]
|
||||
import { createEventDispatcher } from 'svelte';
|
||||
import { uploadFile } from '../../services/storageService';
|
||||
import { addToast } from '../../lib/toasts';
|
||||
import { t } from '../../lib/i18n';
|
||||
// [/SECTION: IMPORTS]
|
||||
|
||||
// [DEF:handleUpload:Function]
|
||||
/**
|
||||
* @purpose Handles the file upload process.
|
||||
* @pre A file must be selected in the file input.
|
||||
* @post The file is uploaded to the server and a success toast is shown.
|
||||
*/
|
||||
const dispatch = createEventDispatcher();
|
||||
let fileInput;
|
||||
export let category = 'backups';
|
||||
export let path = '';
|
||||
let isUploading = false;
|
||||
let dragOver = false;
|
||||
|
||||
async function handleUpload() {
|
||||
const file = fileInput.files[0];
|
||||
if (!file) return;
|
||||
|
||||
isUploading = true;
|
||||
try {
|
||||
// path is relative to root, but upload endpoint expects path within category
|
||||
// FileList.path is like "backup/folder", we need just "folder"
|
||||
const subpath = path.startsWith(category)
|
||||
? path.substring(category.length).replace(/^\/+/, '')
|
||||
: path;
|
||||
|
||||
await uploadFile(file, category, subpath);
|
||||
addToast($t.storage.messages.upload_success.replace('{name}', file.name), 'success');
|
||||
fileInput.value = '';
|
||||
dispatch('uploaded');
|
||||
} catch (error) {
|
||||
addToast($t.storage.messages.upload_failed.replace('{error}', error.message), 'error');
|
||||
} finally {
|
||||
isUploading = false;
|
||||
}
|
||||
}
|
||||
// [/DEF:handleUpload:Function]
|
||||
|
||||
// [DEF:handleDrop:Function]
|
||||
/**
|
||||
* @purpose Handles the file drop event for drag-and-drop.
|
||||
* @param {DragEvent} event - The drop event.
|
||||
*/
|
||||
function handleDrop(event) {
|
||||
event.preventDefault();
|
||||
dragOver = false;
|
||||
const files = event.dataTransfer.files;
|
||||
if (files.length > 0) {
|
||||
fileInput.files = files;
|
||||
handleUpload();
|
||||
}
|
||||
}
|
||||
// [/DEF:handleDrop:Function]
|
||||
</script>
|
||||
|
||||
<!-- [SECTION: TEMPLATE] -->
|
||||
<div class="bg-white p-6 rounded-lg border border-gray-200 shadow-sm">
|
||||
<h2 class="text-lg font-semibold mb-4">{$t.storage.upload_title}</h2>
|
||||
|
||||
<div class="space-y-4">
|
||||
<div>
|
||||
<label class="block text-sm font-medium text-gray-700 mb-1">{$t.storage.target_category}</label>
|
||||
<select
|
||||
bind:value={category}
|
||||
class="block w-full border-gray-300 rounded-md shadow-sm focus:ring-indigo-500 focus:border-indigo-500 sm:text-sm"
|
||||
>
|
||||
<option value="backups">{$t.storage.backups}</option>
|
||||
<option value="repositorys">{$t.storage.repositories}</option>
|
||||
</select>
|
||||
</div>
|
||||
|
||||
<div
|
||||
class="mt-1 flex justify-center px-6 pt-5 pb-6 border-2 border-dashed rounded-md transition-colors
|
||||
{dragOver ? 'border-indigo-500 bg-indigo-50' : 'border-gray-300'}"
|
||||
on:dragover|preventDefault={() => dragOver = true}
|
||||
on:dragleave|preventDefault={() => dragOver = false}
|
||||
on:drop|preventDefault={handleDrop}
|
||||
>
|
||||
<div class="space-y-1 text-center">
|
||||
<svg class="mx-auto h-12 w-12 text-gray-400" stroke="currentColor" fill="none" viewBox="0 0 48 48" aria-hidden="true">
|
||||
<path d="M28 8H12a4 4 0 00-4 4v20m32-12v8m0 0v8a4 4 0 01-4 4H12a4 4 0 01-4-4v-4m32-4l-3.172-3.172a4 4 0 00-5.656 0L28 28M8 32l9.172-9.172a4 4 0 015.656 0L28 28m0 0l4 4m4-24h8m-4-4v8m-12 4h.02" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" />
|
||||
</svg>
|
||||
<div class="flex text-sm text-gray-600">
|
||||
<label for="file-upload" class="relative cursor-pointer bg-white rounded-md font-medium text-indigo-600 hover:text-indigo-500 focus-within:outline-none focus-within:ring-2 focus-within:ring-offset-2 focus-within:ring-indigo-500">
|
||||
<span>{$t.storage.upload_button}</span>
|
||||
<input
|
||||
id="file-upload"
|
||||
name="file-upload"
|
||||
type="file"
|
||||
class="sr-only"
|
||||
bind:this={fileInput}
|
||||
on:change={handleUpload}
|
||||
disabled={isUploading}
|
||||
>
|
||||
</label>
|
||||
<p class="pl-1">{$t.storage.drag_drop}</p>
|
||||
</div>
|
||||
<p class="text-xs text-gray-500">{$t.storage.supported_formats}</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{#if isUploading}
|
||||
<div class="flex items-center justify-center space-x-2 text-indigo-600">
|
||||
<div class="animate-spin rounded-full h-4 w-4 border-b-2 border-indigo-600"></div>
|
||||
<span class="text-sm font-medium">{$t.storage.uploading}</span>
|
||||
</div>
|
||||
{/if}
|
||||
</div>
|
||||
</div>
|
||||
<!-- [/SECTION: TEMPLATE] -->
|
||||
|
||||
<style>
|
||||
/* ... */
|
||||
</style>
|
||||
|
||||
<!-- [/DEF:FileUpload:Component] -->
|
||||
@@ -123,6 +123,8 @@ export const api = {
|
||||
deleteEnvironment: (id) => requestApi(`/settings/environments/${id}`, 'DELETE'),
|
||||
testEnvironmentConnection: (id) => postApi(`/settings/environments/${id}/test`, {}),
|
||||
updateEnvironmentSchedule: (id, schedule) => requestApi(`/environments/${id}/schedule`, 'PUT', schedule),
|
||||
getStorageSettings: () => fetchApi('/settings/storage'),
|
||||
updateStorageSettings: (storage) => requestApi('/settings/storage', 'PUT', storage),
|
||||
getEnvironmentsList: () => fetchApi('/environments'),
|
||||
};
|
||||
// [/DEF:api:Data]
|
||||
@@ -143,3 +145,5 @@ export const deleteEnvironment = api.deleteEnvironment;
|
||||
export const testEnvironmentConnection = api.testEnvironmentConnection;
|
||||
export const updateEnvironmentSchedule = api.updateEnvironmentSchedule;
|
||||
export const getEnvironmentsList = api.getEnvironmentsList;
|
||||
export const getStorageSettings = api.getStorageSettings;
|
||||
export const updateStorageSettings = api.updateStorageSettings;
|
||||
|
||||
@@ -20,7 +20,6 @@
|
||||
let settings = {
|
||||
environments: [],
|
||||
settings: {
|
||||
backup_path: '',
|
||||
default_environment_id: null,
|
||||
logging: {
|
||||
level: 'INFO',
|
||||
@@ -204,12 +203,6 @@
|
||||
|
||||
<section class="mb-8 bg-white p-6 rounded shadow">
|
||||
<h2 class="text-xl font-semibold mb-4">Global Settings</h2>
|
||||
<div class="grid grid-cols-1 gap-4">
|
||||
<div>
|
||||
<label for="backup_path" class="block text-sm font-medium text-gray-700">Backup Storage Path</label>
|
||||
<input type="text" id="backup_path" bind:value={settings.settings.backup_path} class="mt-1 block w-full border border-gray-300 rounded-md shadow-sm p-2" />
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<h3 class="text-lg font-medium mb-4 mt-6">Logging Configuration</h3>
|
||||
<div class="grid grid-cols-1 md:grid-cols-2 gap-4">
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
<script>
|
||||
import { onMount } from 'svelte';
|
||||
import { updateGlobalSettings, addEnvironment, updateEnvironment, deleteEnvironment, testEnvironmentConnection } from '../../lib/api';
|
||||
import { updateGlobalSettings, addEnvironment, updateEnvironment, deleteEnvironment, testEnvironmentConnection, updateStorageSettings } from '../../lib/api';
|
||||
import { addToast } from '../../lib/toasts';
|
||||
import { t } from '$lib/i18n';
|
||||
import { Button, Input, Card, PageHeader } from '$lib/ui';
|
||||
@@ -41,6 +41,24 @@
|
||||
}
|
||||
// [/DEF:handleSaveGlobal:Function]
|
||||
|
||||
// [DEF:handleSaveStorage:Function]
|
||||
/* @PURPOSE: Saves storage-specific settings.
|
||||
@PRE: settings.settings.storage must contain valid configuration.
|
||||
@POST: Storage settings are updated via API.
|
||||
*/
|
||||
async function handleSaveStorage() {
|
||||
try {
|
||||
console.log("[Settings.handleSaveStorage][Action] Saving storage settings.");
|
||||
await updateStorageSettings(settings.settings.storage);
|
||||
addToast('Storage settings saved', 'success');
|
||||
console.log("[Settings.handleSaveStorage][Coherence:OK] Storage settings saved.");
|
||||
} catch (error) {
|
||||
console.error("[Settings.handleSaveStorage][Coherence:Failed] Failed to save storage settings:", error);
|
||||
addToast(error.message || 'Failed to save storage settings', 'error');
|
||||
}
|
||||
}
|
||||
// [/DEF:handleSaveStorage:Function]
|
||||
|
||||
// [DEF:handleAddOrUpdateEnv:Function]
|
||||
/* @PURPOSE: Adds a new environment or updates an existing one.
|
||||
@PRE: newEnv must contain valid environment details.
|
||||
@@ -152,14 +170,37 @@
|
||||
</div>
|
||||
{/if}
|
||||
|
||||
|
||||
<div class="mb-8">
|
||||
<Card title={$t.settings?.global_title || "Global Settings"}>
|
||||
<div class="grid grid-cols-1 gap-6">
|
||||
<Card title={$t.settings?.storage_title || "File Storage Configuration"}>
|
||||
<div class="grid grid-cols-1 md:grid-cols-2 gap-6">
|
||||
<div class="md:col-span-2">
|
||||
<Input
|
||||
label={$t.settings?.storage_root || "Storage Root Path"}
|
||||
bind:value={settings.settings.storage.root_path}
|
||||
/>
|
||||
</div>
|
||||
<Input
|
||||
label={$t.settings?.backup_path || "Backup Storage Path"}
|
||||
bind:value={settings.settings.backup_path}
|
||||
label={$t.settings?.storage_backup_pattern || "Backup Directory Pattern"}
|
||||
bind:value={settings.settings.storage.backup_structure_pattern}
|
||||
/>
|
||||
<Button on:click={handleSaveGlobal}>
|
||||
<Input
|
||||
label={$t.settings?.storage_repo_pattern || "Repository Directory Pattern"}
|
||||
bind:value={settings.settings.storage.repo_structure_pattern}
|
||||
/>
|
||||
<Input
|
||||
label={$t.settings?.storage_filename_pattern || "Filename Pattern"}
|
||||
bind:value={settings.settings.storage.filename_pattern}
|
||||
/>
|
||||
<div class="bg-gray-50 p-4 rounded border border-gray-200">
|
||||
<span class="block text-xs font-semibold text-gray-500 uppercase mb-2">{$t.settings?.storage_preview || "Path Preview"}</span>
|
||||
<code class="text-sm text-indigo-600">
|
||||
{settings.settings.storage.root_path}/backups/sample_backup.zip
|
||||
</code>
|
||||
</div>
|
||||
</div>
|
||||
<div class="mt-6">
|
||||
<Button on:click={handleSaveStorage}>
|
||||
{$t.common.save}
|
||||
</Button>
|
||||
</div>
|
||||
|
||||
@@ -18,7 +18,6 @@ export async function load() {
|
||||
settings: {
|
||||
environments: [],
|
||||
settings: {
|
||||
backup_path: '',
|
||||
default_environment_id: null
|
||||
}
|
||||
},
|
||||
|
||||
187
frontend/src/routes/tools/storage/+page.svelte
Normal file
187
frontend/src/routes/tools/storage/+page.svelte
Normal file
@@ -0,0 +1,187 @@
|
||||
<!-- [DEF:StoragePage:Component] -->
|
||||
<!--
|
||||
@SEMANTICS: storage, files, management
|
||||
@PURPOSE: Main page for file storage management.
|
||||
@LAYER: Feature
|
||||
@RELATION: DEPENDS_ON -> storageService
|
||||
@RELATION: CONTAINS -> FileList
|
||||
@RELATION: CONTAINS -> FileUpload
|
||||
|
||||
@INVARIANT: Always displays tabs for Backups and Repositories.
|
||||
-->
|
||||
|
||||
<script lang="ts">
|
||||
// [SECTION: IMPORTS]
|
||||
import { onMount } from 'svelte';
|
||||
import { listFiles, deleteFile } from '../../../services/storageService';
|
||||
import { addToast } from '../../../lib/toasts';
|
||||
import { t } from '../../../lib/i18n';
|
||||
import FileList from '../../../components/storage/FileList.svelte';
|
||||
import FileUpload from '../../../components/storage/FileUpload.svelte';
|
||||
// [/SECTION: IMPORTS]
|
||||
|
||||
// [DEF:loadFiles:Function]
|
||||
/**
|
||||
* @purpose Fetches the list of files from the server.
|
||||
* @post Updates the `files` array with the latest data.
|
||||
*/
|
||||
let files = [];
|
||||
let isLoading = false;
|
||||
let activeTab = 'backups';
|
||||
let currentPath = 'backups'; // Relative to storage root
|
||||
|
||||
async function loadFiles() {
|
||||
isLoading = true;
|
||||
try {
|
||||
const category = activeTab;
|
||||
|
||||
// If we have a currentPath, we use it.
|
||||
// But if user switched tabs, we should reset currentPath to category root
|
||||
let effectivePath = currentPath;
|
||||
if (category && !currentPath.startsWith(category)) {
|
||||
effectivePath = category;
|
||||
currentPath = category;
|
||||
}
|
||||
|
||||
// API expects path relative to category root if category is provided
|
||||
const subpath = (category && effectivePath.startsWith(category))
|
||||
? effectivePath.substring(category.length).replace(/^\/+/, '')
|
||||
: effectivePath;
|
||||
|
||||
files = await listFiles(category, subpath);
|
||||
} catch (error) {
|
||||
addToast($t.storage.messages.load_failed.replace('{error}', error.message), 'error');
|
||||
} finally {
|
||||
isLoading = false;
|
||||
}
|
||||
}
|
||||
// [/DEF:loadFiles:Function]
|
||||
|
||||
// [DEF:handleDelete:Function]
|
||||
/**
|
||||
* @purpose Handles the file deletion process.
|
||||
* @param {CustomEvent} event - The delete event containing category and path.
|
||||
*/
|
||||
async function handleDelete(event) {
|
||||
const { category, path, name } = event.detail;
|
||||
if (!confirm($t.storage.messages.delete_confirm.replace('{name}', name))) return;
|
||||
|
||||
try {
|
||||
await deleteFile(category, path);
|
||||
addToast($t.storage.messages.delete_success.replace('{name}', name), 'success');
|
||||
await loadFiles();
|
||||
} catch (error) {
|
||||
addToast($t.storage.messages.delete_failed.replace('{error}', error.message), 'error');
|
||||
}
|
||||
}
|
||||
// [/DEF:handleDelete:Function]
|
||||
|
||||
function handleNavigate(event) {
|
||||
currentPath = event.detail;
|
||||
loadFiles();
|
||||
}
|
||||
|
||||
function navigateUp() {
|
||||
if (!currentPath || currentPath === activeTab) return;
|
||||
const parts = currentPath.split('/');
|
||||
parts.pop();
|
||||
currentPath = parts.join('/') || '';
|
||||
loadFiles();
|
||||
}
|
||||
|
||||
onMount(loadFiles);
|
||||
|
||||
$: if (activeTab) {
|
||||
// Reset path when switching tabs
|
||||
if (!currentPath.startsWith(activeTab)) {
|
||||
currentPath = activeTab;
|
||||
}
|
||||
loadFiles();
|
||||
}
|
||||
</script>
|
||||
|
||||
<!-- [SECTION: TEMPLATE] -->
|
||||
<div class="container mx-auto p-4 max-w-6xl">
|
||||
<div class="mb-6">
|
||||
<h1 class="text-2xl font-bold text-gray-900">{$t.storage.management}</h1>
|
||||
{#if currentPath}
|
||||
<div class="flex items-center mt-2 text-sm text-gray-500">
|
||||
<button on:click={() => { currentPath = activeTab; loadFiles(); }} class="hover:text-indigo-600">{$t.storage.root}</button>
|
||||
{#each currentPath.split('/').slice(1) as part, i}
|
||||
<span class="mx-2">/</span>
|
||||
<button
|
||||
on:click={() => { currentPath = currentPath.split('/').slice(0, i + 1).join('/'); loadFiles(); }}
|
||||
class="hover:text-indigo-600 capitalize"
|
||||
>
|
||||
{part}
|
||||
</button>
|
||||
{/each}
|
||||
</div>
|
||||
{/if}
|
||||
</div>
|
||||
|
||||
<div class="flex justify-end mb-4">
|
||||
<button
|
||||
on:click={loadFiles}
|
||||
class="inline-flex items-center px-4 py-2 border border-gray-300 rounded-md shadow-sm text-sm font-medium text-gray-700 bg-white hover:bg-gray-50"
|
||||
disabled={isLoading}
|
||||
>
|
||||
{isLoading ? $t.storage.refreshing : $t.storage.refresh}
|
||||
</button>
|
||||
</div>
|
||||
|
||||
<div class="grid grid-cols-1 lg:grid-cols-3 gap-6">
|
||||
<!-- Main Content: File List -->
|
||||
<div class="lg:col-span-2 space-y-4">
|
||||
<!-- Tabs -->
|
||||
<div class="border-b border-gray-200">
|
||||
<nav class="-mb-px flex space-x-8">
|
||||
<button
|
||||
on:click={() => activeTab = 'backups'}
|
||||
class="py-4 px-1 border-b-2 font-medium text-sm {activeTab === 'backups' ? 'border-indigo-500 text-indigo-600' : 'border-transparent text-gray-500 hover:text-gray-700 hover:border-gray-300'}"
|
||||
>
|
||||
{$t.storage.backups}
|
||||
</button>
|
||||
<button
|
||||
on:click={() => activeTab = 'repositorys'}
|
||||
class="py-4 px-1 border-b-2 font-medium text-sm {activeTab === 'repositorys' ? 'border-indigo-500 text-indigo-600' : 'border-transparent text-gray-500 hover:text-gray-700 hover:border-gray-300'}"
|
||||
>
|
||||
{$t.storage.repositories}
|
||||
</button>
|
||||
</nav>
|
||||
</div>
|
||||
|
||||
<div class="flex items-center mb-2">
|
||||
{#if currentPath && currentPath !== activeTab}
|
||||
<button
|
||||
on:click={navigateUp}
|
||||
class="mr-4 inline-flex items-center px-3 py-1 border border-gray-300 shadow-sm text-xs font-medium rounded text-gray-700 bg-white hover:bg-gray-50"
|
||||
>
|
||||
<svg class="h-4 w-4 mr-1" fill="none" stroke="currentColor" viewBox="0 0 24 24">
|
||||
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M10 19l-7-7m0 0l7-7m-7 7h18" />
|
||||
</svg>
|
||||
Back
|
||||
</button>
|
||||
{/if}
|
||||
</div>
|
||||
|
||||
<FileList {files} on:delete={handleDelete} on:navigate={handleNavigate} />
|
||||
</div>
|
||||
|
||||
<!-- Sidebar: Upload -->
|
||||
<div class="lg:col-span-1">
|
||||
<FileUpload
|
||||
category={activeTab}
|
||||
path={currentPath}
|
||||
on:uploaded={loadFiles}
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<!-- [/SECTION: TEMPLATE] -->
|
||||
|
||||
<style>
|
||||
/* ... */
|
||||
</style>
|
||||
|
||||
<!-- [/DEF:StoragePage:Component] -->
|
||||
100
frontend/src/services/storageService.js
Normal file
100
frontend/src/services/storageService.js
Normal file
@@ -0,0 +1,100 @@
|
||||
// [DEF:storageService:Module]
|
||||
/**
|
||||
* @purpose Frontend API client for file storage management.
|
||||
* @layer Service
|
||||
* @relation DEPENDS_ON -> backend.api.storage
|
||||
*/
|
||||
|
||||
const API_BASE = '/api/storage';
|
||||
|
||||
// [DEF:listFiles:Function]
|
||||
/**
|
||||
* @purpose Fetches the list of files for a given category and subpath.
|
||||
* @param {string} [category] - Optional category filter.
|
||||
* @param {string} [path] - Optional subpath filter.
|
||||
* @returns {Promise<Array>}
|
||||
*/
|
||||
export async function listFiles(category, path) {
|
||||
const params = new URLSearchParams();
|
||||
if (category) {
|
||||
params.append('category', category);
|
||||
}
|
||||
if (path) {
|
||||
params.append('path', path);
|
||||
}
|
||||
const response = await fetch(`${API_BASE}/files?${params.toString()}`);
|
||||
if (!response.ok) {
|
||||
throw new Error(`Failed to fetch files: ${response.statusText}`);
|
||||
}
|
||||
return await response.json();
|
||||
}
|
||||
// [/DEF:listFiles:Function]
|
||||
|
||||
// [DEF:uploadFile:Function]
|
||||
/**
|
||||
* @purpose Uploads a file to the storage system.
|
||||
* @param {File} file - The file to upload.
|
||||
* @param {string} category - Target category.
|
||||
* @param {string} [path] - Target subpath.
|
||||
* @returns {Promise<Object>}
|
||||
*/
|
||||
export async function uploadFile(file, category, path) {
|
||||
const formData = new FormData();
|
||||
formData.append('file', file);
|
||||
formData.append('category', category);
|
||||
if (path) {
|
||||
formData.append('path', path);
|
||||
}
|
||||
|
||||
const response = await fetch(`${API_BASE}/upload`, {
|
||||
method: 'POST',
|
||||
body: formData
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
const errorData = await response.json().catch(() => ({}));
|
||||
throw new Error(errorData.detail || `Failed to upload file: ${response.statusText}`);
|
||||
}
|
||||
return await response.json();
|
||||
}
|
||||
// [/DEF:uploadFile:Function]
|
||||
|
||||
// [DEF:deleteFile:Function]
|
||||
/**
|
||||
* @purpose Deletes a file or directory from storage.
|
||||
* @param {string} category - File category.
|
||||
* @param {string} path - Relative path of the item.
|
||||
* @returns {Promise<void>}
|
||||
*/
|
||||
export async function deleteFile(category, path) {
|
||||
const response = await fetch(`${API_BASE}/files/${category}/${path}`, {
|
||||
method: 'DELETE'
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
const errorData = await response.json().catch(() => ({}));
|
||||
throw new Error(errorData.detail || `Failed to delete: ${response.statusText}`);
|
||||
}
|
||||
}
|
||||
// [/DEF:deleteFile:Function]
|
||||
|
||||
// [DEF:downloadFileUrl:Function]
|
||||
/**
|
||||
* @purpose Returns the URL for downloading a file.
|
||||
* @param {string} category - File category.
|
||||
* @param {string} path - Relative path of the file.
|
||||
* @returns {string}
|
||||
*/
|
||||
export function downloadFileUrl(category, path) {
|
||||
return `${API_BASE}/download/${category}/${path}`;
|
||||
}
|
||||
// [/DEF:downloadFileUrl:Function]
|
||||
|
||||
export default {
|
||||
listFiles,
|
||||
uploadFile,
|
||||
deleteFile,
|
||||
downloadFileUrl
|
||||
};
|
||||
|
||||
// [/DEF:storageService:Module]
|
||||
@@ -15,7 +15,7 @@
|
||||
## Phase 3: [US1] Scheduled Backups
|
||||
- [x] T009 [US1] Implement schedule loading and registration logic in `SchedulerService`
|
||||
- [x] T010 [US1] Update `Environment` settings API to handle `backup_schedule` updates in `backend/src/api/routes/environments.py`
|
||||
- [x] T011 [P] [US1] Add schedule configuration fields to Environment edit form in `frontend/src/components/EnvSelector.svelte` (or appropriate component)
|
||||
- [ ] T011 [P] [US1] Add schedule configuration fields to Environment edit form in `frontend/src/components/EnvSelector.svelte` (or appropriate component)
|
||||
- [x] T012 [US1] Implement validation for Cron expressions in backend and frontend
|
||||
|
||||
## Phase 4: [US2] Unified Task Management UI
|
||||
@@ -34,7 +34,7 @@
|
||||
- [x] T021 [US4] Integrate log viewer into TaskList or as a separate modal/page
|
||||
|
||||
## Final Phase: Polish & Cross-cutting concerns
|
||||
- [x] T022 Implement task cleanup/retention policy (e.g., delete tasks older than 30 days)
|
||||
- [ ] T022 Implement task cleanup/retention policy (e.g., delete tasks older than 30 days)
|
||||
- [ ] T023 Add real-time updates for task status using WebSockets (optional/refinement)
|
||||
- [x] T024 Ensure consistent error handling and logging across scheduler and task manager
|
||||
|
||||
|
||||
34
specs/014-file-storage-ui/checklists/requirements.md
Normal file
34
specs/014-file-storage-ui/checklists/requirements.md
Normal file
@@ -0,0 +1,34 @@
|
||||
# Specification Quality Checklist: File Storage Management & UI
|
||||
|
||||
**Purpose**: Validate specification completeness and quality before proceeding to planning
|
||||
**Created**: 2026-01-24
|
||||
**Feature**: [Link to spec.md](../spec.md)
|
||||
|
||||
## Content Quality
|
||||
|
||||
- [x] No implementation details (languages, frameworks, APIs)
|
||||
- [x] Focused on user value and business needs
|
||||
- [x] Written for non-technical stakeholders
|
||||
- [x] All mandatory sections completed
|
||||
|
||||
## Requirement Completeness
|
||||
|
||||
- [x] No [NEEDS CLARIFICATION] markers remain
|
||||
- [x] Requirements are testable and unambiguous
|
||||
- [x] Success criteria are measurable
|
||||
- [x] Success criteria are technology-agnostic (no implementation details)
|
||||
- [x] All acceptance scenarios are defined
|
||||
- [x] Edge cases are identified
|
||||
- [x] Scope is clearly bounded
|
||||
- [x] Dependencies and assumptions identified
|
||||
|
||||
## Feature Readiness
|
||||
|
||||
- [x] All functional requirements have clear acceptance criteria
|
||||
- [x] User scenarios cover primary flows
|
||||
- [x] Feature meets measurable outcomes defined in Success Criteria
|
||||
- [x] No implementation details leak into specification
|
||||
|
||||
## Notes
|
||||
|
||||
- Items marked incomplete require spec updates before `/speckit.clarify` or `/speckit.plan`
|
||||
35
specs/014-file-storage-ui/checklists/ux.md
Normal file
35
specs/014-file-storage-ui/checklists/ux.md
Normal file
@@ -0,0 +1,35 @@
|
||||
# Checklist: File Storage UX & Configuration
|
||||
|
||||
**Purpose**: Validate implementation of User Experience and Configuration Flexibility requirements.
|
||||
**Created**: 2026-01-24
|
||||
**Feature**: [File Storage Management & UI](../spec.md)
|
||||
|
||||
## User Experience (File Management)
|
||||
|
||||
- [x] CHK001 Are loading states displayed while fetching the file list? [Completeness]
|
||||
- [x] CHK002 Is visual feedback provided immediately after file upload starts? [Clarity]
|
||||
- [x] CHK003 Are error messages user-friendly when upload fails (e.g., "File too large" vs "Error 413")? [Clarity]
|
||||
- [x] CHK004 Is a confirmation modal shown before permanently deleting a file? [Safety]
|
||||
- [x] CHK005 Does the UI clearly distinguish between "Backups" and "Repositories" tabs? [Clarity]
|
||||
- [x] CHK006 Is the file list sortable by Date and Name? [Usability]
|
||||
- [x] CHK007 Are file sizes formatted in human-readable units (KB, MB, GB)? [Usability]
|
||||
- [x] CHK008 Is the download action easily accessible for each file item? [Accessibility]
|
||||
- [x] CHK009 Does the upload component support drag-and-drop interactions? [Usability]
|
||||
- [x] CHK010 Is the "Upload" button disabled or hidden when no category is selected? [Consistency]
|
||||
|
||||
## Configuration Flexibility
|
||||
|
||||
- [x] CHK011 Can the storage root path be configured to any writable directory on the server? [Flexibility]
|
||||
- [x] CHK012 Does the system support defining custom directory structures using variables like `{environment}`? [Flexibility]
|
||||
- [x] CHK013 Does the system support defining custom filename patterns using variables like `{timestamp}`? [Flexibility]
|
||||
- [ ] CHK014 Are the supported pattern variables (e.g., `{dashboard_name}`) clearly documented in the UI? [Clarity]
|
||||
- [x] CHK015 Can the configuration be updated without restarting the application? [Usability]
|
||||
- [ ] CHK016 Is the current resolved path shown as a preview when editing patterns? [Usability]
|
||||
- [ ] CHK017 Does the system allow reverting configuration to default values? [Recovery]
|
||||
|
||||
## Edge Cases & Error Handling
|
||||
|
||||
- [x] CHK018 Is the UI behavior defined for an empty file list (zero state)? [Coverage]
|
||||
- [ ] CHK019 Is the behavior defined when the configured storage path becomes inaccessible? [Resilience]
|
||||
- [x] CHK020 Are long filenames handled gracefully in the UI (e.g., truncation with tooltip)? [Layout]
|
||||
- [x] CHK021 Is the behavior defined for uploading a file with a duplicate name? [Conflict Resolution]
|
||||
74
specs/014-file-storage-ui/contracts/api.md
Normal file
74
specs/014-file-storage-ui/contracts/api.md
Normal file
@@ -0,0 +1,74 @@
|
||||
# API Contracts: File Storage Management & UI
|
||||
|
||||
## Endpoints
|
||||
|
||||
### GET /api/storage/files
|
||||
List all files in the storage system.
|
||||
|
||||
**Query Parameters:**
|
||||
- `category` (optional): Filter by category (`backup` or `repository`).
|
||||
|
||||
**Response:**
|
||||
- `200 OK`: List of `StoredFile` objects.
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"name": "dashboard_backup_20260124.zip",
|
||||
"path": "backups/dashboard_backup_20260124.zip",
|
||||
"size": 102400,
|
||||
"created_at": "2026-01-24T12:00:00Z",
|
||||
"category": "backup",
|
||||
"mime_type": "application/zip"
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
### POST /api/storage/upload
|
||||
Upload a file to the storage system.
|
||||
|
||||
**Form Data:**
|
||||
- `file`: The file content.
|
||||
- `category`: Target category (`backup` or `repository`).
|
||||
|
||||
**Response:**
|
||||
- `201 Created`: The uploaded `StoredFile` object.
|
||||
- `400 Bad Request`: Invalid category or file.
|
||||
|
||||
### DELETE /api/storage/files/{category}/{filename}
|
||||
Delete a file from storage.
|
||||
|
||||
**Path Parameters:**
|
||||
- `category`: `backup` or `repository`.
|
||||
- `filename`: Name of the file to delete.
|
||||
|
||||
**Response:**
|
||||
- `204 No Content`: File deleted successfully.
|
||||
- `404 Not Found`: File does not exist.
|
||||
|
||||
### GET /api/storage/download/{category}/{filename}
|
||||
Download a file.
|
||||
|
||||
**Path Parameters:**
|
||||
- `category`: `backup` or `repository`.
|
||||
- `filename`: Name of the file to download.
|
||||
|
||||
**Response:**
|
||||
- `200 OK`: File stream.
|
||||
- `404 Not Found`: File does not exist.
|
||||
|
||||
### GET /api/settings/storage
|
||||
Get current storage configuration.
|
||||
|
||||
**Response:**
|
||||
- `200 OK`: `StorageConfig` object.
|
||||
|
||||
### PUT /api/settings/storage
|
||||
Update storage configuration.
|
||||
|
||||
**Body:**
|
||||
- `StorageConfig` object.
|
||||
|
||||
**Response:**
|
||||
- `200 OK`: Updated `StorageConfig`.
|
||||
- `400 Bad Request`: Invalid path or not writable.
|
||||
34
specs/014-file-storage-ui/data-model.md
Normal file
34
specs/014-file-storage-ui/data-model.md
Normal file
@@ -0,0 +1,34 @@
|
||||
# Data Model: File Storage Management & UI
|
||||
|
||||
## Entities
|
||||
|
||||
### StorageConfig
|
||||
*Configuration for the storage system.*
|
||||
|
||||
| Field | Type | Description | Constraints |
|
||||
|---|---|---|---|
|
||||
| `root_path` | `string` | Absolute path to the storage root directory. | Must be a valid, writable path. Default: `../ss-tools-storage` |
|
||||
| `backup_structure_pattern` | `string` | Pattern for backup directory structure. | Default: `{category}/` |
|
||||
| `repo_structure_pattern` | `string` | Pattern for repository directory structure. | Default: `{category}/` |
|
||||
| `filename_pattern` | `string` | Pattern for filenames. | Default: `{name}_{timestamp}` |
|
||||
|
||||
### StoredFile
|
||||
*Representation of a file in the storage system.*
|
||||
|
||||
| Field | Type | Description | Constraints |
|
||||
|---|---|---|---|
|
||||
| `name` | `string` | Name of the file (including extension). | No path separators. |
|
||||
| `path` | `string` | Relative path from storage root. | |
|
||||
| `size` | `integer` | Size of the file in bytes. | >= 0 |
|
||||
| `created_at` | `datetime` | Creation timestamp. | |
|
||||
| `category` | `enum` | Category of the file. | `backup`, `repository` |
|
||||
| `mime_type` | `string` | MIME type of the file. | Optional |
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```text
|
||||
{root_path}/
|
||||
├── backups/
|
||||
│ └── {filename}.zip
|
||||
└── repositories/
|
||||
└── {filename}.zip
|
||||
106
specs/014-file-storage-ui/plan.md
Normal file
106
specs/014-file-storage-ui/plan.md
Normal file
@@ -0,0 +1,106 @@
|
||||
# Implementation Plan: File Storage Management & UI
|
||||
|
||||
**Branch**: `014-file-storage-ui` | **Date**: 2026-01-24 | **Spec**: [specs/014-file-storage-ui/spec.md](../014-file-storage-ui/spec.md)
|
||||
**Input**: Feature specification from `specs/014-file-storage-ui/spec.md`
|
||||
|
||||
## Summary
|
||||
|
||||
This feature implements a managed file storage system for dashboard backups and exported repositories. It introduces a configurable storage root (defaulting to outside the workspace) and a Web UI to list, upload, download, and delete files. The system enforces a structured layout with `backups/` and `repositories/` subdirectories to keep artifacts organized. The UI supports hierarchical folder navigation (e.g., `backups/SS2/DashboardName`), allowing users to browse, download, and manage files within nested directories.
|
||||
|
||||
## Technical Context
|
||||
|
||||
**Language/Version**: Python 3.9+ (Backend), Node.js 18+ (Frontend)
|
||||
**Primary Dependencies**: FastAPI (Backend), SvelteKit (Frontend)
|
||||
**Storage**: Local Filesystem (for artifacts), Config (for storage path)
|
||||
**Testing**: pytest (Backend), vitest/playwright (Frontend - implied)
|
||||
**Target Platform**: Linux server
|
||||
**Project Type**: Web application
|
||||
**Performance Goals**: File list load < 1s for 100 files, supports 50MB+ uploads
|
||||
**Constraints**: Must prevent path traversal, must not pollute git repo
|
||||
**Scale/Scope**: ~2-3 backend endpoints, 1-2 frontend pages/components
|
||||
|
||||
## Constitution Check
|
||||
|
||||
*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.*
|
||||
|
||||
- **I. Semantic Protocol Compliance**:
|
||||
- **Status**: PASSED
|
||||
- **Check**: Will use `[DEF]` anchors and `@RELATION` tags.
|
||||
- **Check**: Will follow File Structure Standard.
|
||||
|
||||
- **II. Causal Validity (Contracts First)**:
|
||||
- **Status**: PASSED
|
||||
- **Check**: Contracts will be defined in `specs/014-file-storage-ui/contracts/` before implementation.
|
||||
|
||||
- **III. Immutability of Architecture**:
|
||||
- **Status**: PASSED
|
||||
- **Check**: No changes to immutable architectural constraints expected.
|
||||
|
||||
- **IV. Design by Contract (DbC)**:
|
||||
- **Status**: PASSED
|
||||
- **Check**: Functions will define `@PRE` and `@POST` conditions.
|
||||
|
||||
- **V. Belief State Logging**:
|
||||
- **Status**: PASSED
|
||||
- **Check**: Will use standard logging patterns.
|
||||
|
||||
- **VI. Fractal Complexity Limit**:
|
||||
- **Status**: PASSED
|
||||
- **Check**: Feature scope is small, unlikely to exceed complexity limits.
|
||||
|
||||
- **VII. Everything is a Plugin**:
|
||||
- **Status**: PASSED
|
||||
- **Check**: New functionality will be implemented as a `StoragePlugin` (or similar) inheriting from `PluginBase`.
|
||||
|
||||
## Project Structure
|
||||
|
||||
### Documentation (this feature)
|
||||
|
||||
```text
|
||||
specs/014-file-storage-ui/
|
||||
├── plan.md # This file
|
||||
├── research.md # Phase 0 output
|
||||
├── data-model.md # Phase 1 output
|
||||
├── quickstart.md # Phase 1 output
|
||||
├── contracts/ # Phase 1 output
|
||||
└── tasks.md # Phase 2 output
|
||||
```
|
||||
|
||||
### Source Code (repository root)
|
||||
|
||||
```text
|
||||
backend/
|
||||
├── src/
|
||||
│ ├── api/
|
||||
│ │ └── routes/
|
||||
│ │ └── storage.py # New route handler
|
||||
│ ├── plugins/
|
||||
│ │ └── storage.py # New plugin implementation
|
||||
│ └── models/
|
||||
│ └── storage.py # Pydantic models (StoredFile, StorageConfig)
|
||||
└── tests/
|
||||
└── test_storage.py # Backend tests
|
||||
|
||||
frontend/
|
||||
├── src/
|
||||
│ ├── routes/
|
||||
│ │ └── storage/
|
||||
│ │ └── +page.svelte # Main storage UI
|
||||
│ ├── components/
|
||||
│ │ └── storage/
|
||||
│ │ ├── FileList.svelte # Component for listing files and folders (explorer view)
|
||||
│ │ ├── Breadcrumbs.svelte # Component for navigation
|
||||
│ │ └── FileUpload.svelte # Component for uploading
|
||||
│ └── services/
|
||||
│ └── storageService.js # Frontend API client
|
||||
```
|
||||
|
||||
**Structure Decision**: Standard Web Application structure (Backend + Frontend) with Plugin architecture for the backend logic.
|
||||
|
||||
## Complexity Tracking
|
||||
|
||||
> **Fill ONLY if Constitution Check has violations that must be justified**
|
||||
|
||||
| Violation | Why Needed | Simpler Alternative Rejected Because |
|
||||
|-----------|------------|-------------------------------------|
|
||||
| N/A | | |
|
||||
31
specs/014-file-storage-ui/quickstart.md
Normal file
31
specs/014-file-storage-ui/quickstart.md
Normal file
@@ -0,0 +1,31 @@
|
||||
# Quickstart: File Storage Management & UI
|
||||
|
||||
## Usage Guide
|
||||
|
||||
1. **Access File Storage**: Navigate to the "Tools" > "File Storage" section in the main navigation.
|
||||
2. **View Files**: You will see two tabs: "Backups" and "Repositories". Click on a tab to view files in that category.
|
||||
3. **Upload File**:
|
||||
- Click the "Upload" button.
|
||||
- Select a file from your computer.
|
||||
- Choose the target category (Backup or Repository).
|
||||
- Click "Upload" to start the transfer.
|
||||
4. **Download File**: Click the "Download" icon next to any file in the list.
|
||||
5. **Delete File**: Click the "Trash" icon next to any file to delete it permanently.
|
||||
6. **Configure Storage Path**:
|
||||
- Go to "Settings".
|
||||
- Locate the "File Storage" section.
|
||||
- Enter a new absolute path for the storage root.
|
||||
- Click "Save". The system will verify write access to the new path.
|
||||
|
||||
## Development
|
||||
|
||||
### Backend
|
||||
|
||||
- **Plugin**: `backend/src/plugins/storage.py`
|
||||
- **API Routes**: `backend/src/api/routes/storage.py`
|
||||
- **Models**: `backend/src/models/storage.py`
|
||||
|
||||
### Frontend
|
||||
|
||||
- **Page**: `frontend/src/routes/tools/storage/+page.svelte`
|
||||
- **Components**: `frontend/src/components/storage/*`
|
||||
17
specs/014-file-storage-ui/research.md
Normal file
17
specs/014-file-storage-ui/research.md
Normal file
@@ -0,0 +1,17 @@
|
||||
# Research: File Storage Management & UI
|
||||
|
||||
**Decision**: Use Python's built-in `pathlib` and `shutil` for filesystem operations.
|
||||
**Rationale**: Standard library, robust, cross-platform (though target is Linux), and sufficient for local file management. No external dependencies needed.
|
||||
**Alternatives considered**: `os` module (lower level, less ergonomic), `pyfilesystem2` (external dependency, unnecessary overhead for simple local storage).
|
||||
|
||||
**Decision**: Use `pydantic` for configuration and file metadata models.
|
||||
**Rationale**: Already used in the project, provides validation and serialization.
|
||||
|
||||
**Decision**: Use `multipart/form-data` for file uploads.
|
||||
**Rationale**: Standard web practice for file uploads. FastAPI supports `UploadFile` natively.
|
||||
|
||||
**Decision**: Default storage path strategy.
|
||||
**Rationale**: The default path will be `../ss-tools-storage` (relative to workspace root) or similar to ensure it sits outside the git repository by default, but allows configuration override.
|
||||
|
||||
**Decision**: Path Traversal Prevention.
|
||||
**Rationale**: Will use `os.path.commonpath` or `pathlib.Path.resolve()` to strictly validate that any accessed file path is a child of the configured storage root.
|
||||
97
specs/014-file-storage-ui/spec.md
Normal file
97
specs/014-file-storage-ui/spec.md
Normal file
@@ -0,0 +1,97 @@
|
||||
# Feature Specification: File Storage Management & UI
|
||||
|
||||
**Feature Branch**: `014-file-storage-ui`
|
||||
**Created**: 2026-01-24
|
||||
**Status**: Draft
|
||||
**Input**: User description: "Я хочу проработать механизм хранения файлов и доступа к ним - бекапов дашбордов и репозиториев дашбордов. Во первых, нужно иметь указывать место хранения, по умолчанию оно должно быть за файловой системой сервера (чтобы не влиять на git репозиторий. Во вторых, нужен web ui для базового доступа ко всем файлам - возможность скачивания/удаления, информация о датах создания, возможность загрузки"
|
||||
|
||||
## User Scenarios & Testing *(mandatory)*
|
||||
|
||||
### User Story 1 - File Management Dashboard (Priority: P1)
|
||||
|
||||
Users need a visual interface to manage the artifacts generated by the system (dashboard backups, exported repositories) without needing direct server access. Users must be able to navigate through the folder structure (e.g., `backups/SS2/Sales Dashboard`) to locate specific files.
|
||||
|
||||
**Why this priority**: Core functionality requested. Without the UI, the storage mechanism is opaque and hard to use.
|
||||
|
||||
**Independent Test**: Can be fully tested by opening the new "File Storage" page, navigating into a subdirectory, uploading a test file, verifying it appears in the list, downloading it, and then deleting it.
|
||||
|
||||
**Acceptance Scenarios**:
|
||||
|
||||
1. **Given** the File Storage page is open, **When** I view the list, **Then** I see the top-level folders (e.g., `backups`, `repositories`) or files.
|
||||
2. **Given** I am viewing a folder, **When** I click a subfolder name, **Then** the view updates to show the contents of that subfolder.
|
||||
3. **Given** I am in a subfolder, **When** I click "Download" on a file, **Then** the file is downloaded to my local machine.
|
||||
4. **Given** a file exists in the list, **When** I click "Delete" and confirm, **Then** the file is removed from the list and the server filesystem.
|
||||
5. **Given** I have a file locally, **When** I drag and drop it or use the "Upload" button, **Then** the file is uploaded to the current directory and appears in the list.
|
||||
|
||||
---
|
||||
|
||||
### User Story 2 - Storage Location Configuration (Priority: P2)
|
||||
|
||||
Administrators need to control where potentially large or sensitive files are stored. Crucially, these files must not accidentally pollute the source code repository or the application's working directory.
|
||||
|
||||
**Why this priority**: Essential for system stability and cleanliness (preventing git pollution), but the system could theoretically start with a hardcoded safe default.
|
||||
|
||||
**Independent Test**: Change the storage path in Settings, generate a file (or upload one), and verify it exists in the new location on the server disk.
|
||||
|
||||
**Acceptance Scenarios**:
|
||||
|
||||
1. **Given** I am in the Settings page, **When** I enter a new absolute path for "File Storage Path" and save, **Then** the system updates the configuration.
|
||||
2. **Given** the default configuration, **When** the system starts, **Then** the storage path defaults to a location outside the project's git scope (or is properly ignored).
|
||||
3. **Given** an invalid path (e.g., no write permissions), **When** I try to save, **Then** the system shows an error message.
|
||||
|
||||
---
|
||||
|
||||
### Edge Cases
|
||||
|
||||
- **File Name Conflicts**: What happens when uploading a file that already exists? (System should likely rename or ask to overwrite).
|
||||
- **Storage Quota/Disk Space**: What happens if the disk is full during upload?
|
||||
- **Path Traversal**: Ensure users cannot access files outside the configured storage directory via the API.
|
||||
- **Large Files**: Handling uploads/downloads of large backup archives (e.g., > 100MB).
|
||||
|
||||
## Clarifications
|
||||
|
||||
### Session 2026-01-24
|
||||
|
||||
- Q: Should the system enforce a structure to keep backups and repositories separate? → A: **Structured**: System creates and enforces `backups/` and `repositories/` folders; UI separates them.
|
||||
|
||||
### Session 2026-01-24 (Update)
|
||||
|
||||
- Q: Should the system allow advanced configuration of file structure and naming conventions? → A: **Yes**: Users should be able to configure the directory structure (e.g., include dashboard/environment names) and file naming patterns (e.g., include timestamps).
|
||||
|
||||
## Requirements *(mandatory)*
|
||||
|
||||
### Functional Requirements
|
||||
|
||||
- **FR-001**: System MUST allow configuring a local filesystem root path for storing artifacts.
|
||||
- **FR-002**: The default storage path MUST be configured such that it does not interfere with the application's git repository (e.g., a directory outside the workspace or explicitly git-ignored).
|
||||
- **FR-003**: System MUST enforce a directory structure within the storage root: `backups/` for dashboard backups and `repositories/` for exported repositories.
|
||||
- **FR-004**: System MUST provide a Web UI to list files and folders, organized by their type (Backup vs Repository).
|
||||
- **FR-005**: System MUST display file metadata in the UI: Filename, Size, Creation Date.
|
||||
- **FR-006**: System MUST allow users to download files from the storage directory (including subdirectories) via the Web UI.
|
||||
- **FR-007**: System MUST allow users to delete files from the storage directory via the Web UI.
|
||||
- **FR-008**: System MUST allow users to upload files to the specific folder in the storage directory via the Web UI.
|
||||
- **FR-013**: System MUST support navigating through the directory hierarchy within the allowed categories.
|
||||
- **FR-014**: System MUST display breadcrumbs or similar navigation aid to show current path.
|
||||
- **FR-009**: System MUST validate that the configured storage path is accessible and writable.
|
||||
- **FR-010**: System MUST prevent access to files outside the configured storage directory (Path Traversal protection).
|
||||
- **FR-011**: System MUST allow configuring the directory structure pattern for backups and repositories (e.g., `{environment}/{dashboard_name}/`).
|
||||
- **FR-012**: System MUST allow configuring the filename pattern for generated files (e.g., `{dashboard_name}_{timestamp}.zip`).
|
||||
|
||||
### Key Entities
|
||||
|
||||
- **StorageConfig**: Settings defining the root directory path.
|
||||
- **StoredFile**: Conceptual representation of a file (Name, Path, Size, CreatedAt, MimeType).
|
||||
|
||||
### Assumptions
|
||||
|
||||
- The application server has access to a writable local filesystem.
|
||||
- Users utilizing this feature have appropriate permissions within the application to manage system-wide storage settings and files.
|
||||
|
||||
## Success Criteria *(mandatory)*
|
||||
|
||||
### Measurable Outcomes
|
||||
|
||||
- **SC-001**: Users can successfully upload and then download a file of at least 50MB size.
|
||||
- **SC-002**: Files created or uploaded via the system do not appear in the application's `git status` output by default.
|
||||
- **SC-003**: File list loads in under 1 second for a directory containing 100 files.
|
||||
- **SC-004**: Users can delete a file via UI and confirm it is physically removed from the disk.
|
||||
96
specs/014-file-storage-ui/tasks.md
Normal file
96
specs/014-file-storage-ui/tasks.md
Normal file
@@ -0,0 +1,96 @@
|
||||
# Tasks: File Storage Management & UI
|
||||
|
||||
**Branch**: `014-file-storage-ui` | **Spec**: [specs/014-file-storage-ui/spec.md](../014-file-storage-ui/spec.md)
|
||||
|
||||
## Phase 1: Setup
|
||||
*Goal: Initialize backend plugin structure and frontend route scaffolding.*
|
||||
|
||||
- [x] T001 Create storage plugin directory and `__init__.py` in `backend/src/plugins/storage/`
|
||||
- [x] T002 Create storage models file `backend/src/models/storage.py` with `StorageConfig` and `StoredFile` Pydantic models
|
||||
- [x] T003 Create empty storage route handler `backend/src/api/routes/storage.py` and register in `backend/src/api/routes/__init__.py`
|
||||
- [x] T004 Create frontend storage route directory `frontend/src/routes/tools/storage/` and empty `+page.svelte`
|
||||
- [x] T005 Create frontend service `frontend/src/services/storageService.js` stub
|
||||
|
||||
## Phase 2: Foundational
|
||||
*Goal: Implement core backend logic for storage management, configuration, and security.*
|
||||
|
||||
- [x] T006 Implement `StoragePlugin` class in `backend/src/plugins/storage/plugin.py` inheriting from `PluginBase`
|
||||
- [x] T007 Implement `get_storage_root()` method in `StoragePlugin` with default path logic (`../ss-tools-storage`)
|
||||
- [x] T008 Implement `ensure_directories()` method to create `backups/` and `repositories/` subfolders on init
|
||||
- [x] T009 Implement path traversal protection helper `validate_path(path)` in `StoragePlugin`
|
||||
- [x] T010 Implement `list_files(category)` method in `StoragePlugin` returning `StoredFile` objects
|
||||
- [x] T011 Implement `save_file(file, category)` method in `StoragePlugin` handling uploads
|
||||
- [x] T012 Implement `delete_file(category, filename)` method in `StoragePlugin`
|
||||
- [x] T013 Implement `get_file_path(category, filename)` method in `StoragePlugin` for downloads
|
||||
- [x] T014 Register `StoragePlugin` in `backend/src/core/plugin_loader.py` (if manual registration needed)
|
||||
|
||||
## Phase 3: User Story 1 - File Management Dashboard (Priority: P1)
|
||||
*Goal: Enable users to list, upload, download, and delete files via Web UI.*
|
||||
|
||||
### Backend Endpoints
|
||||
- [x] T015 [US1] Implement `GET /api/storage/files` endpoint in `backend/src/api/routes/storage.py` using `StoragePlugin.list_files`
|
||||
- [x] T016 [US1] Implement `POST /api/storage/upload` endpoint in `backend/src/api/routes/storage.py` using `StoragePlugin.save_file`
|
||||
- [x] T017 [US1] Implement `DELETE /api/storage/files/{category}/{filename}` endpoint in `backend/src/api/routes/storage.py`
|
||||
- [x] T018 [US1] Implement `GET /api/storage/download/{category}/{filename}` endpoint in `backend/src/api/routes/storage.py`
|
||||
|
||||
### Frontend Implementation
|
||||
- [x] T019 [US1] Implement `listFiles`, `uploadFile`, `deleteFile`, `downloadFileUrl` in `frontend/src/services/storageService.js`
|
||||
- [x] T020 [US1] Create `frontend/src/components/storage/FileList.svelte` to display files in a table with metadata
|
||||
- [x] T021 [US1] Create `frontend/src/components/storage/FileUpload.svelte` with category selection and drag-drop support
|
||||
- [x] T022 [US1] Implement main logic in `frontend/src/routes/tools/storage/+page.svelte` to fetch files and handle tabs (Backups vs Repositories)
|
||||
- [x] T023 [US1] Integrate `FileList` and `FileUpload` components into `+page.svelte`
|
||||
|
||||
## Phase 4: User Story 2 - Storage Location Configuration (Priority: P2)
|
||||
*Goal: Allow administrators to configure the storage root path via Settings.*
|
||||
|
||||
### Backend
|
||||
- [x] T024 [US2] Add `storage_path` field to main configuration model in `backend/src/core/config_models.py` (if not using separate storage config)
|
||||
- [x] T025 [US2] Implement `GET /api/settings/storage` and `PUT /api/settings/storage` endpoints in `backend/src/api/routes/settings.py` (or `storage.py`)
|
||||
- [x] T026 [US2] Update `StoragePlugin` to read root path from global configuration instead of hardcoded default
|
||||
- [x] T027 [US2] Add validation logic to `PUT` endpoint to ensure new path is writable
|
||||
|
||||
### Frontend
|
||||
- [x] T028 [US2] Add `getStorageConfig` and `updateStorageConfig` to `frontend/src/services/storageService.js`
|
||||
- [x] T029 [US2] Create configuration section in `frontend/src/routes/settings/+page.svelte` (or dedicated Storage Settings component)
|
||||
- [x] T030 [US2] Implement form to update storage path with validation feedback
|
||||
- [x] T031 [US2] Add configuration fields for directory structure and filename patterns in `backend/src/models/storage.py` and `frontend/src/routes/settings/+page.svelte`
|
||||
- [x] T032 [US2] Implement logic in `StoragePlugin` to resolve dynamic paths based on configured patterns
|
||||
|
||||
## Phase 5: Polish & Cross-Cutting
|
||||
*Goal: Finalize UI/UX and ensure robustness.*
|
||||
|
||||
- [x] T033 Add link to "File Storage" in main navigation `frontend/src/components/Navbar.svelte`
|
||||
- [x] T034 Add error handling toasts for failed uploads or file operations
|
||||
- [x] T035 Verify large file upload support (50MB+) in Nginx/FastAPI config if applicable
|
||||
- [x] T036 Add confirmation modal for file deletion
|
||||
|
||||
## Phase 6: Folder Structure Support (Refactor)
|
||||
*Goal: Enable hierarchical navigation, nested file management, and downloading.*
|
||||
|
||||
- [x] T037 Refactor `StoragePlugin.list_files` in `backend/src/plugins/storage/plugin.py` to accept `subpath` and return directories/files
|
||||
- [x] T038 Refactor `StoragePlugin` methods (`save_file`, `delete_file`, `get_file_path`) to support nested paths
|
||||
- [x] T039 Update backend endpoints in `backend/src/api/routes/storage.py` (`GET /files`, `POST /upload`, `DELETE /files`, `GET /download`) to accept `path` parameter
|
||||
- [x] T040 Update `frontend/src/services/storageService.js` to pass `path` argument in all API calls
|
||||
- [x] T041 Update `frontend/src/components/storage/FileList.svelte` to display folder icons, handle navigation events, and show breadcrumbs
|
||||
- [x] T042 Update `frontend/src/components/storage/FileUpload.svelte` to upload to the currently active directory
|
||||
- [x] T043 Update `frontend/src/routes/tools/storage/+page.svelte` to manage current path state and handle navigation logic
|
||||
|
||||
## Dependencies
|
||||
|
||||
1. **Phase 1 (Setup)**: No dependencies.
|
||||
2. **Phase 2 (Foundational)**: Depends on Phase 1.
|
||||
3. **Phase 3 (US1)**: Depends on Phase 2.
|
||||
4. **Phase 4 (US2)**: Depends on Phase 2. Can run parallel to Phase 3.
|
||||
5. **Phase 5 (Polish)**: Depends on Phase 3 and 4.
|
||||
6. **Phase 6 (Refactor)**: Depends on Phase 3.
|
||||
|
||||
## Parallel Execution Examples
|
||||
|
||||
- **Backend/Frontend Split**: T015-T018 (Backend Endpoints) can be developed in parallel with T020-T021 (Frontend Components) using mock data.
|
||||
- **Story Split**: US1 (File Management) and US2 (Configuration) are largely independent after Phase 2 is complete.
|
||||
|
||||
## Implementation Strategy
|
||||
|
||||
1. **MVP**: Complete Phases 1, 2, and 3. This delivers a working file manager with a default storage location.
|
||||
2. **Full Feature**: Complete Phase 4 to allow path configuration.
|
||||
3. **Polish**: Complete Phase 5 for better UX.
|
||||
Reference in New Issue
Block a user