Файловое хранилище готово

This commit is contained in:
2026-01-26 11:08:18 +03:00
parent a542e7d2df
commit edf9286071
35 changed files with 377 additions and 497 deletions

Submodule backend/backend/git_repos/12 updated: d592fa7ed5...f46772443a

View File

@@ -1,269 +0,0 @@
2025-12-20 19:55:11,325 - INFO - [BackupPlugin][Entry] Starting backup for superset.
2025-12-20 19:55:11,325 - INFO - [setup_clients][Enter] Starting Superset clients initialization.
2025-12-20 19:55:11,327 - CRITICAL - [setup_clients][Failure] Critical error during client initialization: 1 validation error for SupersetConfig
base_url
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
For further information visit https://errors.pydantic.dev/2.12/v/value_error
Traceback (most recent call last):
File "/home/user/ss-tools/superset_tool/utils/init_clients.py", line 43, in setup_clients
config = SupersetConfig(
^^^^^^^^^^^^^^^
File "/home/user/ss-tools/backend/venv/lib/python3.12/site-packages/pydantic/main.py", line 250, in __init__
validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 1 validation error for SupersetConfig
base_url
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
For further information visit https://errors.pydantic.dev/2.12/v/value_error
2025-12-20 21:01:49,905 - INFO - [BackupPlugin][Entry] Starting backup for superset.
2025-12-20 21:01:49,906 - INFO - [setup_clients][Enter] Starting Superset clients initialization.
2025-12-20 21:01:49,988 - INFO - [setup_clients][Action] Loading environments from ConfigManager
2025-12-20 21:01:49,990 - CRITICAL - [setup_clients][Failure] Critical error during client initialization: 1 validation error for SupersetConfig
base_url
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
For further information visit https://errors.pydantic.dev/2.12/v/value_error
Traceback (most recent call last):
File "/home/user/ss-tools/superset_tool/utils/init_clients.py", line 66, in setup_clients
config = SupersetConfig(
^^^^^^^^^^^^^^^
File "/home/user/ss-tools/venv/lib/python3.12/site-packages/pydantic/main.py", line 250, in __init__
validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 1 validation error for SupersetConfig
base_url
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
For further information visit https://errors.pydantic.dev/2.12/v/value_error
2025-12-20 22:42:32,538 - INFO - [BackupPlugin][Entry] Starting backup for superset.
2025-12-20 22:42:32,538 - INFO - [setup_clients][Enter] Starting Superset clients initialization.
2025-12-20 22:42:32,583 - INFO - [setup_clients][Action] Loading environments from ConfigManager
2025-12-20 22:42:32,587 - CRITICAL - [setup_clients][Failure] Critical error during client initialization: 1 validation error for SupersetConfig
base_url
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
For further information visit https://errors.pydantic.dev/2.12/v/value_error
Traceback (most recent call last):
File "/home/user/ss-tools/superset_tool/utils/init_clients.py", line 66, in setup_clients
config = SupersetConfig(
^^^^^^^^^^^^^^^
File "/home/user/ss-tools/backend/.venv/lib/python3.12/site-packages/pydantic/main.py", line 250, in __init__
validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 1 validation error for SupersetConfig
base_url
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
For further information visit https://errors.pydantic.dev/2.12/v/value_error
2025-12-20 22:54:29,770 - INFO - [BackupPlugin][Entry] Starting backup for .
2025-12-20 22:54:29,771 - INFO - [setup_clients][Enter] Starting Superset clients initialization.
2025-12-20 22:54:29,831 - INFO - [setup_clients][Action] Loading environments from ConfigManager
2025-12-20 22:54:29,833 - CRITICAL - [setup_clients][Failure] Critical error during client initialization: 1 validation error for SupersetConfig
base_url
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
For further information visit https://errors.pydantic.dev/2.12/v/value_error
Traceback (most recent call last):
File "/home/user/ss-tools/superset_tool/utils/init_clients.py", line 66, in setup_clients
config = SupersetConfig(
^^^^^^^^^^^^^^^
File "/home/user/ss-tools/backend/.venv/lib/python3.12/site-packages/pydantic/main.py", line 250, in __init__
validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 1 validation error for SupersetConfig
base_url
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
For further information visit https://errors.pydantic.dev/2.12/v/value_error
2025-12-20 22:54:34,078 - INFO - [BackupPlugin][Entry] Starting backup for superset.
2025-12-20 22:54:34,078 - INFO - [setup_clients][Enter] Starting Superset clients initialization.
2025-12-20 22:54:34,079 - INFO - [setup_clients][Action] Loading environments from ConfigManager
2025-12-20 22:54:34,079 - CRITICAL - [setup_clients][Failure] Critical error during client initialization: 1 validation error for SupersetConfig
base_url
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
For further information visit https://errors.pydantic.dev/2.12/v/value_error
Traceback (most recent call last):
File "/home/user/ss-tools/superset_tool/utils/init_clients.py", line 66, in setup_clients
config = SupersetConfig(
^^^^^^^^^^^^^^^
File "/home/user/ss-tools/backend/.venv/lib/python3.12/site-packages/pydantic/main.py", line 250, in __init__
validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 1 validation error for SupersetConfig
base_url
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
For further information visit https://errors.pydantic.dev/2.12/v/value_error
2025-12-20 22:59:25,060 - INFO - [BackupPlugin][Entry] Starting backup for superset.
2025-12-20 22:59:25,060 - INFO - [setup_clients][Enter] Starting Superset clients initialization.
2025-12-20 22:59:25,114 - INFO - [setup_clients][Action] Loading environments from ConfigManager
2025-12-20 22:59:25,117 - CRITICAL - [setup_clients][Failure] Critical error during client initialization: 1 validation error for SupersetConfig
base_url
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
For further information visit https://errors.pydantic.dev/2.12/v/value_error
Traceback (most recent call last):
File "/home/user/ss-tools/superset_tool/utils/init_clients.py", line 66, in setup_clients
config = SupersetConfig(
^^^^^^^^^^^^^^^
File "/home/user/ss-tools/backend/.venv/lib/python3.12/site-packages/pydantic/main.py", line 250, in __init__
validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 1 validation error for SupersetConfig
base_url
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
For further information visit https://errors.pydantic.dev/2.12/v/value_error
2025-12-20 23:00:31,156 - INFO - [BackupPlugin][Entry] Starting backup for superset.
2025-12-20 23:00:31,156 - INFO - [setup_clients][Enter] Starting Superset clients initialization.
2025-12-20 23:00:31,157 - INFO - [setup_clients][Action] Loading environments from ConfigManager
2025-12-20 23:00:31,162 - CRITICAL - [setup_clients][Failure] Critical error during client initialization: 1 validation error for SupersetConfig
base_url
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
For further information visit https://errors.pydantic.dev/2.12/v/value_error
Traceback (most recent call last):
File "/home/user/ss-tools/superset_tool/utils/init_clients.py", line 66, in setup_clients
config = SupersetConfig(
^^^^^^^^^^^^^^^
File "/home/user/ss-tools/backend/.venv/lib/python3.12/site-packages/pydantic/main.py", line 250, in __init__
validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 1 validation error for SupersetConfig
base_url
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
For further information visit https://errors.pydantic.dev/2.12/v/value_error
2025-12-20 23:00:34,710 - INFO - [BackupPlugin][Entry] Starting backup for superset.
2025-12-20 23:00:34,710 - INFO - [setup_clients][Enter] Starting Superset clients initialization.
2025-12-20 23:00:34,710 - INFO - [setup_clients][Action] Loading environments from ConfigManager
2025-12-20 23:00:34,711 - CRITICAL - [setup_clients][Failure] Critical error during client initialization: 1 validation error for SupersetConfig
base_url
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
For further information visit https://errors.pydantic.dev/2.12/v/value_error
Traceback (most recent call last):
File "/home/user/ss-tools/superset_tool/utils/init_clients.py", line 66, in setup_clients
config = SupersetConfig(
^^^^^^^^^^^^^^^
File "/home/user/ss-tools/backend/.venv/lib/python3.12/site-packages/pydantic/main.py", line 250, in __init__
validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 1 validation error for SupersetConfig
base_url
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
For further information visit https://errors.pydantic.dev/2.12/v/value_error
2025-12-20 23:01:43,894 - INFO - [BackupPlugin][Entry] Starting backup for superset.
2025-12-20 23:01:43,894 - INFO - [setup_clients][Enter] Starting Superset clients initialization.
2025-12-20 23:01:43,895 - INFO - [setup_clients][Action] Loading environments from ConfigManager
2025-12-20 23:01:43,895 - CRITICAL - [setup_clients][Failure] Critical error during client initialization: 1 validation error for SupersetConfig
base_url
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
For further information visit https://errors.pydantic.dev/2.12/v/value_error
Traceback (most recent call last):
File "/home/user/ss-tools/superset_tool/utils/init_clients.py", line 66, in setup_clients
config = SupersetConfig(
^^^^^^^^^^^^^^^
File "/home/user/ss-tools/backend/.venv/lib/python3.12/site-packages/pydantic/main.py", line 250, in __init__
validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 1 validation error for SupersetConfig
base_url
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
For further information visit https://errors.pydantic.dev/2.12/v/value_error
2025-12-20 23:04:07,731 - INFO - [BackupPlugin][Entry] Starting backup for superset.
2025-12-20 23:04:07,731 - INFO - [setup_clients][Enter] Starting Superset clients initialization.
2025-12-20 23:04:07,732 - INFO - [setup_clients][Action] Loading environments from ConfigManager
2025-12-20 23:04:07,732 - CRITICAL - [setup_clients][Failure] Critical error during client initialization: 1 validation error for SupersetConfig
base_url
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
For further information visit https://errors.pydantic.dev/2.12/v/value_error
Traceback (most recent call last):
File "/home/user/ss-tools/superset_tool/utils/init_clients.py", line 66, in setup_clients
config = SupersetConfig(
^^^^^^^^^^^^^^^
File "/home/user/ss-tools/backend/.venv/lib/python3.12/site-packages/pydantic/main.py", line 250, in __init__
validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 1 validation error for SupersetConfig
base_url
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
For further information visit https://errors.pydantic.dev/2.12/v/value_error
2025-12-20 23:06:39,641 - INFO - [BackupPlugin][Entry] Starting backup for superset.
2025-12-20 23:06:39,642 - INFO - [setup_clients][Enter] Starting Superset clients initialization.
2025-12-20 23:06:39,687 - INFO - [setup_clients][Action] Loading environments from ConfigManager
2025-12-20 23:06:39,689 - CRITICAL - [setup_clients][Failure] Critical error during client initialization: 1 validation error for SupersetConfig
base_url
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
For further information visit https://errors.pydantic.dev/2.12/v/value_error
Traceback (most recent call last):
File "/home/user/ss-tools/superset_tool/utils/init_clients.py", line 66, in setup_clients
config = SupersetConfig(
^^^^^^^^^^^^^^^
File "/home/user/ss-tools/backend/.venv/lib/python3.12/site-packages/pydantic/main.py", line 250, in __init__
validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 1 validation error for SupersetConfig
base_url
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
For further information visit https://errors.pydantic.dev/2.12/v/value_error
2025-12-20 23:30:36,090 - INFO - [BackupPlugin][Entry] Starting backup for superset.
2025-12-20 23:30:36,093 - INFO - [setup_clients][Enter] Starting Superset clients initialization.
2025-12-20 23:30:36,128 - INFO - [setup_clients][Action] Loading environments from ConfigManager
2025-12-20 23:30:36,129 - INFO - [SupersetClient.__init__][Enter] Initializing SupersetClient.
2025-12-20 23:30:36,129 - INFO - [APIClient.__init__][Entry] Initializing APIClient.
2025-12-20 23:30:36,130 - WARNING - [_init_session][State] SSL verification disabled.
2025-12-20 23:30:36,130 - INFO - [APIClient.__init__][Exit] APIClient initialized.
2025-12-20 23:30:36,130 - INFO - [SupersetClient.__init__][Exit] SupersetClient initialized.
2025-12-20 23:30:36,130 - INFO - [get_dashboards][Enter] Fetching dashboards.
2025-12-20 23:30:36,131 - INFO - [authenticate][Enter] Authenticating to https://superset.bebesh.ru/api/v1
2025-12-20 23:30:36,897 - INFO - [authenticate][Exit] Authenticated successfully.
2025-12-20 23:30:37,527 - INFO - [get_dashboards][Exit] Found 11 dashboards.
2025-12-20 23:30:37,527 - INFO - [BackupPlugin][Progress] Found 11 dashboards to export in superset.
2025-12-20 23:30:37,529 - INFO - [export_dashboard][Enter] Exporting dashboard 11.
2025-12-20 23:30:38,224 - INFO - [export_dashboard][Exit] Exported dashboard 11 to dashboard_export_20251220T203037.zip.
2025-12-20 23:30:38,225 - INFO - [save_and_unpack_dashboard][Enter] Processing dashboard. Unpack: False
2025-12-20 23:30:38,226 - INFO - [save_and_unpack_dashboard][State] Dashboard saved to: backups/SUPERSET/FCC New Coder Survey 2018/dashboard_export_20251220T203037.zip
2025-12-20 23:30:38,227 - INFO - [archive_exports][Enter] Managing archive in backups/SUPERSET/FCC New Coder Survey 2018
2025-12-20 23:30:38,230 - INFO - [export_dashboard][Enter] Exporting dashboard 10.
2025-12-20 23:30:38,438 - INFO - [export_dashboard][Exit] Exported dashboard 10 to dashboard_export_20251220T203038.zip.
2025-12-20 23:30:38,438 - INFO - [save_and_unpack_dashboard][Enter] Processing dashboard. Unpack: False
2025-12-20 23:30:38,439 - INFO - [save_and_unpack_dashboard][State] Dashboard saved to: backups/SUPERSET/COVID Vaccine Dashboard/dashboard_export_20251220T203038.zip
2025-12-20 23:30:38,439 - INFO - [archive_exports][Enter] Managing archive in backups/SUPERSET/COVID Vaccine Dashboard
2025-12-20 23:30:38,440 - INFO - [export_dashboard][Enter] Exporting dashboard 9.
2025-12-20 23:30:38,853 - INFO - [export_dashboard][Exit] Exported dashboard 9 to dashboard_export_20251220T203038.zip.
2025-12-20 23:30:38,853 - INFO - [save_and_unpack_dashboard][Enter] Processing dashboard. Unpack: False
2025-12-20 23:30:38,856 - INFO - [save_and_unpack_dashboard][State] Dashboard saved to: backups/SUPERSET/Sales Dashboard/dashboard_export_20251220T203038.zip
2025-12-20 23:30:38,856 - INFO - [archive_exports][Enter] Managing archive in backups/SUPERSET/Sales Dashboard
2025-12-20 23:30:38,858 - INFO - [export_dashboard][Enter] Exporting dashboard 8.
2025-12-20 23:30:38,939 - INFO - [export_dashboard][Exit] Exported dashboard 8 to dashboard_export_20251220T203038.zip.
2025-12-20 23:30:38,940 - INFO - [save_and_unpack_dashboard][Enter] Processing dashboard. Unpack: False
2025-12-20 23:30:38,941 - INFO - [save_and_unpack_dashboard][State] Dashboard saved to: backups/SUPERSET/Unicode Test/dashboard_export_20251220T203038.zip
2025-12-20 23:30:38,941 - INFO - [archive_exports][Enter] Managing archive in backups/SUPERSET/Unicode Test
2025-12-20 23:30:38,942 - INFO - [export_dashboard][Enter] Exporting dashboard 7.
2025-12-20 23:30:39,148 - INFO - [export_dashboard][Exit] Exported dashboard 7 to dashboard_export_20251220T203038.zip.
2025-12-20 23:30:39,148 - INFO - [save_and_unpack_dashboard][Enter] Processing dashboard. Unpack: False
2025-12-20 23:30:39,149 - INFO - [save_and_unpack_dashboard][State] Dashboard saved to: backups/SUPERSET/Video Game Sales/dashboard_export_20251220T203038.zip
2025-12-20 23:30:39,149 - INFO - [archive_exports][Enter] Managing archive in backups/SUPERSET/Video Game Sales
2025-12-20 23:30:39,150 - INFO - [export_dashboard][Enter] Exporting dashboard 6.
2025-12-20 23:30:39,689 - INFO - [export_dashboard][Exit] Exported dashboard 6 to dashboard_export_20251220T203039.zip.
2025-12-20 23:30:39,689 - INFO - [save_and_unpack_dashboard][Enter] Processing dashboard. Unpack: False
2025-12-20 23:30:39,690 - INFO - [save_and_unpack_dashboard][State] Dashboard saved to: backups/SUPERSET/Featured Charts/dashboard_export_20251220T203039.zip
2025-12-20 23:30:39,691 - INFO - [archive_exports][Enter] Managing archive in backups/SUPERSET/Featured Charts
2025-12-20 23:30:39,692 - INFO - [export_dashboard][Enter] Exporting dashboard 5.
2025-12-20 23:30:39,960 - INFO - [export_dashboard][Exit] Exported dashboard 5 to dashboard_export_20251220T203039.zip.
2025-12-20 23:30:39,960 - INFO - [save_and_unpack_dashboard][Enter] Processing dashboard. Unpack: False
2025-12-20 23:30:39,961 - INFO - [save_and_unpack_dashboard][State] Dashboard saved to: backups/SUPERSET/Slack Dashboard/dashboard_export_20251220T203039.zip
2025-12-20 23:30:39,961 - INFO - [archive_exports][Enter] Managing archive in backups/SUPERSET/Slack Dashboard
2025-12-20 23:30:39,962 - INFO - [export_dashboard][Enter] Exporting dashboard 4.
2025-12-20 23:30:40,196 - INFO - [export_dashboard][Exit] Exported dashboard 4 to dashboard_export_20251220T203039.zip.
2025-12-20 23:30:40,196 - INFO - [save_and_unpack_dashboard][Enter] Processing dashboard. Unpack: False
2025-12-20 23:30:40,197 - INFO - [save_and_unpack_dashboard][State] Dashboard saved to: backups/SUPERSET/deck.gl Demo/dashboard_export_20251220T203039.zip
2025-12-20 23:30:40,197 - INFO - [archive_exports][Enter] Managing archive in backups/SUPERSET/deck.gl Demo
2025-12-20 23:30:40,198 - INFO - [export_dashboard][Enter] Exporting dashboard 3.
2025-12-20 23:30:40,745 - INFO - [export_dashboard][Exit] Exported dashboard 3 to dashboard_export_20251220T203040.zip.
2025-12-20 23:30:40,746 - INFO - [save_and_unpack_dashboard][Enter] Processing dashboard. Unpack: False
2025-12-20 23:30:40,760 - INFO - [save_and_unpack_dashboard][State] Dashboard saved to: backups/SUPERSET/Misc Charts/dashboard_export_20251220T203040.zip
2025-12-20 23:30:40,761 - INFO - [archive_exports][Enter] Managing archive in backups/SUPERSET/Misc Charts
2025-12-20 23:30:40,762 - INFO - [export_dashboard][Enter] Exporting dashboard 2.
2025-12-20 23:30:40,928 - INFO - [export_dashboard][Exit] Exported dashboard 2 to dashboard_export_20251220T203040.zip.
2025-12-20 23:30:40,929 - INFO - [save_and_unpack_dashboard][Enter] Processing dashboard. Unpack: False
2025-12-20 23:30:40,930 - INFO - [save_and_unpack_dashboard][State] Dashboard saved to: backups/SUPERSET/USA Births Names/dashboard_export_20251220T203040.zip
2025-12-20 23:30:40,931 - INFO - [archive_exports][Enter] Managing archive in backups/SUPERSET/USA Births Names
2025-12-20 23:30:40,932 - INFO - [export_dashboard][Enter] Exporting dashboard 1.
2025-12-20 23:30:41,582 - INFO - [export_dashboard][Exit] Exported dashboard 1 to dashboard_export_20251220T203040.zip.
2025-12-20 23:30:41,582 - INFO - [save_and_unpack_dashboard][Enter] Processing dashboard. Unpack: False
2025-12-20 23:30:41,749 - INFO - [save_and_unpack_dashboard][State] Dashboard saved to: backups/SUPERSET/World Bank's Data/dashboard_export_20251220T203040.zip
2025-12-20 23:30:41,750 - INFO - [archive_exports][Enter] Managing archive in backups/SUPERSET/World Bank's Data
2025-12-20 23:30:41,752 - INFO - [consolidate_archive_folders][Enter] Consolidating archives in backups/SUPERSET
2025-12-20 23:30:41,753 - INFO - [remove_empty_directories][Enter] Starting cleanup of empty directories in backups/SUPERSET
2025-12-20 23:30:41,758 - INFO - [remove_empty_directories][Exit] Removed 0 empty directories.
2025-12-20 23:30:41,758 - INFO - [BackupPlugin][CoherenceCheck:Passed] Backup logic completed for superset.

Binary file not shown.

View File

@@ -53,6 +53,7 @@ async def update_global_settings(
): ):
with belief_scope("update_global_settings"): with belief_scope("update_global_settings"):
logger.info("[update_global_settings][Entry] Updating global settings") logger.info("[update_global_settings][Entry] Updating global settings")
config_manager.update_global_settings(settings) config_manager.update_global_settings(settings)
return settings return settings
# [/DEF:update_global_settings:Function] # [/DEF:update_global_settings:Function]
@@ -207,30 +208,5 @@ async def test_environment_connection(
return {"status": "error", "message": str(e)} return {"status": "error", "message": str(e)}
# [/DEF:test_environment_connection:Function] # [/DEF:test_environment_connection:Function]
# [DEF:validate_backup_path:Function]
# @PURPOSE: Validates if a backup path exists and is writable.
# @PRE: Path is provided in path_data.
# @POST: Returns success or error status.
# @PARAM: path (str) - The path to validate.
# @RETURN: dict - Validation result.
@router.post("/validate-path")
async def validate_backup_path(
path_data: dict,
config_manager: ConfigManager = Depends(get_config_manager)
):
with belief_scope("validate_backup_path"):
path = path_data.get("path")
if not path:
raise HTTPException(status_code=400, detail="Path is required")
logger.info(f"[validate_backup_path][Entry] Validating path: {path}")
valid, message = config_manager.validate_path(path)
if not valid:
return {"status": "error", "message": message}
return {"status": "success", "message": message}
# [/DEF:validate_backup_path:Function]
# [/DEF:SettingsRouter:Module] # [/DEF:SettingsRouter:Module]

View File

@@ -11,41 +11,47 @@
from fastapi import APIRouter, Depends, UploadFile, File, Form, HTTPException from fastapi import APIRouter, Depends, UploadFile, File, Form, HTTPException
from fastapi.responses import FileResponse from fastapi.responses import FileResponse
from typing import List, Optional from typing import List, Optional
from backend.src.models.storage import StoredFile, FileCategory from ...models.storage import StoredFile, FileCategory
from backend.src.dependencies import get_plugin_loader from ...dependencies import get_plugin_loader
from backend.src.plugins.storage.plugin import StoragePlugin from ...plugins.storage.plugin import StoragePlugin
from backend.src.core.logger import belief_scope from ...core.logger import belief_scope
# [/SECTION] # [/SECTION]
router = APIRouter(tags=["storage"]) router = APIRouter(tags=["storage"])
# [DEF:list_files:Function] # [DEF:list_files:Function]
# @PURPOSE: List all files in the storage system, optionally filtered by category. # @PURPOSE: List all files and directories in the storage system.
# #
# @PRE: None. # @PRE: None.
# @POST: Returns a list of StoredFile objects. # @POST: Returns a list of StoredFile objects.
# #
# @PARAM: category (Optional[FileCategory]) - Filter by category. # @PARAM: category (Optional[FileCategory]) - Filter by category.
# @RETURN: List[StoredFile] - List of files. # @PARAM: path (Optional[str]) - Subpath within the category.
# @RETURN: List[StoredFile] - List of files/directories.
# #
# @RELATION: CALLS -> StoragePlugin.list_files # @RELATION: CALLS -> StoragePlugin.list_files
@router.get("/files", response_model=List[StoredFile]) @router.get("/files", response_model=List[StoredFile])
async def list_files(category: Optional[FileCategory] = None, plugin_loader=Depends(get_plugin_loader)): async def list_files(
category: Optional[FileCategory] = None,
path: Optional[str] = None,
plugin_loader=Depends(get_plugin_loader)
):
with belief_scope("list_files"): with belief_scope("list_files"):
storage_plugin: StoragePlugin = plugin_loader.get_plugin("storage-manager") storage_plugin: StoragePlugin = plugin_loader.get_plugin("storage-manager")
if not storage_plugin: if not storage_plugin:
raise HTTPException(status_code=500, detail="Storage plugin not loaded") raise HTTPException(status_code=500, detail="Storage plugin not loaded")
return storage_plugin.list_files(category) return storage_plugin.list_files(category, path)
# [/DEF:list_files:Function] # [/DEF:list_files:Function]
# [DEF:upload_file:Function] # [DEF:upload_file:Function]
# @PURPOSE: Upload a file to the storage system under a specific category. # @PURPOSE: Upload a file to the storage system.
# #
# @PRE: category must be a valid FileCategory. # @PRE: category must be a valid FileCategory.
# @PRE: file must be a valid UploadFile. # @PRE: file must be a valid UploadFile.
# @POST: Returns the StoredFile object of the uploaded file. # @POST: Returns the StoredFile object of the uploaded file.
# #
# @PARAM: category (FileCategory) - Target category. # @PARAM: category (FileCategory) - Target category.
# @PARAM: path (Optional[str]) - Target subpath.
# @PARAM: file (UploadFile) - The file content. # @PARAM: file (UploadFile) - The file content.
# @RETURN: StoredFile - Metadata of the uploaded file. # @RETURN: StoredFile - Metadata of the uploaded file.
# #
@@ -55,6 +61,7 @@ async def list_files(category: Optional[FileCategory] = None, plugin_loader=Depe
@router.post("/upload", response_model=StoredFile, status_code=201) @router.post("/upload", response_model=StoredFile, status_code=201)
async def upload_file( async def upload_file(
category: FileCategory = Form(...), category: FileCategory = Form(...),
path: Optional[str] = Form(None),
file: UploadFile = File(...), file: UploadFile = File(...),
plugin_loader=Depends(get_plugin_loader) plugin_loader=Depends(get_plugin_loader)
): ):
@@ -63,33 +70,32 @@ async def upload_file(
if not storage_plugin: if not storage_plugin:
raise HTTPException(status_code=500, detail="Storage plugin not loaded") raise HTTPException(status_code=500, detail="Storage plugin not loaded")
try: try:
return await storage_plugin.save_file(file, category) return await storage_plugin.save_file(file, category, path)
except ValueError as e: except ValueError as e:
raise HTTPException(status_code=400, detail=str(e)) raise HTTPException(status_code=400, detail=str(e))
# [/DEF:upload_file:Function] # [/DEF:upload_file:Function]
# [DEF:delete_file:Function] # [DEF:delete_file:Function]
# @PURPOSE: Delete a specific file from the storage system. # @PURPOSE: Delete a specific file or directory.
# #
# @PRE: category must be a valid FileCategory. # @PRE: category must be a valid FileCategory.
# @PRE: filename must not contain path separators. # @POST: Item is removed from storage.
# @POST: File is removed from storage.
# #
# @PARAM: category (FileCategory) - File category. # @PARAM: category (FileCategory) - File category.
# @PARAM: filename (str) - Name of the file. # @PARAM: path (str) - Relative path of the item.
# @RETURN: None # @RETURN: None
# #
# @SIDE_EFFECT: Deletes file from the filesystem. # @SIDE_EFFECT: Deletes item from the filesystem.
# #
# @RELATION: CALLS -> StoragePlugin.delete_file # @RELATION: CALLS -> StoragePlugin.delete_file
@router.delete("/files/{category}/{filename}", status_code=204) @router.delete("/files/{category}/{path:path}", status_code=204)
async def delete_file(category: FileCategory, filename: str, plugin_loader=Depends(get_plugin_loader)): async def delete_file(category: FileCategory, path: str, plugin_loader=Depends(get_plugin_loader)):
with belief_scope("delete_file"): with belief_scope("delete_file"):
storage_plugin: StoragePlugin = plugin_loader.get_plugin("storage-manager") storage_plugin: StoragePlugin = plugin_loader.get_plugin("storage-manager")
if not storage_plugin: if not storage_plugin:
raise HTTPException(status_code=500, detail="Storage plugin not loaded") raise HTTPException(status_code=500, detail="Storage plugin not loaded")
try: try:
storage_plugin.delete_file(category, filename) storage_plugin.delete_file(category, path)
except FileNotFoundError: except FileNotFoundError:
raise HTTPException(status_code=404, detail="File not found") raise HTTPException(status_code=404, detail="File not found")
except ValueError as e: except ValueError as e:
@@ -100,23 +106,23 @@ async def delete_file(category: FileCategory, filename: str, plugin_loader=Depen
# @PURPOSE: Retrieve a file for download. # @PURPOSE: Retrieve a file for download.
# #
# @PRE: category must be a valid FileCategory. # @PRE: category must be a valid FileCategory.
# @PRE: filename must exist in the specified category.
# @POST: Returns a FileResponse. # @POST: Returns a FileResponse.
# #
# @PARAM: category (FileCategory) - File category. # @PARAM: category (FileCategory) - File category.
# @PARAM: filename (str) - Name of the file. # @PARAM: path (str) - Relative path of the file.
# @RETURN: FileResponse - The file content. # @RETURN: FileResponse - The file content.
# #
# @RELATION: CALLS -> StoragePlugin.get_file_path # @RELATION: CALLS -> StoragePlugin.get_file_path
@router.get("/download/{category}/{filename}") @router.get("/download/{category}/{path:path}")
async def download_file(category: FileCategory, filename: str, plugin_loader=Depends(get_plugin_loader)): async def download_file(category: FileCategory, path: str, plugin_loader=Depends(get_plugin_loader)):
with belief_scope("download_file"): with belief_scope("download_file"):
storage_plugin: StoragePlugin = plugin_loader.get_plugin("storage-manager") storage_plugin: StoragePlugin = plugin_loader.get_plugin("storage-manager")
if not storage_plugin: if not storage_plugin:
raise HTTPException(status_code=500, detail="Storage plugin not loaded") raise HTTPException(status_code=500, detail="Storage plugin not loaded")
try: try:
path = storage_plugin.get_file_path(category, filename) abs_path = storage_plugin.get_file_path(category, path)
return FileResponse(path=path, filename=filename) filename = Path(path).name
return FileResponse(path=abs_path, filename=filename)
except FileNotFoundError: except FileNotFoundError:
raise HTTPException(status_code=404, detail="File not found") raise HTTPException(status_code=404, detail="File not found")
except ValueError as e: except ValueError as e:

View File

@@ -62,14 +62,18 @@ class ConfigManager:
logger.info(f"[_load_config][Action] Config file not found. Creating default.") logger.info(f"[_load_config][Action] Config file not found. Creating default.")
default_config = AppConfig( default_config = AppConfig(
environments=[], environments=[],
settings=GlobalSettings(backup_path="backups") settings=GlobalSettings()
) )
self._save_config_to_disk(default_config) self._save_config_to_disk(default_config)
return default_config return default_config
try: try:
with open(self.config_path, "r") as f: with open(self.config_path, "r") as f:
data = json.load(f) data = json.load(f)
# Check for deprecated field
if "settings" in data and "backup_path" in data["settings"]:
del data["settings"]["backup_path"]
config = AppConfig(**data) config = AppConfig(**data)
logger.info(f"[_load_config][Coherence:OK] Configuration loaded") logger.info(f"[_load_config][Coherence:OK] Configuration loaded")
return config return config
@@ -79,7 +83,7 @@ class ConfigManager:
# For now, return default to be safe, but log the error prominently. # For now, return default to be safe, but log the error prominently.
return AppConfig( return AppConfig(
environments=[], environments=[],
settings=GlobalSettings(backup_path="backups") settings=GlobalSettings(storage=StorageConfig())
) )
# [/DEF:_load_config:Function] # [/DEF:_load_config:Function]

View File

@@ -43,7 +43,6 @@ class LoggingConfig(BaseModel):
# [DEF:GlobalSettings:DataClass] # [DEF:GlobalSettings:DataClass]
# @PURPOSE: Represents global application settings. # @PURPOSE: Represents global application settings.
class GlobalSettings(BaseModel): class GlobalSettings(BaseModel):
backup_path: str
storage: StorageConfig = Field(default_factory=StorageConfig) storage: StorageConfig = Field(default_factory=StorageConfig)
default_environment_id: Optional[str] = None default_environment_id: Optional[str] = None
logging: LoggingConfig = Field(default_factory=LoggingConfig) logging: LoggingConfig = Field(default_factory=LoggingConfig)

View File

@@ -5,13 +5,13 @@ from pydantic import BaseModel, Field
# [DEF:FileCategory:Class] # [DEF:FileCategory:Class]
class FileCategory(str, Enum): class FileCategory(str, Enum):
BACKUP = "backup" BACKUP = "backups"
REPOSITORY = "repository" REPOSITORY = "repositorys"
# [/DEF:FileCategory:Class] # [/DEF:FileCategory:Class]
# [DEF:StorageConfig:Class] # [DEF:StorageConfig:Class]
class StorageConfig(BaseModel): class StorageConfig(BaseModel):
root_path: str = Field(default="../ss-tools-storage", description="Absolute path to the storage root directory.") root_path: str = Field(default="backups", description="Absolute path to the storage root directory.")
backup_structure_pattern: str = Field(default="{category}/", description="Pattern for backup directory structure.") backup_structure_pattern: str = Field(default="{category}/", description="Pattern for backup directory structure.")
repo_structure_pattern: str = Field(default="{category}/", description="Pattern for repository directory structure.") repo_structure_pattern: str = Field(default="{category}/", description="Pattern for repository directory structure.")
filename_pattern: str = Field(default="{name}_{timestamp}", description="Pattern for filenames.") filename_pattern: str = Field(default="{name}_{timestamp}", description="Pattern for filenames.")

View File

@@ -84,7 +84,7 @@ class BackupPlugin(PluginBase):
with belief_scope("get_schema"): with belief_scope("get_schema"):
config_manager = get_config_manager() config_manager = get_config_manager()
envs = [e.name for e in config_manager.get_environments()] envs = [e.name for e in config_manager.get_environments()]
default_path = config_manager.get_config().settings.backup_path default_path = config_manager.get_config().settings.storage.root_path
return { return {
"type": "object", "type": "object",
@@ -95,14 +95,8 @@ class BackupPlugin(PluginBase):
"description": "The Superset environment to back up.", "description": "The Superset environment to back up.",
"enum": envs if envs else [], "enum": envs if envs else [],
}, },
"backup_path": {
"type": "string",
"title": "Backup Path",
"description": "The root directory to save backups to.",
"default": default_path
}
}, },
"required": ["env", "backup_path"], "required": ["env"],
} }
# [/DEF:get_schema:Function] # [/DEF:get_schema:Function]
@@ -126,8 +120,9 @@ class BackupPlugin(PluginBase):
if not env: if not env:
raise KeyError("env") raise KeyError("env")
backup_path_str = params.get("backup_path") or config_manager.get_config().settings.backup_path storage_settings = config_manager.get_config().settings.storage
backup_path = Path(backup_path_str) # Use 'backups' subfolder within the storage root
backup_path = Path(storage_settings.root_path) / "backups"
from ..core.logger import logger as app_logger from ..core.logger import logger as app_logger
app_logger.info(f"[BackupPlugin][Entry] Starting backup for {env}.") app_logger.info(f"[BackupPlugin][Entry] Starting backup for {env}.")

View File

@@ -98,14 +98,43 @@ class StoragePlugin(PluginBase):
def get_storage_root(self) -> Path: def get_storage_root(self) -> Path:
with belief_scope("StoragePlugin:get_storage_root"): with belief_scope("StoragePlugin:get_storage_root"):
config_manager = get_config_manager() config_manager = get_config_manager()
storage_config = config_manager.get_config().settings.storage global_settings = config_manager.get_config().settings
root = Path(storage_config.root_path)
# Use storage.root_path as the source of truth for storage UI
root = Path(global_settings.storage.root_path)
if not root.is_absolute(): if not root.is_absolute():
# Resolve relative to the workspace root (ss-tools) # Resolve relative to the backend directory
root = (Path(__file__).parents[4] / root).resolve() # Path(__file__) is backend/src/plugins/storage/plugin.py
# parents[3] is the project root (ss-tools)
# We need to ensure it's relative to where backend/ is
project_root = Path(__file__).parents[3]
root = (project_root / root).resolve()
return root return root
# [/DEF:get_storage_root:Function] # [/DEF:get_storage_root:Function]
# [DEF:resolve_path:Function]
# @PURPOSE: Resolves a dynamic path pattern using provided variables.
# @PARAM: pattern (str) - The path pattern to resolve.
# @PARAM: variables (Dict[str, str]) - Variables to substitute in the pattern.
# @RETURN: str - The resolved path.
def resolve_path(self, pattern: str, variables: Dict[str, str]) -> str:
with belief_scope("StoragePlugin:resolve_path"):
# Add common variables
vars_with_defaults = {
"timestamp": datetime.now().strftime("%Y%m%dT%H%M%S"),
**variables
}
try:
resolved = pattern.format(**vars_with_defaults)
# Clean up any double slashes or leading/trailing slashes for relative path
return os.path.normpath(resolved).strip("/")
except KeyError as e:
logger.warning(f"[StoragePlugin][Coherence:Failed] Missing variable for path resolution: {e}")
# Fallback to literal pattern if formatting fails partially (or handle as needed)
return pattern.replace("{", "").replace("}", "")
# [/DEF:resolve_path:Function]
# [DEF:ensure_directories:Function] # [DEF:ensure_directories:Function]
# @PURPOSE: Creates the storage root and category subdirectories if they don't exist. # @PURPOSE: Creates the storage root and category subdirectories if they don't exist.
# @SIDE_EFFECT: Creates directories on the filesystem. # @SIDE_EFFECT: Creates directories on the filesystem.
@@ -113,7 +142,8 @@ class StoragePlugin(PluginBase):
with belief_scope("StoragePlugin:ensure_directories"): with belief_scope("StoragePlugin:ensure_directories"):
root = self.get_storage_root() root = self.get_storage_root()
for category in FileCategory: for category in FileCategory:
path = root / f"{category.value}s" # Use singular name for consistency with BackupPlugin and GitService
path = root / category.value
path.mkdir(parents=True, exist_ok=True) path.mkdir(parents=True, exist_ok=True)
logger.debug(f"[StoragePlugin][Action] Ensured directory: {path}") logger.debug(f"[StoragePlugin][Action] Ensured directory: {path}")
# [/DEF:ensure_directories:Function] # [/DEF:ensure_directories:Function]
@@ -135,46 +165,68 @@ class StoragePlugin(PluginBase):
# [/DEF:validate_path:Function] # [/DEF:validate_path:Function]
# [DEF:list_files:Function] # [DEF:list_files:Function]
# @PURPOSE: Lists all files in a specific category. # @PURPOSE: Lists all files and directories in a specific category and subpath.
# @PARAM: category (Optional[FileCategory]) - The category to list. # @PARAM: category (Optional[FileCategory]) - The category to list.
# @RETURN: List[StoredFile] - List of file metadata objects. # @PARAM: subpath (Optional[str]) - Nested path within the category.
def list_files(self, category: Optional[FileCategory] = None) -> List[StoredFile]: # @RETURN: List[StoredFile] - List of file and directory metadata objects.
def list_files(self, category: Optional[FileCategory] = None, subpath: Optional[str] = None) -> List[StoredFile]:
with belief_scope("StoragePlugin:list_files"): with belief_scope("StoragePlugin:list_files"):
root = self.get_storage_root() root = self.get_storage_root()
logger.info(f"[StoragePlugin][Action] Listing files in root: {root}, category: {category}, subpath: {subpath}")
files = [] files = []
categories = [category] if category else list(FileCategory) categories = [category] if category else list(FileCategory)
for cat in categories: for cat in categories:
cat_dir = root / f"{cat.value}s" # Scan the category subfolder + optional subpath
if not cat_dir.exists(): base_dir = root / cat.value
if subpath:
target_dir = self.validate_path(base_dir / subpath)
else:
target_dir = base_dir
if not target_dir.exists():
continue continue
for item in cat_dir.iterdir(): logger.debug(f"[StoragePlugin][Action] Scanning directory: {target_dir}")
if item.is_file():
stat = item.stat() # Use os.scandir for better performance and to distinguish files vs dirs
with os.scandir(target_dir) as it:
for entry in it:
# Skip logs
if "Logs" in entry.path:
continue
stat = entry.stat()
is_dir = entry.is_dir()
files.append(StoredFile( files.append(StoredFile(
name=item.name, name=entry.name,
path=str(item.relative_to(root)), path=str(Path(entry.path).relative_to(root)),
size=stat.st_size, size=stat.st_size if not is_dir else 0,
created_at=datetime.fromtimestamp(stat.st_ctime), created_at=datetime.fromtimestamp(stat.st_ctime),
category=cat, category=cat,
mime_type=None # Could use python-magic here if needed mime_type="directory" if is_dir else None
)) ))
return sorted(files, key=lambda x: x.created_at, reverse=True) # Sort: directories first, then by name
return sorted(files, key=lambda x: (x.mime_type != "directory", x.name))
# [/DEF:list_files:Function] # [/DEF:list_files:Function]
# [DEF:save_file:Function] # [DEF:save_file:Function]
# @PURPOSE: Saves an uploaded file to the specified category. # @PURPOSE: Saves an uploaded file to the specified category and optional subpath.
# @PARAM: file (UploadFile) - The uploaded file. # @PARAM: file (UploadFile) - The uploaded file.
# @PARAM: category (FileCategory) - The target category. # @PARAM: category (FileCategory) - The target category.
# @PARAM: subpath (Optional[str]) - The target subpath.
# @RETURN: StoredFile - Metadata of the saved file. # @RETURN: StoredFile - Metadata of the saved file.
# @SIDE_EFFECT: Writes file to disk. # @SIDE_EFFECT: Writes file to disk.
async def save_file(self, file: UploadFile, category: FileCategory) -> StoredFile: async def save_file(self, file: UploadFile, category: FileCategory, subpath: Optional[str] = None) -> StoredFile:
with belief_scope("StoragePlugin:save_file"): with belief_scope("StoragePlugin:save_file"):
root = self.get_storage_root() root = self.get_storage_root()
dest_dir = root / f"{category.value}s" dest_dir = root / category.value
if subpath:
dest_dir = dest_dir / subpath
dest_dir.mkdir(parents=True, exist_ok=True) dest_dir.mkdir(parents=True, exist_ok=True)
dest_path = self.validate_path(dest_dir / file.filename) dest_path = self.validate_path(dest_dir / file.filename)
@@ -194,34 +246,44 @@ class StoragePlugin(PluginBase):
# [/DEF:save_file:Function] # [/DEF:save_file:Function]
# [DEF:delete_file:Function] # [DEF:delete_file:Function]
# @PURPOSE: Deletes a file from the specified category. # @PURPOSE: Deletes a file or directory from the specified category and path.
# @PARAM: category (FileCategory) - The category. # @PARAM: category (FileCategory) - The category.
# @PARAM: filename (str) - The name of the file. # @PARAM: path (str) - The relative path of the file or directory.
# @SIDE_EFFECT: Removes file from disk. # @SIDE_EFFECT: Removes item from disk.
def delete_file(self, category: FileCategory, filename: str): def delete_file(self, category: FileCategory, path: str):
with belief_scope("StoragePlugin:delete_file"): with belief_scope("StoragePlugin:delete_file"):
root = self.get_storage_root() root = self.get_storage_root()
file_path = self.validate_path(root / f"{category.value}s" / filename) # path is relative to root, but we ensure it starts with category
full_path = self.validate_path(root / path)
if file_path.exists(): if not str(Path(path)).startswith(category.value):
file_path.unlink() raise ValueError(f"Path {path} does not belong to category {category}")
logger.info(f"[StoragePlugin][Action] Deleted file: {file_path}")
if full_path.exists():
if full_path.is_dir():
shutil.rmtree(full_path)
else: else:
raise FileNotFoundError(f"File {filename} not found in {category.value}s") full_path.unlink()
logger.info(f"[StoragePlugin][Action] Deleted: {full_path}")
else:
raise FileNotFoundError(f"Item {path} not found")
# [/DEF:delete_file:Function] # [/DEF:delete_file:Function]
# [DEF:get_file_path:Function] # [DEF:get_file_path:Function]
# @PURPOSE: Returns the absolute path of a file for download. # @PURPOSE: Returns the absolute path of a file for download.
# @PARAM: category (FileCategory) - The category. # @PARAM: category (FileCategory) - The category.
# @PARAM: filename (str) - The name of the file. # @PARAM: path (str) - The relative path of the file.
# @RETURN: Path - Absolute path to the file. # @RETURN: Path - Absolute path to the file.
def get_file_path(self, category: FileCategory, filename: str) -> Path: def get_file_path(self, category: FileCategory, path: str) -> Path:
with belief_scope("StoragePlugin:get_file_path"): with belief_scope("StoragePlugin:get_file_path"):
root = self.get_storage_root() root = self.get_storage_root()
file_path = self.validate_path(root / f"{category.value}s" / filename) file_path = self.validate_path(root / path)
if not file_path.exists(): if not str(Path(path)).startswith(category.value):
raise FileNotFoundError(f"File {filename} not found in {category.value}s") raise ValueError(f"Path {path} does not belong to category {category}")
if not file_path.exists() or file_path.is_dir():
raise FileNotFoundError(f"File {path} not found")
return file_path return file_path
# [/DEF:get_file_path:Function] # [/DEF:get_file_path:Function]

View File

@@ -31,9 +31,15 @@ class GitService:
# @PARAM: base_path (str) - Root directory for all Git clones. # @PARAM: base_path (str) - Root directory for all Git clones.
# @PRE: base_path is a valid string path. # @PRE: base_path is a valid string path.
# @POST: GitService is initialized; base_path directory exists. # @POST: GitService is initialized; base_path directory exists.
def __init__(self, base_path: str = "backend/git_repos"): def __init__(self, base_path: str = "git_repos"):
with belief_scope("GitService.__init__"): with belief_scope("GitService.__init__"):
self.base_path = base_path # Resolve relative to the backend directory
# Path(__file__) is backend/src/services/git_service.py
# parents[2] is backend/
from pathlib import Path
backend_root = Path(__file__).parents[2]
self.base_path = str((backend_root / base_path).resolve())
if not os.path.exists(self.base_path): if not os.path.exists(self.base_path):
os.makedirs(self.base_path) os.makedirs(self.base_path)
# [/DEF:__init__:Function] # [/DEF:__init__:Function]

Binary file not shown.

View File

@@ -13,7 +13,7 @@ The settings mechanism allows users to configure multiple Superset environments
Configuration is structured using Pydantic models in `backend/src/core/config_models.py`: Configuration is structured using Pydantic models in `backend/src/core/config_models.py`:
- `Environment`: Represents a Superset instance (URL, credentials). The `base_url` is automatically normalized to include the `/api/v1` suffix if missing. - `Environment`: Represents a Superset instance (URL, credentials). The `base_url` is automatically normalized to include the `/api/v1` suffix if missing.
- `GlobalSettings`: Global application parameters (e.g., `backup_path`). - `GlobalSettings`: Global application parameters (e.g., `storage.root_path`).
- `AppConfig`: The root configuration object. - `AppConfig`: The root configuration object.
### Configuration Manager ### Configuration Manager
@@ -43,4 +43,4 @@ The settings page is located at `frontend/src/pages/Settings.svelte`. It provide
Existing plugins and utilities use the `ConfigManager` to fetch configuration: Existing plugins and utilities use the `ConfigManager` to fetch configuration:
- `superset_tool/utils/init_clients.py`: Dynamically initializes Superset clients from the configured environments. - `superset_tool/utils/init_clients.py`: Dynamically initializes Superset clients from the configured environments.
- `BackupPlugin`: Uses the configured `backup_path` as the default storage location. - `BackupPlugin`: Uses the configured `storage.root_path` as the default storage location.

View File

@@ -13,11 +13,16 @@
// [SECTION: IMPORTS] // [SECTION: IMPORTS]
import { createEventDispatcher } from 'svelte'; import { createEventDispatcher } from 'svelte';
import { downloadFileUrl } from '../../services/storageService'; import { downloadFileUrl } from '../../services/storageService';
import { t } from '../../lib/i18n';
// [/SECTION: IMPORTS] // [/SECTION: IMPORTS]
export let files = []; export let files = [];
const dispatch = createEventDispatcher(); const dispatch = createEventDispatcher();
function isDirectory(file) {
return file.mime_type === 'directory';
}
function formatSize(bytes) { function formatSize(bytes) {
if (bytes === 0) return '0 B'; if (bytes === 0) return '0 B';
const k = 1024; const k = 1024;
@@ -36,40 +41,63 @@
<table class="min-w-full bg-white border border-gray-200"> <table class="min-w-full bg-white border border-gray-200">
<thead class="bg-gray-50"> <thead class="bg-gray-50">
<tr> <tr>
<th class="px-6 py-3 text-left text-xs font-medium text-gray-500 uppercase tracking-wider">Name</th> <th class="px-6 py-3 text-left text-xs font-medium text-gray-500 uppercase tracking-wider">{$t.storage.table.name}</th>
<th class="px-6 py-3 text-left text-xs font-medium text-gray-500 uppercase tracking-wider">Category</th> <th class="px-6 py-3 text-left text-xs font-medium text-gray-500 uppercase tracking-wider">{$t.storage.table.category}</th>
<th class="px-6 py-3 text-left text-xs font-medium text-gray-500 uppercase tracking-wider">Size</th> <th class="px-6 py-3 text-left text-xs font-medium text-gray-500 uppercase tracking-wider">{$t.storage.table.size}</th>
<th class="px-6 py-3 text-left text-xs font-medium text-gray-500 uppercase tracking-wider">Created At</th> <th class="px-6 py-3 text-left text-xs font-medium text-gray-500 uppercase tracking-wider">{$t.storage.table.created_at}</th>
<th class="px-6 py-3 text-right text-xs font-medium text-gray-500 uppercase tracking-wider">Actions</th> <th class="px-6 py-3 text-right text-xs font-medium text-gray-500 uppercase tracking-wider">{$t.storage.table.actions}</th>
</tr> </tr>
</thead> </thead>
<tbody class="divide-y divide-gray-200"> <tbody class="divide-y divide-gray-200">
{#each files as file} {#each files as file}
<tr class="hover:bg-gray-50"> <tr class="hover:bg-gray-50">
<td class="px-6 py-4 whitespace-nowrap text-sm font-medium text-gray-900">{file.name}</td> <td class="px-6 py-4 whitespace-nowrap text-sm font-medium text-gray-900">
{#if isDirectory(file)}
<button
on:click={() => dispatch('navigate', file.path)}
class="flex items-center text-indigo-600 hover:text-indigo-900"
>
<svg class="h-5 w-5 mr-2 text-yellow-400" fill="currentColor" viewBox="0 0 20 20">
<path d="M2 6a2 2 0 012-2h5l2 2h5a2 2 0 012 2v6a2 2 0 01-2 2H4a2 2 0 01-2-2V6z" />
</svg>
{file.name}
</button>
{:else}
<div class="flex items-center">
<svg class="h-5 w-5 mr-2 text-gray-400" fill="none" stroke="currentColor" viewBox="0 0 24 24">
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M7 21h10a2 2 0 002-2V9.414a1 1 0 00-.293-.707l-5.414-5.414A1 1 0 0012.586 3H7a2 2 0 00-2 2v14a2 2 0 002 2z" />
</svg>
{file.name}
</div>
{/if}
</td>
<td class="px-6 py-4 whitespace-nowrap text-sm text-gray-500 capitalize">{file.category}</td> <td class="px-6 py-4 whitespace-nowrap text-sm text-gray-500 capitalize">{file.category}</td>
<td class="px-6 py-4 whitespace-nowrap text-sm text-gray-500">{formatSize(file.size)}</td> <td class="px-6 py-4 whitespace-nowrap text-sm text-gray-500">
{isDirectory(file) ? '--' : formatSize(file.size)}
</td>
<td class="px-6 py-4 whitespace-nowrap text-sm text-gray-500">{formatDate(file.created_at)}</td> <td class="px-6 py-4 whitespace-nowrap text-sm text-gray-500">{formatDate(file.created_at)}</td>
<td class="px-6 py-4 whitespace-nowrap text-right text-sm font-medium"> <td class="px-6 py-4 whitespace-nowrap text-right text-sm font-medium">
{#if !isDirectory(file)}
<a <a
href={downloadFileUrl(file.category, file.name)} href={downloadFileUrl(file.category, file.path)}
download={file.name} download={file.name}
class="text-indigo-600 hover:text-indigo-900 mr-4" class="text-indigo-600 hover:text-indigo-900 mr-4"
> >
Download {$t.storage.table.download}
</a> </a>
{/if}
<button <button
on:click={() => dispatch('delete', { category: file.category, filename: file.name })} on:click={() => dispatch('delete', { category: file.category, path: file.path, name: file.name })}
class="text-red-600 hover:text-red-900" class="text-red-600 hover:text-red-900"
> >
Delete {$t.storage.table.delete}
</button> </button>
</td> </td>
</tr> </tr>
{:else} {:else}
<tr> <tr>
<td colspan="5" class="px-6 py-10 text-center text-sm text-gray-500"> <td colspan="5" class="px-6 py-10 text-center text-sm text-gray-500">
No files found. {$t.storage.no_files}
</td> </td>
</tr> </tr>
{/each} {/each}

View File

@@ -14,6 +14,7 @@
import { createEventDispatcher } from 'svelte'; import { createEventDispatcher } from 'svelte';
import { uploadFile } from '../../services/storageService'; import { uploadFile } from '../../services/storageService';
import { addToast } from '../../lib/toasts'; import { addToast } from '../../lib/toasts';
import { t } from '../../lib/i18n';
// [/SECTION: IMPORTS] // [/SECTION: IMPORTS]
// [DEF:handleUpload:Function] // [DEF:handleUpload:Function]
@@ -24,7 +25,8 @@
*/ */
const dispatch = createEventDispatcher(); const dispatch = createEventDispatcher();
let fileInput; let fileInput;
let category = 'backup'; export let category = 'backups';
export let path = '';
let isUploading = false; let isUploading = false;
let dragOver = false; let dragOver = false;
@@ -34,12 +36,18 @@
isUploading = true; isUploading = true;
try { try {
await uploadFile(file, category); // path is relative to root, but upload endpoint expects path within category
addToast(`File ${file.name} uploaded successfully.`, 'success'); // FileList.path is like "backup/folder", we need just "folder"
const subpath = path.startsWith(category)
? path.substring(category.length).replace(/^\/+/, '')
: path;
await uploadFile(file, category, subpath);
addToast($t.storage.messages.upload_success.replace('{name}', file.name), 'success');
fileInput.value = ''; fileInput.value = '';
dispatch('uploaded'); dispatch('uploaded');
} catch (error) { } catch (error) {
addToast(`Upload failed: ${error.message}`, 'error'); addToast($t.storage.messages.upload_failed.replace('{error}', error.message), 'error');
} finally { } finally {
isUploading = false; isUploading = false;
} }
@@ -65,17 +73,17 @@
<!-- [SECTION: TEMPLATE] --> <!-- [SECTION: TEMPLATE] -->
<div class="bg-white p-6 rounded-lg border border-gray-200 shadow-sm"> <div class="bg-white p-6 rounded-lg border border-gray-200 shadow-sm">
<h2 class="text-lg font-semibold mb-4">Upload File</h2> <h2 class="text-lg font-semibold mb-4">{$t.storage.upload_title}</h2>
<div class="space-y-4"> <div class="space-y-4">
<div> <div>
<label class="block text-sm font-medium text-gray-700 mb-1">Target Category</label> <label class="block text-sm font-medium text-gray-700 mb-1">{$t.storage.target_category}</label>
<select <select
bind:value={category} bind:value={category}
class="block w-full border-gray-300 rounded-md shadow-sm focus:ring-indigo-500 focus:border-indigo-500 sm:text-sm" class="block w-full border-gray-300 rounded-md shadow-sm focus:ring-indigo-500 focus:border-indigo-500 sm:text-sm"
> >
<option value="backup">Backup</option> <option value="backups">{$t.storage.backups}</option>
<option value="repository">Repository</option> <option value="repositorys">{$t.storage.repositories}</option>
</select> </select>
</div> </div>
@@ -92,7 +100,7 @@
</svg> </svg>
<div class="flex text-sm text-gray-600"> <div class="flex text-sm text-gray-600">
<label for="file-upload" class="relative cursor-pointer bg-white rounded-md font-medium text-indigo-600 hover:text-indigo-500 focus-within:outline-none focus-within:ring-2 focus-within:ring-offset-2 focus-within:ring-indigo-500"> <label for="file-upload" class="relative cursor-pointer bg-white rounded-md font-medium text-indigo-600 hover:text-indigo-500 focus-within:outline-none focus-within:ring-2 focus-within:ring-offset-2 focus-within:ring-indigo-500">
<span>Upload a file</span> <span>{$t.storage.upload_button}</span>
<input <input
id="file-upload" id="file-upload"
name="file-upload" name="file-upload"
@@ -103,16 +111,16 @@
disabled={isUploading} disabled={isUploading}
> >
</label> </label>
<p class="pl-1">or drag and drop</p> <p class="pl-1">{$t.storage.drag_drop}</p>
</div> </div>
<p class="text-xs text-gray-500">ZIP, YAML, JSON up to 50MB</p> <p class="text-xs text-gray-500">{$t.storage.supported_formats}</p>
</div> </div>
</div> </div>
{#if isUploading} {#if isUploading}
<div class="flex items-center justify-center space-x-2 text-indigo-600"> <div class="flex items-center justify-center space-x-2 text-indigo-600">
<div class="animate-spin rounded-full h-4 w-4 border-b-2 border-indigo-600"></div> <div class="animate-spin rounded-full h-4 w-4 border-b-2 border-indigo-600"></div>
<span class="text-sm font-medium">Uploading...</span> <span class="text-sm font-medium">{$t.storage.uploading}</span>
</div> </div>
{/if} {/if}
</div> </div>

View File

@@ -20,7 +20,6 @@
let settings = { let settings = {
environments: [], environments: [],
settings: { settings: {
backup_path: '',
default_environment_id: null, default_environment_id: null,
logging: { logging: {
level: 'INFO', level: 'INFO',
@@ -204,12 +203,6 @@
<section class="mb-8 bg-white p-6 rounded shadow"> <section class="mb-8 bg-white p-6 rounded shadow">
<h2 class="text-xl font-semibold mb-4">Global Settings</h2> <h2 class="text-xl font-semibold mb-4">Global Settings</h2>
<div class="grid grid-cols-1 gap-4">
<div>
<label for="backup_path" class="block text-sm font-medium text-gray-700">Backup Storage Path</label>
<input type="text" id="backup_path" bind:value={settings.settings.backup_path} class="mt-1 block w-full border border-gray-300 rounded-md shadow-sm p-2" />
</div>
</div>
<h3 class="text-lg font-medium mb-4 mt-6">Logging Configuration</h3> <h3 class="text-lg font-medium mb-4 mt-6">Logging Configuration</h3>
<div class="grid grid-cols-1 md:grid-cols-2 gap-4"> <div class="grid grid-cols-1 md:grid-cols-2 gap-4">

View File

@@ -170,19 +170,6 @@
</div> </div>
{/if} {/if}
<div class="mb-8">
<Card title={$t.settings?.global_title || "Global Settings"}>
<div class="grid grid-cols-1 gap-6">
<Input
label={$t.settings?.backup_path || "Backup Storage Path"}
bind:value={settings.settings.backup_path}
/>
<Button on:click={handleSaveGlobal}>
{$t.common.save}
</Button>
</div>
</Card>
</div>
<div class="mb-8"> <div class="mb-8">
<Card title={$t.settings?.storage_title || "File Storage Configuration"}> <Card title={$t.settings?.storage_title || "File Storage Configuration"}>

View File

@@ -18,7 +18,6 @@ export async function load() {
settings: { settings: {
environments: [], environments: [],
settings: { settings: {
backup_path: '',
default_environment_id: null default_environment_id: null
} }
}, },

View File

@@ -15,6 +15,7 @@
import { onMount } from 'svelte'; import { onMount } from 'svelte';
import { listFiles, deleteFile } from '../../../services/storageService'; import { listFiles, deleteFile } from '../../../services/storageService';
import { addToast } from '../../../lib/toasts'; import { addToast } from '../../../lib/toasts';
import { t } from '../../../lib/i18n';
import FileList from '../../../components/storage/FileList.svelte'; import FileList from '../../../components/storage/FileList.svelte';
import FileUpload from '../../../components/storage/FileUpload.svelte'; import FileUpload from '../../../components/storage/FileUpload.svelte';
// [/SECTION: IMPORTS] // [/SECTION: IMPORTS]
@@ -26,15 +27,30 @@
*/ */
let files = []; let files = [];
let isLoading = false; let isLoading = false;
let activeTab = 'all'; let activeTab = 'backups';
let currentPath = 'backups'; // Relative to storage root
async function loadFiles() { async function loadFiles() {
isLoading = true; isLoading = true;
try { try {
const category = activeTab === 'all' ? null : activeTab; const category = activeTab;
files = await listFiles(category);
// If we have a currentPath, we use it.
// But if user switched tabs, we should reset currentPath to category root
let effectivePath = currentPath;
if (category && !currentPath.startsWith(category)) {
effectivePath = category;
currentPath = category;
}
// API expects path relative to category root if category is provided
const subpath = (category && effectivePath.startsWith(category))
? effectivePath.substring(category.length).replace(/^\/+/, '')
: effectivePath;
files = await listFiles(category, subpath);
} catch (error) { } catch (error) {
addToast(`Failed to load files: ${error.message}`, 'error'); addToast($t.storage.messages.load_failed.replace('{error}', error.message), 'error');
} finally { } finally {
isLoading = false; isLoading = false;
} }
@@ -44,39 +60,73 @@
// [DEF:handleDelete:Function] // [DEF:handleDelete:Function]
/** /**
* @purpose Handles the file deletion process. * @purpose Handles the file deletion process.
* @param {CustomEvent} event - The delete event containing category and filename. * @param {CustomEvent} event - The delete event containing category and path.
*/ */
async function handleDelete(event) { async function handleDelete(event) {
const { category, filename } = event.detail; const { category, path, name } = event.detail;
if (!confirm(`Are you sure you want to delete ${filename}?`)) return; if (!confirm($t.storage.messages.delete_confirm.replace('{name}', name))) return;
try { try {
await deleteFile(category, filename); await deleteFile(category, path);
addToast(`File ${filename} deleted.`, 'success'); addToast($t.storage.messages.delete_success.replace('{name}', name), 'success');
await loadFiles(); await loadFiles();
} catch (error) { } catch (error) {
addToast(`Delete failed: ${error.message}`, 'error'); addToast($t.storage.messages.delete_failed.replace('{error}', error.message), 'error');
} }
} }
// [/DEF:handleDelete:Function] // [/DEF:handleDelete:Function]
function handleNavigate(event) {
currentPath = event.detail;
loadFiles();
}
function navigateUp() {
if (!currentPath || currentPath === activeTab) return;
const parts = currentPath.split('/');
parts.pop();
currentPath = parts.join('/') || '';
loadFiles();
}
onMount(loadFiles); onMount(loadFiles);
$: if (activeTab) { $: if (activeTab) {
// Reset path when switching tabs
if (!currentPath.startsWith(activeTab)) {
currentPath = activeTab;
}
loadFiles(); loadFiles();
} }
</script> </script>
<!-- [SECTION: TEMPLATE] --> <!-- [SECTION: TEMPLATE] -->
<div class="container mx-auto p-4 max-w-6xl"> <div class="container mx-auto p-4 max-w-6xl">
<div class="flex justify-between items-center mb-6"> <div class="mb-6">
<h1 class="text-2xl font-bold text-gray-900">File Storage Management</h1> <h1 class="text-2xl font-bold text-gray-900">{$t.storage.management}</h1>
{#if currentPath}
<div class="flex items-center mt-2 text-sm text-gray-500">
<button on:click={() => { currentPath = activeTab; loadFiles(); }} class="hover:text-indigo-600">{$t.storage.root}</button>
{#each currentPath.split('/').slice(1) as part, i}
<span class="mx-2">/</span>
<button
on:click={() => { currentPath = currentPath.split('/').slice(0, i + 1).join('/'); loadFiles(); }}
class="hover:text-indigo-600 capitalize"
>
{part}
</button>
{/each}
</div>
{/if}
</div>
<div class="flex justify-end mb-4">
<button <button
on:click={loadFiles} on:click={loadFiles}
class="inline-flex items-center px-4 py-2 border border-gray-300 rounded-md shadow-sm text-sm font-medium text-gray-700 bg-white hover:bg-gray-50" class="inline-flex items-center px-4 py-2 border border-gray-300 rounded-md shadow-sm text-sm font-medium text-gray-700 bg-white hover:bg-gray-50"
disabled={isLoading} disabled={isLoading}
> >
{isLoading ? 'Refreshing...' : 'Refresh'} {isLoading ? $t.storage.refreshing : $t.storage.refresh}
</button> </button>
</div> </div>
@@ -87,32 +137,44 @@
<div class="border-b border-gray-200"> <div class="border-b border-gray-200">
<nav class="-mb-px flex space-x-8"> <nav class="-mb-px flex space-x-8">
<button <button
on:click={() => activeTab = 'all'} on:click={() => activeTab = 'backups'}
class="py-4 px-1 border-b-2 font-medium text-sm {activeTab === 'all' ? 'border-indigo-500 text-indigo-600' : 'border-transparent text-gray-500 hover:text-gray-700 hover:border-gray-300'}" class="py-4 px-1 border-b-2 font-medium text-sm {activeTab === 'backups' ? 'border-indigo-500 text-indigo-600' : 'border-transparent text-gray-500 hover:text-gray-700 hover:border-gray-300'}"
> >
All Files {$t.storage.backups}
</button> </button>
<button <button
on:click={() => activeTab = 'backup'} on:click={() => activeTab = 'repositorys'}
class="py-4 px-1 border-b-2 font-medium text-sm {activeTab === 'backup' ? 'border-indigo-500 text-indigo-600' : 'border-transparent text-gray-500 hover:text-gray-700 hover:border-gray-300'}" class="py-4 px-1 border-b-2 font-medium text-sm {activeTab === 'repositorys' ? 'border-indigo-500 text-indigo-600' : 'border-transparent text-gray-500 hover:text-gray-700 hover:border-gray-300'}"
> >
Backups {$t.storage.repositories}
</button>
<button
on:click={() => activeTab = 'repository'}
class="py-4 px-1 border-b-2 font-medium text-sm {activeTab === 'repository' ? 'border-indigo-500 text-indigo-600' : 'border-transparent text-gray-500 hover:text-gray-700 hover:border-gray-300'}"
>
Repositories
</button> </button>
</nav> </nav>
</div> </div>
<FileList {files} on:delete={handleDelete} /> <div class="flex items-center mb-2">
{#if currentPath && currentPath !== activeTab}
<button
on:click={navigateUp}
class="mr-4 inline-flex items-center px-3 py-1 border border-gray-300 shadow-sm text-xs font-medium rounded text-gray-700 bg-white hover:bg-gray-50"
>
<svg class="h-4 w-4 mr-1" fill="none" stroke="currentColor" viewBox="0 0 24 24">
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M10 19l-7-7m0 0l7-7m-7 7h18" />
</svg>
Back
</button>
{/if}
</div>
<FileList {files} on:delete={handleDelete} on:navigate={handleNavigate} />
</div> </div>
<!-- Sidebar: Upload --> <!-- Sidebar: Upload -->
<div class="lg:col-span-1"> <div class="lg:col-span-1">
<FileUpload on:uploaded={loadFiles} /> <FileUpload
category={activeTab}
path={currentPath}
on:uploaded={loadFiles}
/>
</div> </div>
</div> </div>
</div> </div>

View File

@@ -9,15 +9,19 @@ const API_BASE = '/api/storage';
// [DEF:listFiles:Function] // [DEF:listFiles:Function]
/** /**
* @purpose Fetches the list of files for a given category. * @purpose Fetches the list of files for a given category and subpath.
* @param {string} [category] - Optional category filter. * @param {string} [category] - Optional category filter.
* @param {string} [path] - Optional subpath filter.
* @returns {Promise<Array>} * @returns {Promise<Array>}
*/ */
export async function listFiles(category) { export async function listFiles(category, path) {
const params = new URLSearchParams(); const params = new URLSearchParams();
if (category) { if (category) {
params.append('category', category); params.append('category', category);
} }
if (path) {
params.append('path', path);
}
const response = await fetch(`${API_BASE}/files?${params.toString()}`); const response = await fetch(`${API_BASE}/files?${params.toString()}`);
if (!response.ok) { if (!response.ok) {
throw new Error(`Failed to fetch files: ${response.statusText}`); throw new Error(`Failed to fetch files: ${response.statusText}`);
@@ -31,12 +35,16 @@ export async function listFiles(category) {
* @purpose Uploads a file to the storage system. * @purpose Uploads a file to the storage system.
* @param {File} file - The file to upload. * @param {File} file - The file to upload.
* @param {string} category - Target category. * @param {string} category - Target category.
* @param {string} [path] - Target subpath.
* @returns {Promise<Object>} * @returns {Promise<Object>}
*/ */
export async function uploadFile(file, category) { export async function uploadFile(file, category, path) {
const formData = new FormData(); const formData = new FormData();
formData.append('file', file); formData.append('file', file);
formData.append('category', category); formData.append('category', category);
if (path) {
formData.append('path', path);
}
const response = await fetch(`${API_BASE}/upload`, { const response = await fetch(`${API_BASE}/upload`, {
method: 'POST', method: 'POST',
@@ -53,19 +61,19 @@ export async function uploadFile(file, category) {
// [DEF:deleteFile:Function] // [DEF:deleteFile:Function]
/** /**
* @purpose Deletes a file from storage. * @purpose Deletes a file or directory from storage.
* @param {string} category - File category. * @param {string} category - File category.
* @param {string} filename - Name of the file. * @param {string} path - Relative path of the item.
* @returns {Promise<void>} * @returns {Promise<void>}
*/ */
export async function deleteFile(category, filename) { export async function deleteFile(category, path) {
const response = await fetch(`${API_BASE}/files/${category}/${filename}`, { const response = await fetch(`${API_BASE}/files/${category}/${path}`, {
method: 'DELETE' method: 'DELETE'
}); });
if (!response.ok) { if (!response.ok) {
const errorData = await response.json().catch(() => ({})); const errorData = await response.json().catch(() => ({}));
throw new Error(errorData.detail || `Failed to delete file: ${response.statusText}`); throw new Error(errorData.detail || `Failed to delete: ${response.statusText}`);
} }
} }
// [/DEF:deleteFile:Function] // [/DEF:deleteFile:Function]
@@ -74,11 +82,11 @@ export async function deleteFile(category, filename) {
/** /**
* @purpose Returns the URL for downloading a file. * @purpose Returns the URL for downloading a file.
* @param {string} category - File category. * @param {string} category - File category.
* @param {string} filename - Name of the file. * @param {string} path - Relative path of the file.
* @returns {string} * @returns {string}
*/ */
export function downloadFileUrl(category, filename) { export function downloadFileUrl(category, path) {
return `${API_BASE}/download/${category}/${filename}`; return `${API_BASE}/download/${category}/${path}`;
} }
// [/DEF:downloadFileUrl:Function] // [/DEF:downloadFileUrl:Function]

View File

@@ -6,30 +6,30 @@
## User Experience (File Management) ## User Experience (File Management)
- [ ] CHK001 Are loading states displayed while fetching the file list? [Completeness] - [x] CHK001 Are loading states displayed while fetching the file list? [Completeness]
- [ ] CHK002 Is visual feedback provided immediately after file upload starts? [Clarity] - [x] CHK002 Is visual feedback provided immediately after file upload starts? [Clarity]
- [ ] CHK003 Are error messages user-friendly when upload fails (e.g., "File too large" vs "Error 413")? [Clarity] - [x] CHK003 Are error messages user-friendly when upload fails (e.g., "File too large" vs "Error 413")? [Clarity]
- [ ] CHK004 Is a confirmation modal shown before permanently deleting a file? [Safety] - [x] CHK004 Is a confirmation modal shown before permanently deleting a file? [Safety]
- [ ] CHK005 Does the UI clearly distinguish between "Backups" and "Repositories" tabs? [Clarity] - [x] CHK005 Does the UI clearly distinguish between "Backups" and "Repositories" tabs? [Clarity]
- [ ] CHK006 Is the file list sortable by Date and Name? [Usability] - [x] CHK006 Is the file list sortable by Date and Name? [Usability]
- [ ] CHK007 Are file sizes formatted in human-readable units (KB, MB, GB)? [Usability] - [x] CHK007 Are file sizes formatted in human-readable units (KB, MB, GB)? [Usability]
- [ ] CHK008 Is the download action easily accessible for each file item? [Accessibility] - [x] CHK008 Is the download action easily accessible for each file item? [Accessibility]
- [ ] CHK009 Does the upload component support drag-and-drop interactions? [Usability] - [x] CHK009 Does the upload component support drag-and-drop interactions? [Usability]
- [ ] CHK010 Is the "Upload" button disabled or hidden when no category is selected? [Consistency] - [x] CHK010 Is the "Upload" button disabled or hidden when no category is selected? [Consistency]
## Configuration Flexibility ## Configuration Flexibility
- [ ] CHK011 Can the storage root path be configured to any writable directory on the server? [Flexibility] - [x] CHK011 Can the storage root path be configured to any writable directory on the server? [Flexibility]
- [ ] CHK012 Does the system support defining custom directory structures using variables like `{environment}`? [Flexibility] - [x] CHK012 Does the system support defining custom directory structures using variables like `{environment}`? [Flexibility]
- [ ] CHK013 Does the system support defining custom filename patterns using variables like `{timestamp}`? [Flexibility] - [x] CHK013 Does the system support defining custom filename patterns using variables like `{timestamp}`? [Flexibility]
- [ ] CHK014 Are the supported pattern variables (e.g., `{dashboard_name}`) clearly documented in the UI? [Clarity] - [ ] CHK014 Are the supported pattern variables (e.g., `{dashboard_name}`) clearly documented in the UI? [Clarity]
- [ ] CHK015 Can the configuration be updated without restarting the application? [Usability] - [x] CHK015 Can the configuration be updated without restarting the application? [Usability]
- [ ] CHK016 Is the current resolved path shown as a preview when editing patterns? [Usability] - [ ] CHK016 Is the current resolved path shown as a preview when editing patterns? [Usability]
- [ ] CHK017 Does the system allow reverting configuration to default values? [Recovery] - [ ] CHK017 Does the system allow reverting configuration to default values? [Recovery]
## Edge Cases & Error Handling ## Edge Cases & Error Handling
- [ ] CHK018 Is the UI behavior defined for an empty file list (zero state)? [Coverage] - [x] CHK018 Is the UI behavior defined for an empty file list (zero state)? [Coverage]
- [ ] CHK019 Is the behavior defined when the configured storage path becomes inaccessible? [Resilience] - [ ] CHK019 Is the behavior defined when the configured storage path becomes inaccessible? [Resilience]
- [ ] CHK020 Are long filenames handled gracefully in the UI (e.g., truncation with tooltip)? [Layout] - [x] CHK020 Are long filenames handled gracefully in the UI (e.g., truncation with tooltip)? [Layout]
- [ ] CHK021 Is the behavior defined for uploading a file with a duplicate name? [Conflict Resolution] - [x] CHK021 Is the behavior defined for uploading a file with a duplicate name? [Conflict Resolution]

View File

@@ -5,7 +5,7 @@
## Summary ## Summary
This feature implements a managed file storage system for dashboard backups and exported repositories. It introduces a configurable storage root (defaulting to outside the workspace) and a Web UI to list, upload, download, and delete files. The system enforces a structured layout with `backups/` and `repositories/` subdirectories to keep artifacts organized, while allowing flexible configuration of directory structures and filename patterns. This feature implements a managed file storage system for dashboard backups and exported repositories. It introduces a configurable storage root (defaulting to outside the workspace) and a Web UI to list, upload, download, and delete files. The system enforces a structured layout with `backups/` and `repositories/` subdirectories to keep artifacts organized. The UI supports hierarchical folder navigation (e.g., `backups/SS2/DashboardName`), allowing users to browse, download, and manage files within nested directories.
## Technical Context ## Technical Context
@@ -88,7 +88,8 @@ frontend/
│ │ └── +page.svelte # Main storage UI │ │ └── +page.svelte # Main storage UI
│ ├── components/ │ ├── components/
│ │ └── storage/ │ │ └── storage/
│ │ ├── FileList.svelte # Component for listing files │ │ ├── FileList.svelte # Component for listing files and folders (explorer view)
│ │ ├── Breadcrumbs.svelte # Component for navigation
│ │ └── FileUpload.svelte # Component for uploading │ │ └── FileUpload.svelte # Component for uploading
│ └── services/ │ └── services/
│ └── storageService.js # Frontend API client │ └── storageService.js # Frontend API client

View File

@@ -9,18 +9,19 @@
### User Story 1 - File Management Dashboard (Priority: P1) ### User Story 1 - File Management Dashboard (Priority: P1)
Users need a visual interface to manage the artifacts generated by the system (dashboard backups, exported repositories) without needing direct server access. This ensures that non-technical users or users without SSH access can still retrieve or clean up data. Users need a visual interface to manage the artifacts generated by the system (dashboard backups, exported repositories) without needing direct server access. Users must be able to navigate through the folder structure (e.g., `backups/SS2/Sales Dashboard`) to locate specific files.
**Why this priority**: Core functionality requested. Without the UI, the storage mechanism is opaque and hard to use. **Why this priority**: Core functionality requested. Without the UI, the storage mechanism is opaque and hard to use.
**Independent Test**: Can be fully tested by opening the new "File Storage" page, uploading a test file, verifying it appears in the list with correct metadata, downloading it, and then deleting it. **Independent Test**: Can be fully tested by opening the new "File Storage" page, navigating into a subdirectory, uploading a test file, verifying it appears in the list, downloading it, and then deleting it.
**Acceptance Scenarios**: **Acceptance Scenarios**:
1. **Given** the File Storage page is open, **When** I view the list, **Then** I see all files in the configured storage directory with their names, sizes, and creation dates. 1. **Given** the File Storage page is open, **When** I view the list, **Then** I see the top-level folders (e.g., `backups`, `repositories`) or files.
2. **Given** a file exists in the list, **When** I click "Download", **Then** the file is downloaded to my local machine. 2. **Given** I am viewing a folder, **When** I click a subfolder name, **Then** the view updates to show the contents of that subfolder.
3. **Given** a file exists in the list, **When** I click "Delete" and confirm, **Then** the file is removed from the list and the server filesystem. 3. **Given** I am in a subfolder, **When** I click "Download" on a file, **Then** the file is downloaded to my local machine.
4. **Given** I have a file locally, **When** I drag and drop it or use the "Upload" button, **Then** the file is uploaded to the server storage and appears in the list. 4. **Given** a file exists in the list, **When** I click "Delete" and confirm, **Then** the file is removed from the list and the server filesystem.
5. **Given** I have a file locally, **When** I drag and drop it or use the "Upload" button, **Then** the file is uploaded to the current directory and appears in the list.
--- ---
@@ -64,11 +65,13 @@ Administrators need to control where potentially large or sensitive files are st
- **FR-001**: System MUST allow configuring a local filesystem root path for storing artifacts. - **FR-001**: System MUST allow configuring a local filesystem root path for storing artifacts.
- **FR-002**: The default storage path MUST be configured such that it does not interfere with the application's git repository (e.g., a directory outside the workspace or explicitly git-ignored). - **FR-002**: The default storage path MUST be configured such that it does not interfere with the application's git repository (e.g., a directory outside the workspace or explicitly git-ignored).
- **FR-003**: System MUST enforce a directory structure within the storage root: `backups/` for dashboard backups and `repositories/` for exported repositories. - **FR-003**: System MUST enforce a directory structure within the storage root: `backups/` for dashboard backups and `repositories/` for exported repositories.
- **FR-004**: System MUST provide a Web UI to list files, organized by their type (Backup vs Repository). - **FR-004**: System MUST provide a Web UI to list files and folders, organized by their type (Backup vs Repository).
- **FR-005**: System MUST display file metadata in the UI: Filename, Size, Creation Date. - **FR-005**: System MUST display file metadata in the UI: Filename, Size, Creation Date.
- **FR-006**: System MUST allow users to download files from the storage directory via the Web UI. - **FR-006**: System MUST allow users to download files from the storage directory (including subdirectories) via the Web UI.
- **FR-007**: System MUST allow users to delete files from the storage directory via the Web UI. - **FR-007**: System MUST allow users to delete files from the storage directory via the Web UI.
- **FR-008**: System MUST allow users to upload files to the storage directory via the Web UI, requiring them to select the target category (Backup or Repository). - **FR-008**: System MUST allow users to upload files to the specific folder in the storage directory via the Web UI.
- **FR-013**: System MUST support navigating through the directory hierarchy within the allowed categories.
- **FR-014**: System MUST display breadcrumbs or similar navigation aid to show current path.
- **FR-009**: System MUST validate that the configured storage path is accessible and writable. - **FR-009**: System MUST validate that the configured storage path is accessible and writable.
- **FR-010**: System MUST prevent access to files outside the configured storage directory (Path Traversal protection). - **FR-010**: System MUST prevent access to files outside the configured storage directory (Path Traversal protection).
- **FR-011**: System MUST allow configuring the directory structure pattern for backups and repositories (e.g., `{environment}/{dashboard_name}/`). - **FR-011**: System MUST allow configuring the directory structure pattern for backups and repositories (e.g., `{environment}/{dashboard_name}/`).

View File

@@ -64,6 +64,17 @@
- [x] T035 Verify large file upload support (50MB+) in Nginx/FastAPI config if applicable - [x] T035 Verify large file upload support (50MB+) in Nginx/FastAPI config if applicable
- [x] T036 Add confirmation modal for file deletion - [x] T036 Add confirmation modal for file deletion
## Phase 6: Folder Structure Support (Refactor)
*Goal: Enable hierarchical navigation, nested file management, and downloading.*
- [x] T037 Refactor `StoragePlugin.list_files` in `backend/src/plugins/storage/plugin.py` to accept `subpath` and return directories/files
- [x] T038 Refactor `StoragePlugin` methods (`save_file`, `delete_file`, `get_file_path`) to support nested paths
- [x] T039 Update backend endpoints in `backend/src/api/routes/storage.py` (`GET /files`, `POST /upload`, `DELETE /files`, `GET /download`) to accept `path` parameter
- [x] T040 Update `frontend/src/services/storageService.js` to pass `path` argument in all API calls
- [x] T041 Update `frontend/src/components/storage/FileList.svelte` to display folder icons, handle navigation events, and show breadcrumbs
- [x] T042 Update `frontend/src/components/storage/FileUpload.svelte` to upload to the currently active directory
- [x] T043 Update `frontend/src/routes/tools/storage/+page.svelte` to manage current path state and handle navigation logic
## Dependencies ## Dependencies
1. **Phase 1 (Setup)**: No dependencies. 1. **Phase 1 (Setup)**: No dependencies.
@@ -71,6 +82,7 @@
3. **Phase 3 (US1)**: Depends on Phase 2. 3. **Phase 3 (US1)**: Depends on Phase 2.
4. **Phase 4 (US2)**: Depends on Phase 2. Can run parallel to Phase 3. 4. **Phase 4 (US2)**: Depends on Phase 2. Can run parallel to Phase 3.
5. **Phase 5 (Polish)**: Depends on Phase 3 and 4. 5. **Phase 5 (Polish)**: Depends on Phase 3 and 4.
6. **Phase 6 (Refactor)**: Depends on Phase 3.
## Parallel Execution Examples ## Parallel Execution Examples