Файловое хранилище готово
This commit is contained in:
Submodule backend/backend/git_repos/12 updated: d592fa7ed5...f46772443a
@@ -1,269 +0,0 @@
|
||||
2025-12-20 19:55:11,325 - INFO - [BackupPlugin][Entry] Starting backup for superset.
|
||||
2025-12-20 19:55:11,325 - INFO - [setup_clients][Enter] Starting Superset clients initialization.
|
||||
2025-12-20 19:55:11,327 - CRITICAL - [setup_clients][Failure] Critical error during client initialization: 1 validation error for SupersetConfig
|
||||
base_url
|
||||
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
|
||||
For further information visit https://errors.pydantic.dev/2.12/v/value_error
|
||||
Traceback (most recent call last):
|
||||
File "/home/user/ss-tools/superset_tool/utils/init_clients.py", line 43, in setup_clients
|
||||
config = SupersetConfig(
|
||||
^^^^^^^^^^^^^^^
|
||||
File "/home/user/ss-tools/backend/venv/lib/python3.12/site-packages/pydantic/main.py", line 250, in __init__
|
||||
validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
pydantic_core._pydantic_core.ValidationError: 1 validation error for SupersetConfig
|
||||
base_url
|
||||
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
|
||||
For further information visit https://errors.pydantic.dev/2.12/v/value_error
|
||||
2025-12-20 21:01:49,905 - INFO - [BackupPlugin][Entry] Starting backup for superset.
|
||||
2025-12-20 21:01:49,906 - INFO - [setup_clients][Enter] Starting Superset clients initialization.
|
||||
2025-12-20 21:01:49,988 - INFO - [setup_clients][Action] Loading environments from ConfigManager
|
||||
2025-12-20 21:01:49,990 - CRITICAL - [setup_clients][Failure] Critical error during client initialization: 1 validation error for SupersetConfig
|
||||
base_url
|
||||
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
|
||||
For further information visit https://errors.pydantic.dev/2.12/v/value_error
|
||||
Traceback (most recent call last):
|
||||
File "/home/user/ss-tools/superset_tool/utils/init_clients.py", line 66, in setup_clients
|
||||
config = SupersetConfig(
|
||||
^^^^^^^^^^^^^^^
|
||||
File "/home/user/ss-tools/venv/lib/python3.12/site-packages/pydantic/main.py", line 250, in __init__
|
||||
validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
pydantic_core._pydantic_core.ValidationError: 1 validation error for SupersetConfig
|
||||
base_url
|
||||
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
|
||||
For further information visit https://errors.pydantic.dev/2.12/v/value_error
|
||||
2025-12-20 22:42:32,538 - INFO - [BackupPlugin][Entry] Starting backup for superset.
|
||||
2025-12-20 22:42:32,538 - INFO - [setup_clients][Enter] Starting Superset clients initialization.
|
||||
2025-12-20 22:42:32,583 - INFO - [setup_clients][Action] Loading environments from ConfigManager
|
||||
2025-12-20 22:42:32,587 - CRITICAL - [setup_clients][Failure] Critical error during client initialization: 1 validation error for SupersetConfig
|
||||
base_url
|
||||
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
|
||||
For further information visit https://errors.pydantic.dev/2.12/v/value_error
|
||||
Traceback (most recent call last):
|
||||
File "/home/user/ss-tools/superset_tool/utils/init_clients.py", line 66, in setup_clients
|
||||
config = SupersetConfig(
|
||||
^^^^^^^^^^^^^^^
|
||||
File "/home/user/ss-tools/backend/.venv/lib/python3.12/site-packages/pydantic/main.py", line 250, in __init__
|
||||
validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
pydantic_core._pydantic_core.ValidationError: 1 validation error for SupersetConfig
|
||||
base_url
|
||||
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
|
||||
For further information visit https://errors.pydantic.dev/2.12/v/value_error
|
||||
2025-12-20 22:54:29,770 - INFO - [BackupPlugin][Entry] Starting backup for .
|
||||
2025-12-20 22:54:29,771 - INFO - [setup_clients][Enter] Starting Superset clients initialization.
|
||||
2025-12-20 22:54:29,831 - INFO - [setup_clients][Action] Loading environments from ConfigManager
|
||||
2025-12-20 22:54:29,833 - CRITICAL - [setup_clients][Failure] Critical error during client initialization: 1 validation error for SupersetConfig
|
||||
base_url
|
||||
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
|
||||
For further information visit https://errors.pydantic.dev/2.12/v/value_error
|
||||
Traceback (most recent call last):
|
||||
File "/home/user/ss-tools/superset_tool/utils/init_clients.py", line 66, in setup_clients
|
||||
config = SupersetConfig(
|
||||
^^^^^^^^^^^^^^^
|
||||
File "/home/user/ss-tools/backend/.venv/lib/python3.12/site-packages/pydantic/main.py", line 250, in __init__
|
||||
validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
pydantic_core._pydantic_core.ValidationError: 1 validation error for SupersetConfig
|
||||
base_url
|
||||
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
|
||||
For further information visit https://errors.pydantic.dev/2.12/v/value_error
|
||||
2025-12-20 22:54:34,078 - INFO - [BackupPlugin][Entry] Starting backup for superset.
|
||||
2025-12-20 22:54:34,078 - INFO - [setup_clients][Enter] Starting Superset clients initialization.
|
||||
2025-12-20 22:54:34,079 - INFO - [setup_clients][Action] Loading environments from ConfigManager
|
||||
2025-12-20 22:54:34,079 - CRITICAL - [setup_clients][Failure] Critical error during client initialization: 1 validation error for SupersetConfig
|
||||
base_url
|
||||
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
|
||||
For further information visit https://errors.pydantic.dev/2.12/v/value_error
|
||||
Traceback (most recent call last):
|
||||
File "/home/user/ss-tools/superset_tool/utils/init_clients.py", line 66, in setup_clients
|
||||
config = SupersetConfig(
|
||||
^^^^^^^^^^^^^^^
|
||||
File "/home/user/ss-tools/backend/.venv/lib/python3.12/site-packages/pydantic/main.py", line 250, in __init__
|
||||
validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
pydantic_core._pydantic_core.ValidationError: 1 validation error for SupersetConfig
|
||||
base_url
|
||||
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
|
||||
For further information visit https://errors.pydantic.dev/2.12/v/value_error
|
||||
2025-12-20 22:59:25,060 - INFO - [BackupPlugin][Entry] Starting backup for superset.
|
||||
2025-12-20 22:59:25,060 - INFO - [setup_clients][Enter] Starting Superset clients initialization.
|
||||
2025-12-20 22:59:25,114 - INFO - [setup_clients][Action] Loading environments from ConfigManager
|
||||
2025-12-20 22:59:25,117 - CRITICAL - [setup_clients][Failure] Critical error during client initialization: 1 validation error for SupersetConfig
|
||||
base_url
|
||||
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
|
||||
For further information visit https://errors.pydantic.dev/2.12/v/value_error
|
||||
Traceback (most recent call last):
|
||||
File "/home/user/ss-tools/superset_tool/utils/init_clients.py", line 66, in setup_clients
|
||||
config = SupersetConfig(
|
||||
^^^^^^^^^^^^^^^
|
||||
File "/home/user/ss-tools/backend/.venv/lib/python3.12/site-packages/pydantic/main.py", line 250, in __init__
|
||||
validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
pydantic_core._pydantic_core.ValidationError: 1 validation error for SupersetConfig
|
||||
base_url
|
||||
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
|
||||
For further information visit https://errors.pydantic.dev/2.12/v/value_error
|
||||
2025-12-20 23:00:31,156 - INFO - [BackupPlugin][Entry] Starting backup for superset.
|
||||
2025-12-20 23:00:31,156 - INFO - [setup_clients][Enter] Starting Superset clients initialization.
|
||||
2025-12-20 23:00:31,157 - INFO - [setup_clients][Action] Loading environments from ConfigManager
|
||||
2025-12-20 23:00:31,162 - CRITICAL - [setup_clients][Failure] Critical error during client initialization: 1 validation error for SupersetConfig
|
||||
base_url
|
||||
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
|
||||
For further information visit https://errors.pydantic.dev/2.12/v/value_error
|
||||
Traceback (most recent call last):
|
||||
File "/home/user/ss-tools/superset_tool/utils/init_clients.py", line 66, in setup_clients
|
||||
config = SupersetConfig(
|
||||
^^^^^^^^^^^^^^^
|
||||
File "/home/user/ss-tools/backend/.venv/lib/python3.12/site-packages/pydantic/main.py", line 250, in __init__
|
||||
validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
pydantic_core._pydantic_core.ValidationError: 1 validation error for SupersetConfig
|
||||
base_url
|
||||
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
|
||||
For further information visit https://errors.pydantic.dev/2.12/v/value_error
|
||||
2025-12-20 23:00:34,710 - INFO - [BackupPlugin][Entry] Starting backup for superset.
|
||||
2025-12-20 23:00:34,710 - INFO - [setup_clients][Enter] Starting Superset clients initialization.
|
||||
2025-12-20 23:00:34,710 - INFO - [setup_clients][Action] Loading environments from ConfigManager
|
||||
2025-12-20 23:00:34,711 - CRITICAL - [setup_clients][Failure] Critical error during client initialization: 1 validation error for SupersetConfig
|
||||
base_url
|
||||
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
|
||||
For further information visit https://errors.pydantic.dev/2.12/v/value_error
|
||||
Traceback (most recent call last):
|
||||
File "/home/user/ss-tools/superset_tool/utils/init_clients.py", line 66, in setup_clients
|
||||
config = SupersetConfig(
|
||||
^^^^^^^^^^^^^^^
|
||||
File "/home/user/ss-tools/backend/.venv/lib/python3.12/site-packages/pydantic/main.py", line 250, in __init__
|
||||
validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
pydantic_core._pydantic_core.ValidationError: 1 validation error for SupersetConfig
|
||||
base_url
|
||||
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
|
||||
For further information visit https://errors.pydantic.dev/2.12/v/value_error
|
||||
2025-12-20 23:01:43,894 - INFO - [BackupPlugin][Entry] Starting backup for superset.
|
||||
2025-12-20 23:01:43,894 - INFO - [setup_clients][Enter] Starting Superset clients initialization.
|
||||
2025-12-20 23:01:43,895 - INFO - [setup_clients][Action] Loading environments from ConfigManager
|
||||
2025-12-20 23:01:43,895 - CRITICAL - [setup_clients][Failure] Critical error during client initialization: 1 validation error for SupersetConfig
|
||||
base_url
|
||||
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
|
||||
For further information visit https://errors.pydantic.dev/2.12/v/value_error
|
||||
Traceback (most recent call last):
|
||||
File "/home/user/ss-tools/superset_tool/utils/init_clients.py", line 66, in setup_clients
|
||||
config = SupersetConfig(
|
||||
^^^^^^^^^^^^^^^
|
||||
File "/home/user/ss-tools/backend/.venv/lib/python3.12/site-packages/pydantic/main.py", line 250, in __init__
|
||||
validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
pydantic_core._pydantic_core.ValidationError: 1 validation error for SupersetConfig
|
||||
base_url
|
||||
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
|
||||
For further information visit https://errors.pydantic.dev/2.12/v/value_error
|
||||
2025-12-20 23:04:07,731 - INFO - [BackupPlugin][Entry] Starting backup for superset.
|
||||
2025-12-20 23:04:07,731 - INFO - [setup_clients][Enter] Starting Superset clients initialization.
|
||||
2025-12-20 23:04:07,732 - INFO - [setup_clients][Action] Loading environments from ConfigManager
|
||||
2025-12-20 23:04:07,732 - CRITICAL - [setup_clients][Failure] Critical error during client initialization: 1 validation error for SupersetConfig
|
||||
base_url
|
||||
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
|
||||
For further information visit https://errors.pydantic.dev/2.12/v/value_error
|
||||
Traceback (most recent call last):
|
||||
File "/home/user/ss-tools/superset_tool/utils/init_clients.py", line 66, in setup_clients
|
||||
config = SupersetConfig(
|
||||
^^^^^^^^^^^^^^^
|
||||
File "/home/user/ss-tools/backend/.venv/lib/python3.12/site-packages/pydantic/main.py", line 250, in __init__
|
||||
validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
pydantic_core._pydantic_core.ValidationError: 1 validation error for SupersetConfig
|
||||
base_url
|
||||
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
|
||||
For further information visit https://errors.pydantic.dev/2.12/v/value_error
|
||||
2025-12-20 23:06:39,641 - INFO - [BackupPlugin][Entry] Starting backup for superset.
|
||||
2025-12-20 23:06:39,642 - INFO - [setup_clients][Enter] Starting Superset clients initialization.
|
||||
2025-12-20 23:06:39,687 - INFO - [setup_clients][Action] Loading environments from ConfigManager
|
||||
2025-12-20 23:06:39,689 - CRITICAL - [setup_clients][Failure] Critical error during client initialization: 1 validation error for SupersetConfig
|
||||
base_url
|
||||
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
|
||||
For further information visit https://errors.pydantic.dev/2.12/v/value_error
|
||||
Traceback (most recent call last):
|
||||
File "/home/user/ss-tools/superset_tool/utils/init_clients.py", line 66, in setup_clients
|
||||
config = SupersetConfig(
|
||||
^^^^^^^^^^^^^^^
|
||||
File "/home/user/ss-tools/backend/.venv/lib/python3.12/site-packages/pydantic/main.py", line 250, in __init__
|
||||
validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
pydantic_core._pydantic_core.ValidationError: 1 validation error for SupersetConfig
|
||||
base_url
|
||||
Value error, Invalid URL format: https://superset.bebesh.ru. Must include '/api/v1'. [type=value_error, input_value='https://superset.bebesh.ru', input_type=str]
|
||||
For further information visit https://errors.pydantic.dev/2.12/v/value_error
|
||||
2025-12-20 23:30:36,090 - INFO - [BackupPlugin][Entry] Starting backup for superset.
|
||||
2025-12-20 23:30:36,093 - INFO - [setup_clients][Enter] Starting Superset clients initialization.
|
||||
2025-12-20 23:30:36,128 - INFO - [setup_clients][Action] Loading environments from ConfigManager
|
||||
2025-12-20 23:30:36,129 - INFO - [SupersetClient.__init__][Enter] Initializing SupersetClient.
|
||||
2025-12-20 23:30:36,129 - INFO - [APIClient.__init__][Entry] Initializing APIClient.
|
||||
2025-12-20 23:30:36,130 - WARNING - [_init_session][State] SSL verification disabled.
|
||||
2025-12-20 23:30:36,130 - INFO - [APIClient.__init__][Exit] APIClient initialized.
|
||||
2025-12-20 23:30:36,130 - INFO - [SupersetClient.__init__][Exit] SupersetClient initialized.
|
||||
2025-12-20 23:30:36,130 - INFO - [get_dashboards][Enter] Fetching dashboards.
|
||||
2025-12-20 23:30:36,131 - INFO - [authenticate][Enter] Authenticating to https://superset.bebesh.ru/api/v1
|
||||
2025-12-20 23:30:36,897 - INFO - [authenticate][Exit] Authenticated successfully.
|
||||
2025-12-20 23:30:37,527 - INFO - [get_dashboards][Exit] Found 11 dashboards.
|
||||
2025-12-20 23:30:37,527 - INFO - [BackupPlugin][Progress] Found 11 dashboards to export in superset.
|
||||
2025-12-20 23:30:37,529 - INFO - [export_dashboard][Enter] Exporting dashboard 11.
|
||||
2025-12-20 23:30:38,224 - INFO - [export_dashboard][Exit] Exported dashboard 11 to dashboard_export_20251220T203037.zip.
|
||||
2025-12-20 23:30:38,225 - INFO - [save_and_unpack_dashboard][Enter] Processing dashboard. Unpack: False
|
||||
2025-12-20 23:30:38,226 - INFO - [save_and_unpack_dashboard][State] Dashboard saved to: backups/SUPERSET/FCC New Coder Survey 2018/dashboard_export_20251220T203037.zip
|
||||
2025-12-20 23:30:38,227 - INFO - [archive_exports][Enter] Managing archive in backups/SUPERSET/FCC New Coder Survey 2018
|
||||
2025-12-20 23:30:38,230 - INFO - [export_dashboard][Enter] Exporting dashboard 10.
|
||||
2025-12-20 23:30:38,438 - INFO - [export_dashboard][Exit] Exported dashboard 10 to dashboard_export_20251220T203038.zip.
|
||||
2025-12-20 23:30:38,438 - INFO - [save_and_unpack_dashboard][Enter] Processing dashboard. Unpack: False
|
||||
2025-12-20 23:30:38,439 - INFO - [save_and_unpack_dashboard][State] Dashboard saved to: backups/SUPERSET/COVID Vaccine Dashboard/dashboard_export_20251220T203038.zip
|
||||
2025-12-20 23:30:38,439 - INFO - [archive_exports][Enter] Managing archive in backups/SUPERSET/COVID Vaccine Dashboard
|
||||
2025-12-20 23:30:38,440 - INFO - [export_dashboard][Enter] Exporting dashboard 9.
|
||||
2025-12-20 23:30:38,853 - INFO - [export_dashboard][Exit] Exported dashboard 9 to dashboard_export_20251220T203038.zip.
|
||||
2025-12-20 23:30:38,853 - INFO - [save_and_unpack_dashboard][Enter] Processing dashboard. Unpack: False
|
||||
2025-12-20 23:30:38,856 - INFO - [save_and_unpack_dashboard][State] Dashboard saved to: backups/SUPERSET/Sales Dashboard/dashboard_export_20251220T203038.zip
|
||||
2025-12-20 23:30:38,856 - INFO - [archive_exports][Enter] Managing archive in backups/SUPERSET/Sales Dashboard
|
||||
2025-12-20 23:30:38,858 - INFO - [export_dashboard][Enter] Exporting dashboard 8.
|
||||
2025-12-20 23:30:38,939 - INFO - [export_dashboard][Exit] Exported dashboard 8 to dashboard_export_20251220T203038.zip.
|
||||
2025-12-20 23:30:38,940 - INFO - [save_and_unpack_dashboard][Enter] Processing dashboard. Unpack: False
|
||||
2025-12-20 23:30:38,941 - INFO - [save_and_unpack_dashboard][State] Dashboard saved to: backups/SUPERSET/Unicode Test/dashboard_export_20251220T203038.zip
|
||||
2025-12-20 23:30:38,941 - INFO - [archive_exports][Enter] Managing archive in backups/SUPERSET/Unicode Test
|
||||
2025-12-20 23:30:38,942 - INFO - [export_dashboard][Enter] Exporting dashboard 7.
|
||||
2025-12-20 23:30:39,148 - INFO - [export_dashboard][Exit] Exported dashboard 7 to dashboard_export_20251220T203038.zip.
|
||||
2025-12-20 23:30:39,148 - INFO - [save_and_unpack_dashboard][Enter] Processing dashboard. Unpack: False
|
||||
2025-12-20 23:30:39,149 - INFO - [save_and_unpack_dashboard][State] Dashboard saved to: backups/SUPERSET/Video Game Sales/dashboard_export_20251220T203038.zip
|
||||
2025-12-20 23:30:39,149 - INFO - [archive_exports][Enter] Managing archive in backups/SUPERSET/Video Game Sales
|
||||
2025-12-20 23:30:39,150 - INFO - [export_dashboard][Enter] Exporting dashboard 6.
|
||||
2025-12-20 23:30:39,689 - INFO - [export_dashboard][Exit] Exported dashboard 6 to dashboard_export_20251220T203039.zip.
|
||||
2025-12-20 23:30:39,689 - INFO - [save_and_unpack_dashboard][Enter] Processing dashboard. Unpack: False
|
||||
2025-12-20 23:30:39,690 - INFO - [save_and_unpack_dashboard][State] Dashboard saved to: backups/SUPERSET/Featured Charts/dashboard_export_20251220T203039.zip
|
||||
2025-12-20 23:30:39,691 - INFO - [archive_exports][Enter] Managing archive in backups/SUPERSET/Featured Charts
|
||||
2025-12-20 23:30:39,692 - INFO - [export_dashboard][Enter] Exporting dashboard 5.
|
||||
2025-12-20 23:30:39,960 - INFO - [export_dashboard][Exit] Exported dashboard 5 to dashboard_export_20251220T203039.zip.
|
||||
2025-12-20 23:30:39,960 - INFO - [save_and_unpack_dashboard][Enter] Processing dashboard. Unpack: False
|
||||
2025-12-20 23:30:39,961 - INFO - [save_and_unpack_dashboard][State] Dashboard saved to: backups/SUPERSET/Slack Dashboard/dashboard_export_20251220T203039.zip
|
||||
2025-12-20 23:30:39,961 - INFO - [archive_exports][Enter] Managing archive in backups/SUPERSET/Slack Dashboard
|
||||
2025-12-20 23:30:39,962 - INFO - [export_dashboard][Enter] Exporting dashboard 4.
|
||||
2025-12-20 23:30:40,196 - INFO - [export_dashboard][Exit] Exported dashboard 4 to dashboard_export_20251220T203039.zip.
|
||||
2025-12-20 23:30:40,196 - INFO - [save_and_unpack_dashboard][Enter] Processing dashboard. Unpack: False
|
||||
2025-12-20 23:30:40,197 - INFO - [save_and_unpack_dashboard][State] Dashboard saved to: backups/SUPERSET/deck.gl Demo/dashboard_export_20251220T203039.zip
|
||||
2025-12-20 23:30:40,197 - INFO - [archive_exports][Enter] Managing archive in backups/SUPERSET/deck.gl Demo
|
||||
2025-12-20 23:30:40,198 - INFO - [export_dashboard][Enter] Exporting dashboard 3.
|
||||
2025-12-20 23:30:40,745 - INFO - [export_dashboard][Exit] Exported dashboard 3 to dashboard_export_20251220T203040.zip.
|
||||
2025-12-20 23:30:40,746 - INFO - [save_and_unpack_dashboard][Enter] Processing dashboard. Unpack: False
|
||||
2025-12-20 23:30:40,760 - INFO - [save_and_unpack_dashboard][State] Dashboard saved to: backups/SUPERSET/Misc Charts/dashboard_export_20251220T203040.zip
|
||||
2025-12-20 23:30:40,761 - INFO - [archive_exports][Enter] Managing archive in backups/SUPERSET/Misc Charts
|
||||
2025-12-20 23:30:40,762 - INFO - [export_dashboard][Enter] Exporting dashboard 2.
|
||||
2025-12-20 23:30:40,928 - INFO - [export_dashboard][Exit] Exported dashboard 2 to dashboard_export_20251220T203040.zip.
|
||||
2025-12-20 23:30:40,929 - INFO - [save_and_unpack_dashboard][Enter] Processing dashboard. Unpack: False
|
||||
2025-12-20 23:30:40,930 - INFO - [save_and_unpack_dashboard][State] Dashboard saved to: backups/SUPERSET/USA Births Names/dashboard_export_20251220T203040.zip
|
||||
2025-12-20 23:30:40,931 - INFO - [archive_exports][Enter] Managing archive in backups/SUPERSET/USA Births Names
|
||||
2025-12-20 23:30:40,932 - INFO - [export_dashboard][Enter] Exporting dashboard 1.
|
||||
2025-12-20 23:30:41,582 - INFO - [export_dashboard][Exit] Exported dashboard 1 to dashboard_export_20251220T203040.zip.
|
||||
2025-12-20 23:30:41,582 - INFO - [save_and_unpack_dashboard][Enter] Processing dashboard. Unpack: False
|
||||
2025-12-20 23:30:41,749 - INFO - [save_and_unpack_dashboard][State] Dashboard saved to: backups/SUPERSET/World Bank's Data/dashboard_export_20251220T203040.zip
|
||||
2025-12-20 23:30:41,750 - INFO - [archive_exports][Enter] Managing archive in backups/SUPERSET/World Bank's Data
|
||||
2025-12-20 23:30:41,752 - INFO - [consolidate_archive_folders][Enter] Consolidating archives in backups/SUPERSET
|
||||
2025-12-20 23:30:41,753 - INFO - [remove_empty_directories][Enter] Starting cleanup of empty directories in backups/SUPERSET
|
||||
2025-12-20 23:30:41,758 - INFO - [remove_empty_directories][Exit] Removed 0 empty directories.
|
||||
2025-12-20 23:30:41,758 - INFO - [BackupPlugin][CoherenceCheck:Passed] Backup logic completed for superset.
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -53,6 +53,7 @@ async def update_global_settings(
|
||||
):
|
||||
with belief_scope("update_global_settings"):
|
||||
logger.info("[update_global_settings][Entry] Updating global settings")
|
||||
|
||||
config_manager.update_global_settings(settings)
|
||||
return settings
|
||||
# [/DEF:update_global_settings:Function]
|
||||
@@ -207,30 +208,5 @@ async def test_environment_connection(
|
||||
return {"status": "error", "message": str(e)}
|
||||
# [/DEF:test_environment_connection:Function]
|
||||
|
||||
# [DEF:validate_backup_path:Function]
|
||||
# @PURPOSE: Validates if a backup path exists and is writable.
|
||||
# @PRE: Path is provided in path_data.
|
||||
# @POST: Returns success or error status.
|
||||
# @PARAM: path (str) - The path to validate.
|
||||
# @RETURN: dict - Validation result.
|
||||
@router.post("/validate-path")
|
||||
async def validate_backup_path(
|
||||
path_data: dict,
|
||||
config_manager: ConfigManager = Depends(get_config_manager)
|
||||
):
|
||||
with belief_scope("validate_backup_path"):
|
||||
path = path_data.get("path")
|
||||
if not path:
|
||||
raise HTTPException(status_code=400, detail="Path is required")
|
||||
|
||||
logger.info(f"[validate_backup_path][Entry] Validating path: {path}")
|
||||
|
||||
valid, message = config_manager.validate_path(path)
|
||||
|
||||
if not valid:
|
||||
return {"status": "error", "message": message}
|
||||
|
||||
return {"status": "success", "message": message}
|
||||
# [/DEF:validate_backup_path:Function]
|
||||
|
||||
# [/DEF:SettingsRouter:Module]
|
||||
|
||||
@@ -11,41 +11,47 @@
|
||||
from fastapi import APIRouter, Depends, UploadFile, File, Form, HTTPException
|
||||
from fastapi.responses import FileResponse
|
||||
from typing import List, Optional
|
||||
from backend.src.models.storage import StoredFile, FileCategory
|
||||
from backend.src.dependencies import get_plugin_loader
|
||||
from backend.src.plugins.storage.plugin import StoragePlugin
|
||||
from backend.src.core.logger import belief_scope
|
||||
from ...models.storage import StoredFile, FileCategory
|
||||
from ...dependencies import get_plugin_loader
|
||||
from ...plugins.storage.plugin import StoragePlugin
|
||||
from ...core.logger import belief_scope
|
||||
# [/SECTION]
|
||||
|
||||
router = APIRouter(tags=["storage"])
|
||||
|
||||
# [DEF:list_files:Function]
|
||||
# @PURPOSE: List all files in the storage system, optionally filtered by category.
|
||||
# @PURPOSE: List all files and directories in the storage system.
|
||||
#
|
||||
# @PRE: None.
|
||||
# @POST: Returns a list of StoredFile objects.
|
||||
#
|
||||
# @PARAM: category (Optional[FileCategory]) - Filter by category.
|
||||
# @RETURN: List[StoredFile] - List of files.
|
||||
# @PARAM: path (Optional[str]) - Subpath within the category.
|
||||
# @RETURN: List[StoredFile] - List of files/directories.
|
||||
#
|
||||
# @RELATION: CALLS -> StoragePlugin.list_files
|
||||
@router.get("/files", response_model=List[StoredFile])
|
||||
async def list_files(category: Optional[FileCategory] = None, plugin_loader=Depends(get_plugin_loader)):
|
||||
async def list_files(
|
||||
category: Optional[FileCategory] = None,
|
||||
path: Optional[str] = None,
|
||||
plugin_loader=Depends(get_plugin_loader)
|
||||
):
|
||||
with belief_scope("list_files"):
|
||||
storage_plugin: StoragePlugin = plugin_loader.get_plugin("storage-manager")
|
||||
if not storage_plugin:
|
||||
raise HTTPException(status_code=500, detail="Storage plugin not loaded")
|
||||
return storage_plugin.list_files(category)
|
||||
return storage_plugin.list_files(category, path)
|
||||
# [/DEF:list_files:Function]
|
||||
|
||||
# [DEF:upload_file:Function]
|
||||
# @PURPOSE: Upload a file to the storage system under a specific category.
|
||||
# @PURPOSE: Upload a file to the storage system.
|
||||
#
|
||||
# @PRE: category must be a valid FileCategory.
|
||||
# @PRE: file must be a valid UploadFile.
|
||||
# @POST: Returns the StoredFile object of the uploaded file.
|
||||
#
|
||||
# @PARAM: category (FileCategory) - Target category.
|
||||
# @PARAM: path (Optional[str]) - Target subpath.
|
||||
# @PARAM: file (UploadFile) - The file content.
|
||||
# @RETURN: StoredFile - Metadata of the uploaded file.
|
||||
#
|
||||
@@ -55,6 +61,7 @@ async def list_files(category: Optional[FileCategory] = None, plugin_loader=Depe
|
||||
@router.post("/upload", response_model=StoredFile, status_code=201)
|
||||
async def upload_file(
|
||||
category: FileCategory = Form(...),
|
||||
path: Optional[str] = Form(None),
|
||||
file: UploadFile = File(...),
|
||||
plugin_loader=Depends(get_plugin_loader)
|
||||
):
|
||||
@@ -63,33 +70,32 @@ async def upload_file(
|
||||
if not storage_plugin:
|
||||
raise HTTPException(status_code=500, detail="Storage plugin not loaded")
|
||||
try:
|
||||
return await storage_plugin.save_file(file, category)
|
||||
return await storage_plugin.save_file(file, category, path)
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
# [/DEF:upload_file:Function]
|
||||
|
||||
# [DEF:delete_file:Function]
|
||||
# @PURPOSE: Delete a specific file from the storage system.
|
||||
# @PURPOSE: Delete a specific file or directory.
|
||||
#
|
||||
# @PRE: category must be a valid FileCategory.
|
||||
# @PRE: filename must not contain path separators.
|
||||
# @POST: File is removed from storage.
|
||||
# @POST: Item is removed from storage.
|
||||
#
|
||||
# @PARAM: category (FileCategory) - File category.
|
||||
# @PARAM: filename (str) - Name of the file.
|
||||
# @PARAM: path (str) - Relative path of the item.
|
||||
# @RETURN: None
|
||||
#
|
||||
# @SIDE_EFFECT: Deletes file from the filesystem.
|
||||
# @SIDE_EFFECT: Deletes item from the filesystem.
|
||||
#
|
||||
# @RELATION: CALLS -> StoragePlugin.delete_file
|
||||
@router.delete("/files/{category}/{filename}", status_code=204)
|
||||
async def delete_file(category: FileCategory, filename: str, plugin_loader=Depends(get_plugin_loader)):
|
||||
@router.delete("/files/{category}/{path:path}", status_code=204)
|
||||
async def delete_file(category: FileCategory, path: str, plugin_loader=Depends(get_plugin_loader)):
|
||||
with belief_scope("delete_file"):
|
||||
storage_plugin: StoragePlugin = plugin_loader.get_plugin("storage-manager")
|
||||
if not storage_plugin:
|
||||
raise HTTPException(status_code=500, detail="Storage plugin not loaded")
|
||||
try:
|
||||
storage_plugin.delete_file(category, filename)
|
||||
storage_plugin.delete_file(category, path)
|
||||
except FileNotFoundError:
|
||||
raise HTTPException(status_code=404, detail="File not found")
|
||||
except ValueError as e:
|
||||
@@ -100,23 +106,23 @@ async def delete_file(category: FileCategory, filename: str, plugin_loader=Depen
|
||||
# @PURPOSE: Retrieve a file for download.
|
||||
#
|
||||
# @PRE: category must be a valid FileCategory.
|
||||
# @PRE: filename must exist in the specified category.
|
||||
# @POST: Returns a FileResponse.
|
||||
#
|
||||
# @PARAM: category (FileCategory) - File category.
|
||||
# @PARAM: filename (str) - Name of the file.
|
||||
# @PARAM: path (str) - Relative path of the file.
|
||||
# @RETURN: FileResponse - The file content.
|
||||
#
|
||||
# @RELATION: CALLS -> StoragePlugin.get_file_path
|
||||
@router.get("/download/{category}/{filename}")
|
||||
async def download_file(category: FileCategory, filename: str, plugin_loader=Depends(get_plugin_loader)):
|
||||
@router.get("/download/{category}/{path:path}")
|
||||
async def download_file(category: FileCategory, path: str, plugin_loader=Depends(get_plugin_loader)):
|
||||
with belief_scope("download_file"):
|
||||
storage_plugin: StoragePlugin = plugin_loader.get_plugin("storage-manager")
|
||||
if not storage_plugin:
|
||||
raise HTTPException(status_code=500, detail="Storage plugin not loaded")
|
||||
try:
|
||||
path = storage_plugin.get_file_path(category, filename)
|
||||
return FileResponse(path=path, filename=filename)
|
||||
abs_path = storage_plugin.get_file_path(category, path)
|
||||
filename = Path(path).name
|
||||
return FileResponse(path=abs_path, filename=filename)
|
||||
except FileNotFoundError:
|
||||
raise HTTPException(status_code=404, detail="File not found")
|
||||
except ValueError as e:
|
||||
|
||||
@@ -62,14 +62,18 @@ class ConfigManager:
|
||||
logger.info(f"[_load_config][Action] Config file not found. Creating default.")
|
||||
default_config = AppConfig(
|
||||
environments=[],
|
||||
settings=GlobalSettings(backup_path="backups")
|
||||
settings=GlobalSettings()
|
||||
)
|
||||
self._save_config_to_disk(default_config)
|
||||
return default_config
|
||||
|
||||
try:
|
||||
with open(self.config_path, "r") as f:
|
||||
data = json.load(f)
|
||||
|
||||
# Check for deprecated field
|
||||
if "settings" in data and "backup_path" in data["settings"]:
|
||||
del data["settings"]["backup_path"]
|
||||
|
||||
config = AppConfig(**data)
|
||||
logger.info(f"[_load_config][Coherence:OK] Configuration loaded")
|
||||
return config
|
||||
@@ -79,7 +83,7 @@ class ConfigManager:
|
||||
# For now, return default to be safe, but log the error prominently.
|
||||
return AppConfig(
|
||||
environments=[],
|
||||
settings=GlobalSettings(backup_path="backups")
|
||||
settings=GlobalSettings(storage=StorageConfig())
|
||||
)
|
||||
# [/DEF:_load_config:Function]
|
||||
|
||||
|
||||
@@ -43,7 +43,6 @@ class LoggingConfig(BaseModel):
|
||||
# [DEF:GlobalSettings:DataClass]
|
||||
# @PURPOSE: Represents global application settings.
|
||||
class GlobalSettings(BaseModel):
|
||||
backup_path: str
|
||||
storage: StorageConfig = Field(default_factory=StorageConfig)
|
||||
default_environment_id: Optional[str] = None
|
||||
logging: LoggingConfig = Field(default_factory=LoggingConfig)
|
||||
|
||||
@@ -5,13 +5,13 @@ from pydantic import BaseModel, Field
|
||||
|
||||
# [DEF:FileCategory:Class]
|
||||
class FileCategory(str, Enum):
|
||||
BACKUP = "backup"
|
||||
REPOSITORY = "repository"
|
||||
BACKUP = "backups"
|
||||
REPOSITORY = "repositorys"
|
||||
# [/DEF:FileCategory:Class]
|
||||
|
||||
# [DEF:StorageConfig:Class]
|
||||
class StorageConfig(BaseModel):
|
||||
root_path: str = Field(default="../ss-tools-storage", description="Absolute path to the storage root directory.")
|
||||
root_path: str = Field(default="backups", description="Absolute path to the storage root directory.")
|
||||
backup_structure_pattern: str = Field(default="{category}/", description="Pattern for backup directory structure.")
|
||||
repo_structure_pattern: str = Field(default="{category}/", description="Pattern for repository directory structure.")
|
||||
filename_pattern: str = Field(default="{name}_{timestamp}", description="Pattern for filenames.")
|
||||
|
||||
@@ -84,7 +84,7 @@ class BackupPlugin(PluginBase):
|
||||
with belief_scope("get_schema"):
|
||||
config_manager = get_config_manager()
|
||||
envs = [e.name for e in config_manager.get_environments()]
|
||||
default_path = config_manager.get_config().settings.backup_path
|
||||
default_path = config_manager.get_config().settings.storage.root_path
|
||||
|
||||
return {
|
||||
"type": "object",
|
||||
@@ -95,14 +95,8 @@ class BackupPlugin(PluginBase):
|
||||
"description": "The Superset environment to back up.",
|
||||
"enum": envs if envs else [],
|
||||
},
|
||||
"backup_path": {
|
||||
"type": "string",
|
||||
"title": "Backup Path",
|
||||
"description": "The root directory to save backups to.",
|
||||
"default": default_path
|
||||
}
|
||||
},
|
||||
"required": ["env", "backup_path"],
|
||||
"required": ["env"],
|
||||
}
|
||||
# [/DEF:get_schema:Function]
|
||||
|
||||
@@ -126,8 +120,9 @@ class BackupPlugin(PluginBase):
|
||||
if not env:
|
||||
raise KeyError("env")
|
||||
|
||||
backup_path_str = params.get("backup_path") or config_manager.get_config().settings.backup_path
|
||||
backup_path = Path(backup_path_str)
|
||||
storage_settings = config_manager.get_config().settings.storage
|
||||
# Use 'backups' subfolder within the storage root
|
||||
backup_path = Path(storage_settings.root_path) / "backups"
|
||||
|
||||
from ..core.logger import logger as app_logger
|
||||
app_logger.info(f"[BackupPlugin][Entry] Starting backup for {env}.")
|
||||
|
||||
@@ -98,14 +98,43 @@ class StoragePlugin(PluginBase):
|
||||
def get_storage_root(self) -> Path:
|
||||
with belief_scope("StoragePlugin:get_storage_root"):
|
||||
config_manager = get_config_manager()
|
||||
storage_config = config_manager.get_config().settings.storage
|
||||
root = Path(storage_config.root_path)
|
||||
global_settings = config_manager.get_config().settings
|
||||
|
||||
# Use storage.root_path as the source of truth for storage UI
|
||||
root = Path(global_settings.storage.root_path)
|
||||
|
||||
if not root.is_absolute():
|
||||
# Resolve relative to the workspace root (ss-tools)
|
||||
root = (Path(__file__).parents[4] / root).resolve()
|
||||
# Resolve relative to the backend directory
|
||||
# Path(__file__) is backend/src/plugins/storage/plugin.py
|
||||
# parents[3] is the project root (ss-tools)
|
||||
# We need to ensure it's relative to where backend/ is
|
||||
project_root = Path(__file__).parents[3]
|
||||
root = (project_root / root).resolve()
|
||||
return root
|
||||
# [/DEF:get_storage_root:Function]
|
||||
|
||||
# [DEF:resolve_path:Function]
|
||||
# @PURPOSE: Resolves a dynamic path pattern using provided variables.
|
||||
# @PARAM: pattern (str) - The path pattern to resolve.
|
||||
# @PARAM: variables (Dict[str, str]) - Variables to substitute in the pattern.
|
||||
# @RETURN: str - The resolved path.
|
||||
def resolve_path(self, pattern: str, variables: Dict[str, str]) -> str:
|
||||
with belief_scope("StoragePlugin:resolve_path"):
|
||||
# Add common variables
|
||||
vars_with_defaults = {
|
||||
"timestamp": datetime.now().strftime("%Y%m%dT%H%M%S"),
|
||||
**variables
|
||||
}
|
||||
try:
|
||||
resolved = pattern.format(**vars_with_defaults)
|
||||
# Clean up any double slashes or leading/trailing slashes for relative path
|
||||
return os.path.normpath(resolved).strip("/")
|
||||
except KeyError as e:
|
||||
logger.warning(f"[StoragePlugin][Coherence:Failed] Missing variable for path resolution: {e}")
|
||||
# Fallback to literal pattern if formatting fails partially (or handle as needed)
|
||||
return pattern.replace("{", "").replace("}", "")
|
||||
# [/DEF:resolve_path:Function]
|
||||
|
||||
# [DEF:ensure_directories:Function]
|
||||
# @PURPOSE: Creates the storage root and category subdirectories if they don't exist.
|
||||
# @SIDE_EFFECT: Creates directories on the filesystem.
|
||||
@@ -113,7 +142,8 @@ class StoragePlugin(PluginBase):
|
||||
with belief_scope("StoragePlugin:ensure_directories"):
|
||||
root = self.get_storage_root()
|
||||
for category in FileCategory:
|
||||
path = root / f"{category.value}s"
|
||||
# Use singular name for consistency with BackupPlugin and GitService
|
||||
path = root / category.value
|
||||
path.mkdir(parents=True, exist_ok=True)
|
||||
logger.debug(f"[StoragePlugin][Action] Ensured directory: {path}")
|
||||
# [/DEF:ensure_directories:Function]
|
||||
@@ -135,46 +165,68 @@ class StoragePlugin(PluginBase):
|
||||
# [/DEF:validate_path:Function]
|
||||
|
||||
# [DEF:list_files:Function]
|
||||
# @PURPOSE: Lists all files in a specific category.
|
||||
# @PURPOSE: Lists all files and directories in a specific category and subpath.
|
||||
# @PARAM: category (Optional[FileCategory]) - The category to list.
|
||||
# @RETURN: List[StoredFile] - List of file metadata objects.
|
||||
def list_files(self, category: Optional[FileCategory] = None) -> List[StoredFile]:
|
||||
# @PARAM: subpath (Optional[str]) - Nested path within the category.
|
||||
# @RETURN: List[StoredFile] - List of file and directory metadata objects.
|
||||
def list_files(self, category: Optional[FileCategory] = None, subpath: Optional[str] = None) -> List[StoredFile]:
|
||||
with belief_scope("StoragePlugin:list_files"):
|
||||
root = self.get_storage_root()
|
||||
logger.info(f"[StoragePlugin][Action] Listing files in root: {root}, category: {category}, subpath: {subpath}")
|
||||
files = []
|
||||
|
||||
categories = [category] if category else list(FileCategory)
|
||||
|
||||
for cat in categories:
|
||||
cat_dir = root / f"{cat.value}s"
|
||||
if not cat_dir.exists():
|
||||
# Scan the category subfolder + optional subpath
|
||||
base_dir = root / cat.value
|
||||
if subpath:
|
||||
target_dir = self.validate_path(base_dir / subpath)
|
||||
else:
|
||||
target_dir = base_dir
|
||||
|
||||
if not target_dir.exists():
|
||||
continue
|
||||
|
||||
for item in cat_dir.iterdir():
|
||||
if item.is_file():
|
||||
stat = item.stat()
|
||||
logger.debug(f"[StoragePlugin][Action] Scanning directory: {target_dir}")
|
||||
|
||||
# Use os.scandir for better performance and to distinguish files vs dirs
|
||||
with os.scandir(target_dir) as it:
|
||||
for entry in it:
|
||||
# Skip logs
|
||||
if "Logs" in entry.path:
|
||||
continue
|
||||
|
||||
stat = entry.stat()
|
||||
is_dir = entry.is_dir()
|
||||
|
||||
files.append(StoredFile(
|
||||
name=item.name,
|
||||
path=str(item.relative_to(root)),
|
||||
size=stat.st_size,
|
||||
name=entry.name,
|
||||
path=str(Path(entry.path).relative_to(root)),
|
||||
size=stat.st_size if not is_dir else 0,
|
||||
created_at=datetime.fromtimestamp(stat.st_ctime),
|
||||
category=cat,
|
||||
mime_type=None # Could use python-magic here if needed
|
||||
mime_type="directory" if is_dir else None
|
||||
))
|
||||
|
||||
return sorted(files, key=lambda x: x.created_at, reverse=True)
|
||||
# Sort: directories first, then by name
|
||||
return sorted(files, key=lambda x: (x.mime_type != "directory", x.name))
|
||||
# [/DEF:list_files:Function]
|
||||
|
||||
# [DEF:save_file:Function]
|
||||
# @PURPOSE: Saves an uploaded file to the specified category.
|
||||
# @PURPOSE: Saves an uploaded file to the specified category and optional subpath.
|
||||
# @PARAM: file (UploadFile) - The uploaded file.
|
||||
# @PARAM: category (FileCategory) - The target category.
|
||||
# @PARAM: subpath (Optional[str]) - The target subpath.
|
||||
# @RETURN: StoredFile - Metadata of the saved file.
|
||||
# @SIDE_EFFECT: Writes file to disk.
|
||||
async def save_file(self, file: UploadFile, category: FileCategory) -> StoredFile:
|
||||
async def save_file(self, file: UploadFile, category: FileCategory, subpath: Optional[str] = None) -> StoredFile:
|
||||
with belief_scope("StoragePlugin:save_file"):
|
||||
root = self.get_storage_root()
|
||||
dest_dir = root / f"{category.value}s"
|
||||
dest_dir = root / category.value
|
||||
if subpath:
|
||||
dest_dir = dest_dir / subpath
|
||||
|
||||
dest_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
dest_path = self.validate_path(dest_dir / file.filename)
|
||||
@@ -194,34 +246,44 @@ class StoragePlugin(PluginBase):
|
||||
# [/DEF:save_file:Function]
|
||||
|
||||
# [DEF:delete_file:Function]
|
||||
# @PURPOSE: Deletes a file from the specified category.
|
||||
# @PURPOSE: Deletes a file or directory from the specified category and path.
|
||||
# @PARAM: category (FileCategory) - The category.
|
||||
# @PARAM: filename (str) - The name of the file.
|
||||
# @SIDE_EFFECT: Removes file from disk.
|
||||
def delete_file(self, category: FileCategory, filename: str):
|
||||
# @PARAM: path (str) - The relative path of the file or directory.
|
||||
# @SIDE_EFFECT: Removes item from disk.
|
||||
def delete_file(self, category: FileCategory, path: str):
|
||||
with belief_scope("StoragePlugin:delete_file"):
|
||||
root = self.get_storage_root()
|
||||
file_path = self.validate_path(root / f"{category.value}s" / filename)
|
||||
# path is relative to root, but we ensure it starts with category
|
||||
full_path = self.validate_path(root / path)
|
||||
|
||||
if file_path.exists():
|
||||
file_path.unlink()
|
||||
logger.info(f"[StoragePlugin][Action] Deleted file: {file_path}")
|
||||
if not str(Path(path)).startswith(category.value):
|
||||
raise ValueError(f"Path {path} does not belong to category {category}")
|
||||
|
||||
if full_path.exists():
|
||||
if full_path.is_dir():
|
||||
shutil.rmtree(full_path)
|
||||
else:
|
||||
full_path.unlink()
|
||||
logger.info(f"[StoragePlugin][Action] Deleted: {full_path}")
|
||||
else:
|
||||
raise FileNotFoundError(f"File {filename} not found in {category.value}s")
|
||||
raise FileNotFoundError(f"Item {path} not found")
|
||||
# [/DEF:delete_file:Function]
|
||||
|
||||
# [DEF:get_file_path:Function]
|
||||
# @PURPOSE: Returns the absolute path of a file for download.
|
||||
# @PARAM: category (FileCategory) - The category.
|
||||
# @PARAM: filename (str) - The name of the file.
|
||||
# @PARAM: path (str) - The relative path of the file.
|
||||
# @RETURN: Path - Absolute path to the file.
|
||||
def get_file_path(self, category: FileCategory, filename: str) -> Path:
|
||||
def get_file_path(self, category: FileCategory, path: str) -> Path:
|
||||
with belief_scope("StoragePlugin:get_file_path"):
|
||||
root = self.get_storage_root()
|
||||
file_path = self.validate_path(root / f"{category.value}s" / filename)
|
||||
file_path = self.validate_path(root / path)
|
||||
|
||||
if not file_path.exists():
|
||||
raise FileNotFoundError(f"File {filename} not found in {category.value}s")
|
||||
if not str(Path(path)).startswith(category.value):
|
||||
raise ValueError(f"Path {path} does not belong to category {category}")
|
||||
|
||||
if not file_path.exists() or file_path.is_dir():
|
||||
raise FileNotFoundError(f"File {path} not found")
|
||||
|
||||
return file_path
|
||||
# [/DEF:get_file_path:Function]
|
||||
|
||||
@@ -31,9 +31,15 @@ class GitService:
|
||||
# @PARAM: base_path (str) - Root directory for all Git clones.
|
||||
# @PRE: base_path is a valid string path.
|
||||
# @POST: GitService is initialized; base_path directory exists.
|
||||
def __init__(self, base_path: str = "backend/git_repos"):
|
||||
def __init__(self, base_path: str = "git_repos"):
|
||||
with belief_scope("GitService.__init__"):
|
||||
self.base_path = base_path
|
||||
# Resolve relative to the backend directory
|
||||
# Path(__file__) is backend/src/services/git_service.py
|
||||
# parents[2] is backend/
|
||||
from pathlib import Path
|
||||
backend_root = Path(__file__).parents[2]
|
||||
|
||||
self.base_path = str((backend_root / base_path).resolve())
|
||||
if not os.path.exists(self.base_path):
|
||||
os.makedirs(self.base_path)
|
||||
# [/DEF:__init__:Function]
|
||||
|
||||
BIN
backend/tasks.db
BIN
backend/tasks.db
Binary file not shown.
@@ -13,7 +13,7 @@ The settings mechanism allows users to configure multiple Superset environments
|
||||
Configuration is structured using Pydantic models in `backend/src/core/config_models.py`:
|
||||
|
||||
- `Environment`: Represents a Superset instance (URL, credentials). The `base_url` is automatically normalized to include the `/api/v1` suffix if missing.
|
||||
- `GlobalSettings`: Global application parameters (e.g., `backup_path`).
|
||||
- `GlobalSettings`: Global application parameters (e.g., `storage.root_path`).
|
||||
- `AppConfig`: The root configuration object.
|
||||
|
||||
### Configuration Manager
|
||||
@@ -43,4 +43,4 @@ The settings page is located at `frontend/src/pages/Settings.svelte`. It provide
|
||||
|
||||
Existing plugins and utilities use the `ConfigManager` to fetch configuration:
|
||||
- `superset_tool/utils/init_clients.py`: Dynamically initializes Superset clients from the configured environments.
|
||||
- `BackupPlugin`: Uses the configured `backup_path` as the default storage location.
|
||||
- `BackupPlugin`: Uses the configured `storage.root_path` as the default storage location.
|
||||
|
||||
@@ -13,11 +13,16 @@
|
||||
// [SECTION: IMPORTS]
|
||||
import { createEventDispatcher } from 'svelte';
|
||||
import { downloadFileUrl } from '../../services/storageService';
|
||||
import { t } from '../../lib/i18n';
|
||||
// [/SECTION: IMPORTS]
|
||||
|
||||
export let files = [];
|
||||
const dispatch = createEventDispatcher();
|
||||
|
||||
function isDirectory(file) {
|
||||
return file.mime_type === 'directory';
|
||||
}
|
||||
|
||||
function formatSize(bytes) {
|
||||
if (bytes === 0) return '0 B';
|
||||
const k = 1024;
|
||||
@@ -36,40 +41,63 @@
|
||||
<table class="min-w-full bg-white border border-gray-200">
|
||||
<thead class="bg-gray-50">
|
||||
<tr>
|
||||
<th class="px-6 py-3 text-left text-xs font-medium text-gray-500 uppercase tracking-wider">Name</th>
|
||||
<th class="px-6 py-3 text-left text-xs font-medium text-gray-500 uppercase tracking-wider">Category</th>
|
||||
<th class="px-6 py-3 text-left text-xs font-medium text-gray-500 uppercase tracking-wider">Size</th>
|
||||
<th class="px-6 py-3 text-left text-xs font-medium text-gray-500 uppercase tracking-wider">Created At</th>
|
||||
<th class="px-6 py-3 text-right text-xs font-medium text-gray-500 uppercase tracking-wider">Actions</th>
|
||||
<th class="px-6 py-3 text-left text-xs font-medium text-gray-500 uppercase tracking-wider">{$t.storage.table.name}</th>
|
||||
<th class="px-6 py-3 text-left text-xs font-medium text-gray-500 uppercase tracking-wider">{$t.storage.table.category}</th>
|
||||
<th class="px-6 py-3 text-left text-xs font-medium text-gray-500 uppercase tracking-wider">{$t.storage.table.size}</th>
|
||||
<th class="px-6 py-3 text-left text-xs font-medium text-gray-500 uppercase tracking-wider">{$t.storage.table.created_at}</th>
|
||||
<th class="px-6 py-3 text-right text-xs font-medium text-gray-500 uppercase tracking-wider">{$t.storage.table.actions}</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody class="divide-y divide-gray-200">
|
||||
{#each files as file}
|
||||
<tr class="hover:bg-gray-50">
|
||||
<td class="px-6 py-4 whitespace-nowrap text-sm font-medium text-gray-900">{file.name}</td>
|
||||
<td class="px-6 py-4 whitespace-nowrap text-sm font-medium text-gray-900">
|
||||
{#if isDirectory(file)}
|
||||
<button
|
||||
on:click={() => dispatch('navigate', file.path)}
|
||||
class="flex items-center text-indigo-600 hover:text-indigo-900"
|
||||
>
|
||||
<svg class="h-5 w-5 mr-2 text-yellow-400" fill="currentColor" viewBox="0 0 20 20">
|
||||
<path d="M2 6a2 2 0 012-2h5l2 2h5a2 2 0 012 2v6a2 2 0 01-2 2H4a2 2 0 01-2-2V6z" />
|
||||
</svg>
|
||||
{file.name}
|
||||
</button>
|
||||
{:else}
|
||||
<div class="flex items-center">
|
||||
<svg class="h-5 w-5 mr-2 text-gray-400" fill="none" stroke="currentColor" viewBox="0 0 24 24">
|
||||
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M7 21h10a2 2 0 002-2V9.414a1 1 0 00-.293-.707l-5.414-5.414A1 1 0 0012.586 3H7a2 2 0 00-2 2v14a2 2 0 002 2z" />
|
||||
</svg>
|
||||
{file.name}
|
||||
</div>
|
||||
{/if}
|
||||
</td>
|
||||
<td class="px-6 py-4 whitespace-nowrap text-sm text-gray-500 capitalize">{file.category}</td>
|
||||
<td class="px-6 py-4 whitespace-nowrap text-sm text-gray-500">{formatSize(file.size)}</td>
|
||||
<td class="px-6 py-4 whitespace-nowrap text-sm text-gray-500">
|
||||
{isDirectory(file) ? '--' : formatSize(file.size)}
|
||||
</td>
|
||||
<td class="px-6 py-4 whitespace-nowrap text-sm text-gray-500">{formatDate(file.created_at)}</td>
|
||||
<td class="px-6 py-4 whitespace-nowrap text-right text-sm font-medium">
|
||||
<a
|
||||
href={downloadFileUrl(file.category, file.name)}
|
||||
download={file.name}
|
||||
class="text-indigo-600 hover:text-indigo-900 mr-4"
|
||||
>
|
||||
Download
|
||||
</a>
|
||||
<button
|
||||
on:click={() => dispatch('delete', { category: file.category, filename: file.name })}
|
||||
{#if !isDirectory(file)}
|
||||
<a
|
||||
href={downloadFileUrl(file.category, file.path)}
|
||||
download={file.name}
|
||||
class="text-indigo-600 hover:text-indigo-900 mr-4"
|
||||
>
|
||||
{$t.storage.table.download}
|
||||
</a>
|
||||
{/if}
|
||||
<button
|
||||
on:click={() => dispatch('delete', { category: file.category, path: file.path, name: file.name })}
|
||||
class="text-red-600 hover:text-red-900"
|
||||
>
|
||||
Delete
|
||||
{$t.storage.table.delete}
|
||||
</button>
|
||||
</td>
|
||||
</tr>
|
||||
{:else}
|
||||
<tr>
|
||||
<td colspan="5" class="px-6 py-10 text-center text-sm text-gray-500">
|
||||
No files found.
|
||||
{$t.storage.no_files}
|
||||
</td>
|
||||
</tr>
|
||||
{/each}
|
||||
|
||||
@@ -14,6 +14,7 @@
|
||||
import { createEventDispatcher } from 'svelte';
|
||||
import { uploadFile } from '../../services/storageService';
|
||||
import { addToast } from '../../lib/toasts';
|
||||
import { t } from '../../lib/i18n';
|
||||
// [/SECTION: IMPORTS]
|
||||
|
||||
// [DEF:handleUpload:Function]
|
||||
@@ -24,7 +25,8 @@
|
||||
*/
|
||||
const dispatch = createEventDispatcher();
|
||||
let fileInput;
|
||||
let category = 'backup';
|
||||
export let category = 'backups';
|
||||
export let path = '';
|
||||
let isUploading = false;
|
||||
let dragOver = false;
|
||||
|
||||
@@ -34,12 +36,18 @@
|
||||
|
||||
isUploading = true;
|
||||
try {
|
||||
await uploadFile(file, category);
|
||||
addToast(`File ${file.name} uploaded successfully.`, 'success');
|
||||
// path is relative to root, but upload endpoint expects path within category
|
||||
// FileList.path is like "backup/folder", we need just "folder"
|
||||
const subpath = path.startsWith(category)
|
||||
? path.substring(category.length).replace(/^\/+/, '')
|
||||
: path;
|
||||
|
||||
await uploadFile(file, category, subpath);
|
||||
addToast($t.storage.messages.upload_success.replace('{name}', file.name), 'success');
|
||||
fileInput.value = '';
|
||||
dispatch('uploaded');
|
||||
} catch (error) {
|
||||
addToast(`Upload failed: ${error.message}`, 'error');
|
||||
addToast($t.storage.messages.upload_failed.replace('{error}', error.message), 'error');
|
||||
} finally {
|
||||
isUploading = false;
|
||||
}
|
||||
@@ -65,17 +73,17 @@
|
||||
|
||||
<!-- [SECTION: TEMPLATE] -->
|
||||
<div class="bg-white p-6 rounded-lg border border-gray-200 shadow-sm">
|
||||
<h2 class="text-lg font-semibold mb-4">Upload File</h2>
|
||||
<h2 class="text-lg font-semibold mb-4">{$t.storage.upload_title}</h2>
|
||||
|
||||
<div class="space-y-4">
|
||||
<div>
|
||||
<label class="block text-sm font-medium text-gray-700 mb-1">Target Category</label>
|
||||
<select
|
||||
<label class="block text-sm font-medium text-gray-700 mb-1">{$t.storage.target_category}</label>
|
||||
<select
|
||||
bind:value={category}
|
||||
class="block w-full border-gray-300 rounded-md shadow-sm focus:ring-indigo-500 focus:border-indigo-500 sm:text-sm"
|
||||
>
|
||||
<option value="backup">Backup</option>
|
||||
<option value="repository">Repository</option>
|
||||
<option value="backups">{$t.storage.backups}</option>
|
||||
<option value="repositorys">{$t.storage.repositories}</option>
|
||||
</select>
|
||||
</div>
|
||||
|
||||
@@ -92,8 +100,8 @@
|
||||
</svg>
|
||||
<div class="flex text-sm text-gray-600">
|
||||
<label for="file-upload" class="relative cursor-pointer bg-white rounded-md font-medium text-indigo-600 hover:text-indigo-500 focus-within:outline-none focus-within:ring-2 focus-within:ring-offset-2 focus-within:ring-indigo-500">
|
||||
<span>Upload a file</span>
|
||||
<input
|
||||
<span>{$t.storage.upload_button}</span>
|
||||
<input
|
||||
id="file-upload"
|
||||
name="file-upload"
|
||||
type="file"
|
||||
@@ -103,16 +111,16 @@
|
||||
disabled={isUploading}
|
||||
>
|
||||
</label>
|
||||
<p class="pl-1">or drag and drop</p>
|
||||
<p class="pl-1">{$t.storage.drag_drop}</p>
|
||||
</div>
|
||||
<p class="text-xs text-gray-500">ZIP, YAML, JSON up to 50MB</p>
|
||||
<p class="text-xs text-gray-500">{$t.storage.supported_formats}</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{#if isUploading}
|
||||
<div class="flex items-center justify-center space-x-2 text-indigo-600">
|
||||
<div class="animate-spin rounded-full h-4 w-4 border-b-2 border-indigo-600"></div>
|
||||
<span class="text-sm font-medium">Uploading...</span>
|
||||
<span class="text-sm font-medium">{$t.storage.uploading}</span>
|
||||
</div>
|
||||
{/if}
|
||||
</div>
|
||||
|
||||
@@ -20,7 +20,6 @@
|
||||
let settings = {
|
||||
environments: [],
|
||||
settings: {
|
||||
backup_path: '',
|
||||
default_environment_id: null,
|
||||
logging: {
|
||||
level: 'INFO',
|
||||
@@ -204,12 +203,6 @@
|
||||
|
||||
<section class="mb-8 bg-white p-6 rounded shadow">
|
||||
<h2 class="text-xl font-semibold mb-4">Global Settings</h2>
|
||||
<div class="grid grid-cols-1 gap-4">
|
||||
<div>
|
||||
<label for="backup_path" class="block text-sm font-medium text-gray-700">Backup Storage Path</label>
|
||||
<input type="text" id="backup_path" bind:value={settings.settings.backup_path} class="mt-1 block w-full border border-gray-300 rounded-md shadow-sm p-2" />
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<h3 class="text-lg font-medium mb-4 mt-6">Logging Configuration</h3>
|
||||
<div class="grid grid-cols-1 md:grid-cols-2 gap-4">
|
||||
|
||||
@@ -170,19 +170,6 @@
|
||||
</div>
|
||||
{/if}
|
||||
|
||||
<div class="mb-8">
|
||||
<Card title={$t.settings?.global_title || "Global Settings"}>
|
||||
<div class="grid grid-cols-1 gap-6">
|
||||
<Input
|
||||
label={$t.settings?.backup_path || "Backup Storage Path"}
|
||||
bind:value={settings.settings.backup_path}
|
||||
/>
|
||||
<Button on:click={handleSaveGlobal}>
|
||||
{$t.common.save}
|
||||
</Button>
|
||||
</div>
|
||||
</Card>
|
||||
</div>
|
||||
|
||||
<div class="mb-8">
|
||||
<Card title={$t.settings?.storage_title || "File Storage Configuration"}>
|
||||
|
||||
@@ -18,7 +18,6 @@ export async function load() {
|
||||
settings: {
|
||||
environments: [],
|
||||
settings: {
|
||||
backup_path: '',
|
||||
default_environment_id: null
|
||||
}
|
||||
},
|
||||
|
||||
@@ -15,6 +15,7 @@
|
||||
import { onMount } from 'svelte';
|
||||
import { listFiles, deleteFile } from '../../../services/storageService';
|
||||
import { addToast } from '../../../lib/toasts';
|
||||
import { t } from '../../../lib/i18n';
|
||||
import FileList from '../../../components/storage/FileList.svelte';
|
||||
import FileUpload from '../../../components/storage/FileUpload.svelte';
|
||||
// [/SECTION: IMPORTS]
|
||||
@@ -26,15 +27,30 @@
|
||||
*/
|
||||
let files = [];
|
||||
let isLoading = false;
|
||||
let activeTab = 'all';
|
||||
let activeTab = 'backups';
|
||||
let currentPath = 'backups'; // Relative to storage root
|
||||
|
||||
async function loadFiles() {
|
||||
isLoading = true;
|
||||
try {
|
||||
const category = activeTab === 'all' ? null : activeTab;
|
||||
files = await listFiles(category);
|
||||
const category = activeTab;
|
||||
|
||||
// If we have a currentPath, we use it.
|
||||
// But if user switched tabs, we should reset currentPath to category root
|
||||
let effectivePath = currentPath;
|
||||
if (category && !currentPath.startsWith(category)) {
|
||||
effectivePath = category;
|
||||
currentPath = category;
|
||||
}
|
||||
|
||||
// API expects path relative to category root if category is provided
|
||||
const subpath = (category && effectivePath.startsWith(category))
|
||||
? effectivePath.substring(category.length).replace(/^\/+/, '')
|
||||
: effectivePath;
|
||||
|
||||
files = await listFiles(category, subpath);
|
||||
} catch (error) {
|
||||
addToast(`Failed to load files: ${error.message}`, 'error');
|
||||
addToast($t.storage.messages.load_failed.replace('{error}', error.message), 'error');
|
||||
} finally {
|
||||
isLoading = false;
|
||||
}
|
||||
@@ -44,39 +60,73 @@
|
||||
// [DEF:handleDelete:Function]
|
||||
/**
|
||||
* @purpose Handles the file deletion process.
|
||||
* @param {CustomEvent} event - The delete event containing category and filename.
|
||||
* @param {CustomEvent} event - The delete event containing category and path.
|
||||
*/
|
||||
async function handleDelete(event) {
|
||||
const { category, filename } = event.detail;
|
||||
if (!confirm(`Are you sure you want to delete ${filename}?`)) return;
|
||||
const { category, path, name } = event.detail;
|
||||
if (!confirm($t.storage.messages.delete_confirm.replace('{name}', name))) return;
|
||||
|
||||
try {
|
||||
await deleteFile(category, filename);
|
||||
addToast(`File ${filename} deleted.`, 'success');
|
||||
await deleteFile(category, path);
|
||||
addToast($t.storage.messages.delete_success.replace('{name}', name), 'success');
|
||||
await loadFiles();
|
||||
} catch (error) {
|
||||
addToast(`Delete failed: ${error.message}`, 'error');
|
||||
addToast($t.storage.messages.delete_failed.replace('{error}', error.message), 'error');
|
||||
}
|
||||
}
|
||||
// [/DEF:handleDelete:Function]
|
||||
|
||||
function handleNavigate(event) {
|
||||
currentPath = event.detail;
|
||||
loadFiles();
|
||||
}
|
||||
|
||||
function navigateUp() {
|
||||
if (!currentPath || currentPath === activeTab) return;
|
||||
const parts = currentPath.split('/');
|
||||
parts.pop();
|
||||
currentPath = parts.join('/') || '';
|
||||
loadFiles();
|
||||
}
|
||||
|
||||
onMount(loadFiles);
|
||||
|
||||
$: if (activeTab) {
|
||||
// Reset path when switching tabs
|
||||
if (!currentPath.startsWith(activeTab)) {
|
||||
currentPath = activeTab;
|
||||
}
|
||||
loadFiles();
|
||||
}
|
||||
</script>
|
||||
|
||||
<!-- [SECTION: TEMPLATE] -->
|
||||
<div class="container mx-auto p-4 max-w-6xl">
|
||||
<div class="flex justify-between items-center mb-6">
|
||||
<h1 class="text-2xl font-bold text-gray-900">File Storage Management</h1>
|
||||
<button
|
||||
<div class="mb-6">
|
||||
<h1 class="text-2xl font-bold text-gray-900">{$t.storage.management}</h1>
|
||||
{#if currentPath}
|
||||
<div class="flex items-center mt-2 text-sm text-gray-500">
|
||||
<button on:click={() => { currentPath = activeTab; loadFiles(); }} class="hover:text-indigo-600">{$t.storage.root}</button>
|
||||
{#each currentPath.split('/').slice(1) as part, i}
|
||||
<span class="mx-2">/</span>
|
||||
<button
|
||||
on:click={() => { currentPath = currentPath.split('/').slice(0, i + 1).join('/'); loadFiles(); }}
|
||||
class="hover:text-indigo-600 capitalize"
|
||||
>
|
||||
{part}
|
||||
</button>
|
||||
{/each}
|
||||
</div>
|
||||
{/if}
|
||||
</div>
|
||||
|
||||
<div class="flex justify-end mb-4">
|
||||
<button
|
||||
on:click={loadFiles}
|
||||
class="inline-flex items-center px-4 py-2 border border-gray-300 rounded-md shadow-sm text-sm font-medium text-gray-700 bg-white hover:bg-gray-50"
|
||||
disabled={isLoading}
|
||||
>
|
||||
{isLoading ? 'Refreshing...' : 'Refresh'}
|
||||
{isLoading ? $t.storage.refreshing : $t.storage.refresh}
|
||||
</button>
|
||||
</div>
|
||||
|
||||
@@ -86,33 +136,45 @@
|
||||
<!-- Tabs -->
|
||||
<div class="border-b border-gray-200">
|
||||
<nav class="-mb-px flex space-x-8">
|
||||
<button
|
||||
on:click={() => activeTab = 'all'}
|
||||
class="py-4 px-1 border-b-2 font-medium text-sm {activeTab === 'all' ? 'border-indigo-500 text-indigo-600' : 'border-transparent text-gray-500 hover:text-gray-700 hover:border-gray-300'}"
|
||||
<button
|
||||
on:click={() => activeTab = 'backups'}
|
||||
class="py-4 px-1 border-b-2 font-medium text-sm {activeTab === 'backups' ? 'border-indigo-500 text-indigo-600' : 'border-transparent text-gray-500 hover:text-gray-700 hover:border-gray-300'}"
|
||||
>
|
||||
All Files
|
||||
{$t.storage.backups}
|
||||
</button>
|
||||
<button
|
||||
on:click={() => activeTab = 'backup'}
|
||||
class="py-4 px-1 border-b-2 font-medium text-sm {activeTab === 'backup' ? 'border-indigo-500 text-indigo-600' : 'border-transparent text-gray-500 hover:text-gray-700 hover:border-gray-300'}"
|
||||
<button
|
||||
on:click={() => activeTab = 'repositorys'}
|
||||
class="py-4 px-1 border-b-2 font-medium text-sm {activeTab === 'repositorys' ? 'border-indigo-500 text-indigo-600' : 'border-transparent text-gray-500 hover:text-gray-700 hover:border-gray-300'}"
|
||||
>
|
||||
Backups
|
||||
</button>
|
||||
<button
|
||||
on:click={() => activeTab = 'repository'}
|
||||
class="py-4 px-1 border-b-2 font-medium text-sm {activeTab === 'repository' ? 'border-indigo-500 text-indigo-600' : 'border-transparent text-gray-500 hover:text-gray-700 hover:border-gray-300'}"
|
||||
>
|
||||
Repositories
|
||||
{$t.storage.repositories}
|
||||
</button>
|
||||
</nav>
|
||||
</div>
|
||||
|
||||
<FileList {files} on:delete={handleDelete} />
|
||||
<div class="flex items-center mb-2">
|
||||
{#if currentPath && currentPath !== activeTab}
|
||||
<button
|
||||
on:click={navigateUp}
|
||||
class="mr-4 inline-flex items-center px-3 py-1 border border-gray-300 shadow-sm text-xs font-medium rounded text-gray-700 bg-white hover:bg-gray-50"
|
||||
>
|
||||
<svg class="h-4 w-4 mr-1" fill="none" stroke="currentColor" viewBox="0 0 24 24">
|
||||
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M10 19l-7-7m0 0l7-7m-7 7h18" />
|
||||
</svg>
|
||||
Back
|
||||
</button>
|
||||
{/if}
|
||||
</div>
|
||||
|
||||
<FileList {files} on:delete={handleDelete} on:navigate={handleNavigate} />
|
||||
</div>
|
||||
|
||||
<!-- Sidebar: Upload -->
|
||||
<div class="lg:col-span-1">
|
||||
<FileUpload on:uploaded={loadFiles} />
|
||||
<FileUpload
|
||||
category={activeTab}
|
||||
path={currentPath}
|
||||
on:uploaded={loadFiles}
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@@ -9,15 +9,19 @@ const API_BASE = '/api/storage';
|
||||
|
||||
// [DEF:listFiles:Function]
|
||||
/**
|
||||
* @purpose Fetches the list of files for a given category.
|
||||
* @purpose Fetches the list of files for a given category and subpath.
|
||||
* @param {string} [category] - Optional category filter.
|
||||
* @param {string} [path] - Optional subpath filter.
|
||||
* @returns {Promise<Array>}
|
||||
*/
|
||||
export async function listFiles(category) {
|
||||
export async function listFiles(category, path) {
|
||||
const params = new URLSearchParams();
|
||||
if (category) {
|
||||
params.append('category', category);
|
||||
}
|
||||
if (path) {
|
||||
params.append('path', path);
|
||||
}
|
||||
const response = await fetch(`${API_BASE}/files?${params.toString()}`);
|
||||
if (!response.ok) {
|
||||
throw new Error(`Failed to fetch files: ${response.statusText}`);
|
||||
@@ -31,12 +35,16 @@ export async function listFiles(category) {
|
||||
* @purpose Uploads a file to the storage system.
|
||||
* @param {File} file - The file to upload.
|
||||
* @param {string} category - Target category.
|
||||
* @param {string} [path] - Target subpath.
|
||||
* @returns {Promise<Object>}
|
||||
*/
|
||||
export async function uploadFile(file, category) {
|
||||
export async function uploadFile(file, category, path) {
|
||||
const formData = new FormData();
|
||||
formData.append('file', file);
|
||||
formData.append('category', category);
|
||||
if (path) {
|
||||
formData.append('path', path);
|
||||
}
|
||||
|
||||
const response = await fetch(`${API_BASE}/upload`, {
|
||||
method: 'POST',
|
||||
@@ -53,19 +61,19 @@ export async function uploadFile(file, category) {
|
||||
|
||||
// [DEF:deleteFile:Function]
|
||||
/**
|
||||
* @purpose Deletes a file from storage.
|
||||
* @purpose Deletes a file or directory from storage.
|
||||
* @param {string} category - File category.
|
||||
* @param {string} filename - Name of the file.
|
||||
* @param {string} path - Relative path of the item.
|
||||
* @returns {Promise<void>}
|
||||
*/
|
||||
export async function deleteFile(category, filename) {
|
||||
const response = await fetch(`${API_BASE}/files/${category}/${filename}`, {
|
||||
export async function deleteFile(category, path) {
|
||||
const response = await fetch(`${API_BASE}/files/${category}/${path}`, {
|
||||
method: 'DELETE'
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
const errorData = await response.json().catch(() => ({}));
|
||||
throw new Error(errorData.detail || `Failed to delete file: ${response.statusText}`);
|
||||
throw new Error(errorData.detail || `Failed to delete: ${response.statusText}`);
|
||||
}
|
||||
}
|
||||
// [/DEF:deleteFile:Function]
|
||||
@@ -74,11 +82,11 @@ export async function deleteFile(category, filename) {
|
||||
/**
|
||||
* @purpose Returns the URL for downloading a file.
|
||||
* @param {string} category - File category.
|
||||
* @param {string} filename - Name of the file.
|
||||
* @param {string} path - Relative path of the file.
|
||||
* @returns {string}
|
||||
*/
|
||||
export function downloadFileUrl(category, filename) {
|
||||
return `${API_BASE}/download/${category}/${filename}`;
|
||||
export function downloadFileUrl(category, path) {
|
||||
return `${API_BASE}/download/${category}/${path}`;
|
||||
}
|
||||
// [/DEF:downloadFileUrl:Function]
|
||||
|
||||
|
||||
@@ -6,30 +6,30 @@
|
||||
|
||||
## User Experience (File Management)
|
||||
|
||||
- [ ] CHK001 Are loading states displayed while fetching the file list? [Completeness]
|
||||
- [ ] CHK002 Is visual feedback provided immediately after file upload starts? [Clarity]
|
||||
- [ ] CHK003 Are error messages user-friendly when upload fails (e.g., "File too large" vs "Error 413")? [Clarity]
|
||||
- [ ] CHK004 Is a confirmation modal shown before permanently deleting a file? [Safety]
|
||||
- [ ] CHK005 Does the UI clearly distinguish between "Backups" and "Repositories" tabs? [Clarity]
|
||||
- [ ] CHK006 Is the file list sortable by Date and Name? [Usability]
|
||||
- [ ] CHK007 Are file sizes formatted in human-readable units (KB, MB, GB)? [Usability]
|
||||
- [ ] CHK008 Is the download action easily accessible for each file item? [Accessibility]
|
||||
- [ ] CHK009 Does the upload component support drag-and-drop interactions? [Usability]
|
||||
- [ ] CHK010 Is the "Upload" button disabled or hidden when no category is selected? [Consistency]
|
||||
- [x] CHK001 Are loading states displayed while fetching the file list? [Completeness]
|
||||
- [x] CHK002 Is visual feedback provided immediately after file upload starts? [Clarity]
|
||||
- [x] CHK003 Are error messages user-friendly when upload fails (e.g., "File too large" vs "Error 413")? [Clarity]
|
||||
- [x] CHK004 Is a confirmation modal shown before permanently deleting a file? [Safety]
|
||||
- [x] CHK005 Does the UI clearly distinguish between "Backups" and "Repositories" tabs? [Clarity]
|
||||
- [x] CHK006 Is the file list sortable by Date and Name? [Usability]
|
||||
- [x] CHK007 Are file sizes formatted in human-readable units (KB, MB, GB)? [Usability]
|
||||
- [x] CHK008 Is the download action easily accessible for each file item? [Accessibility]
|
||||
- [x] CHK009 Does the upload component support drag-and-drop interactions? [Usability]
|
||||
- [x] CHK010 Is the "Upload" button disabled or hidden when no category is selected? [Consistency]
|
||||
|
||||
## Configuration Flexibility
|
||||
|
||||
- [ ] CHK011 Can the storage root path be configured to any writable directory on the server? [Flexibility]
|
||||
- [ ] CHK012 Does the system support defining custom directory structures using variables like `{environment}`? [Flexibility]
|
||||
- [ ] CHK013 Does the system support defining custom filename patterns using variables like `{timestamp}`? [Flexibility]
|
||||
- [x] CHK011 Can the storage root path be configured to any writable directory on the server? [Flexibility]
|
||||
- [x] CHK012 Does the system support defining custom directory structures using variables like `{environment}`? [Flexibility]
|
||||
- [x] CHK013 Does the system support defining custom filename patterns using variables like `{timestamp}`? [Flexibility]
|
||||
- [ ] CHK014 Are the supported pattern variables (e.g., `{dashboard_name}`) clearly documented in the UI? [Clarity]
|
||||
- [ ] CHK015 Can the configuration be updated without restarting the application? [Usability]
|
||||
- [x] CHK015 Can the configuration be updated without restarting the application? [Usability]
|
||||
- [ ] CHK016 Is the current resolved path shown as a preview when editing patterns? [Usability]
|
||||
- [ ] CHK017 Does the system allow reverting configuration to default values? [Recovery]
|
||||
|
||||
## Edge Cases & Error Handling
|
||||
|
||||
- [ ] CHK018 Is the UI behavior defined for an empty file list (zero state)? [Coverage]
|
||||
- [x] CHK018 Is the UI behavior defined for an empty file list (zero state)? [Coverage]
|
||||
- [ ] CHK019 Is the behavior defined when the configured storage path becomes inaccessible? [Resilience]
|
||||
- [ ] CHK020 Are long filenames handled gracefully in the UI (e.g., truncation with tooltip)? [Layout]
|
||||
- [ ] CHK021 Is the behavior defined for uploading a file with a duplicate name? [Conflict Resolution]
|
||||
- [x] CHK020 Are long filenames handled gracefully in the UI (e.g., truncation with tooltip)? [Layout]
|
||||
- [x] CHK021 Is the behavior defined for uploading a file with a duplicate name? [Conflict Resolution]
|
||||
@@ -5,7 +5,7 @@
|
||||
|
||||
## Summary
|
||||
|
||||
This feature implements a managed file storage system for dashboard backups and exported repositories. It introduces a configurable storage root (defaulting to outside the workspace) and a Web UI to list, upload, download, and delete files. The system enforces a structured layout with `backups/` and `repositories/` subdirectories to keep artifacts organized, while allowing flexible configuration of directory structures and filename patterns.
|
||||
This feature implements a managed file storage system for dashboard backups and exported repositories. It introduces a configurable storage root (defaulting to outside the workspace) and a Web UI to list, upload, download, and delete files. The system enforces a structured layout with `backups/` and `repositories/` subdirectories to keep artifacts organized. The UI supports hierarchical folder navigation (e.g., `backups/SS2/DashboardName`), allowing users to browse, download, and manage files within nested directories.
|
||||
|
||||
## Technical Context
|
||||
|
||||
@@ -88,7 +88,8 @@ frontend/
|
||||
│ │ └── +page.svelte # Main storage UI
|
||||
│ ├── components/
|
||||
│ │ └── storage/
|
||||
│ │ ├── FileList.svelte # Component for listing files
|
||||
│ │ ├── FileList.svelte # Component for listing files and folders (explorer view)
|
||||
│ │ ├── Breadcrumbs.svelte # Component for navigation
|
||||
│ │ └── FileUpload.svelte # Component for uploading
|
||||
│ └── services/
|
||||
│ └── storageService.js # Frontend API client
|
||||
|
||||
@@ -9,18 +9,19 @@
|
||||
|
||||
### User Story 1 - File Management Dashboard (Priority: P1)
|
||||
|
||||
Users need a visual interface to manage the artifacts generated by the system (dashboard backups, exported repositories) without needing direct server access. This ensures that non-technical users or users without SSH access can still retrieve or clean up data.
|
||||
Users need a visual interface to manage the artifacts generated by the system (dashboard backups, exported repositories) without needing direct server access. Users must be able to navigate through the folder structure (e.g., `backups/SS2/Sales Dashboard`) to locate specific files.
|
||||
|
||||
**Why this priority**: Core functionality requested. Without the UI, the storage mechanism is opaque and hard to use.
|
||||
|
||||
**Independent Test**: Can be fully tested by opening the new "File Storage" page, uploading a test file, verifying it appears in the list with correct metadata, downloading it, and then deleting it.
|
||||
**Independent Test**: Can be fully tested by opening the new "File Storage" page, navigating into a subdirectory, uploading a test file, verifying it appears in the list, downloading it, and then deleting it.
|
||||
|
||||
**Acceptance Scenarios**:
|
||||
|
||||
1. **Given** the File Storage page is open, **When** I view the list, **Then** I see all files in the configured storage directory with their names, sizes, and creation dates.
|
||||
2. **Given** a file exists in the list, **When** I click "Download", **Then** the file is downloaded to my local machine.
|
||||
3. **Given** a file exists in the list, **When** I click "Delete" and confirm, **Then** the file is removed from the list and the server filesystem.
|
||||
4. **Given** I have a file locally, **When** I drag and drop it or use the "Upload" button, **Then** the file is uploaded to the server storage and appears in the list.
|
||||
1. **Given** the File Storage page is open, **When** I view the list, **Then** I see the top-level folders (e.g., `backups`, `repositories`) or files.
|
||||
2. **Given** I am viewing a folder, **When** I click a subfolder name, **Then** the view updates to show the contents of that subfolder.
|
||||
3. **Given** I am in a subfolder, **When** I click "Download" on a file, **Then** the file is downloaded to my local machine.
|
||||
4. **Given** a file exists in the list, **When** I click "Delete" and confirm, **Then** the file is removed from the list and the server filesystem.
|
||||
5. **Given** I have a file locally, **When** I drag and drop it or use the "Upload" button, **Then** the file is uploaded to the current directory and appears in the list.
|
||||
|
||||
---
|
||||
|
||||
@@ -64,11 +65,13 @@ Administrators need to control where potentially large or sensitive files are st
|
||||
- **FR-001**: System MUST allow configuring a local filesystem root path for storing artifacts.
|
||||
- **FR-002**: The default storage path MUST be configured such that it does not interfere with the application's git repository (e.g., a directory outside the workspace or explicitly git-ignored).
|
||||
- **FR-003**: System MUST enforce a directory structure within the storage root: `backups/` for dashboard backups and `repositories/` for exported repositories.
|
||||
- **FR-004**: System MUST provide a Web UI to list files, organized by their type (Backup vs Repository).
|
||||
- **FR-004**: System MUST provide a Web UI to list files and folders, organized by their type (Backup vs Repository).
|
||||
- **FR-005**: System MUST display file metadata in the UI: Filename, Size, Creation Date.
|
||||
- **FR-006**: System MUST allow users to download files from the storage directory via the Web UI.
|
||||
- **FR-006**: System MUST allow users to download files from the storage directory (including subdirectories) via the Web UI.
|
||||
- **FR-007**: System MUST allow users to delete files from the storage directory via the Web UI.
|
||||
- **FR-008**: System MUST allow users to upload files to the storage directory via the Web UI, requiring them to select the target category (Backup or Repository).
|
||||
- **FR-008**: System MUST allow users to upload files to the specific folder in the storage directory via the Web UI.
|
||||
- **FR-013**: System MUST support navigating through the directory hierarchy within the allowed categories.
|
||||
- **FR-014**: System MUST display breadcrumbs or similar navigation aid to show current path.
|
||||
- **FR-009**: System MUST validate that the configured storage path is accessible and writable.
|
||||
- **FR-010**: System MUST prevent access to files outside the configured storage directory (Path Traversal protection).
|
||||
- **FR-011**: System MUST allow configuring the directory structure pattern for backups and repositories (e.g., `{environment}/{dashboard_name}/`).
|
||||
|
||||
@@ -64,6 +64,17 @@
|
||||
- [x] T035 Verify large file upload support (50MB+) in Nginx/FastAPI config if applicable
|
||||
- [x] T036 Add confirmation modal for file deletion
|
||||
|
||||
## Phase 6: Folder Structure Support (Refactor)
|
||||
*Goal: Enable hierarchical navigation, nested file management, and downloading.*
|
||||
|
||||
- [x] T037 Refactor `StoragePlugin.list_files` in `backend/src/plugins/storage/plugin.py` to accept `subpath` and return directories/files
|
||||
- [x] T038 Refactor `StoragePlugin` methods (`save_file`, `delete_file`, `get_file_path`) to support nested paths
|
||||
- [x] T039 Update backend endpoints in `backend/src/api/routes/storage.py` (`GET /files`, `POST /upload`, `DELETE /files`, `GET /download`) to accept `path` parameter
|
||||
- [x] T040 Update `frontend/src/services/storageService.js` to pass `path` argument in all API calls
|
||||
- [x] T041 Update `frontend/src/components/storage/FileList.svelte` to display folder icons, handle navigation events, and show breadcrumbs
|
||||
- [x] T042 Update `frontend/src/components/storage/FileUpload.svelte` to upload to the currently active directory
|
||||
- [x] T043 Update `frontend/src/routes/tools/storage/+page.svelte` to manage current path state and handle navigation logic
|
||||
|
||||
## Dependencies
|
||||
|
||||
1. **Phase 1 (Setup)**: No dependencies.
|
||||
@@ -71,6 +82,7 @@
|
||||
3. **Phase 3 (US1)**: Depends on Phase 2.
|
||||
4. **Phase 4 (US2)**: Depends on Phase 2. Can run parallel to Phase 3.
|
||||
5. **Phase 5 (Polish)**: Depends on Phase 3 and 4.
|
||||
6. **Phase 6 (Refactor)**: Depends on Phase 3.
|
||||
|
||||
## Parallel Execution Examples
|
||||
|
||||
|
||||
Reference in New Issue
Block a user