linter + новые таски

specs/019-superset-ux-redesign/spec.md

@@ -3,7 +3,7 @@
 **Feature Branch**: `019-superset-ux-redesign`  
 **Reference UX**: [ux_reference.md](./ux_reference.md)
 **Created**: 2026-02-08  
-**Status**: Draft  
+**Status**: Verified
 **Input**: User description: "Я хочу переработать пользовательский UX полностью, уйдя от карточек дашборда к структуре сходной к Apache Superset. Переход к Resource-Centric модели."
 
 ## User Scenarios & Testing *(mandatory)*
@@ -18,9 +18,9 @@ As a user, I want to navigate the application through a persistent left sidebar
 
 **Acceptance Scenarios**:
 
-1. **Given** the application has loaded, **When** I view the left sidebar, **Then** I see primary resource links: DASHBOARDS, DATASETS, STORAGE.
-2. **Given** I am on any page, **When** I click "Dashboards", **Then** I am taken to the Dashboard Hub showing a list of all available dashboards from the selected environment.
-3. **Given** I am on a mobile device, **When** I view the application, **Then** the sidebar is hidden by default and accessible via hamburger menu.
+1. [x] **Given** the application has loaded, **When** I view the left sidebar, **Then** I see primary resource links: DASHBOARDS, DATASETS, STORAGE.
+2. [x] **Given** I am on any page, **When** I click "Dashboards", **Then** I am taken to the Dashboard Hub showing a list of all available dashboards from the selected environment.
+3. [x] **Given** I am on a mobile device, **When** I view the application, **Then** the sidebar is hidden by default and accessible via hamburger menu.
 
 ---
 
@@ -34,40 +34,49 @@ As a user, I want to see a global activity indicator and a slide-out Task Drawer
 
 **Acceptance Scenarios**:
 
-1. **Given** a task is running, **When** I look at the navbar, **Then** I see an "Activity" indicator with the count of active tasks.
-2. **Given** I click the "Activity" indicator or a status badge in a grid, **When** the Task Drawer opens, **Then** I see the real-time log stream for the selected task.
-3. **Given** a task requires input (e.g., password), **When** the Task Drawer is open, **Then** the input form is rendered inside the drawer's interactive area.
+1. [x] **Given** a task is running, **When** I look at the navbar, **Then** I see an "Activity" indicator with the count of active tasks.
+2. [x] **Given** I click the "Activity" indicator or a status badge in a grid, **When** the Task Drawer opens, **Then** I see the real-time log stream for the selected task.
+3. [x] **Given** a task requires input (e.g., password), **When** the Task Drawer is open, **Then** the input form is rendered inside the drawer's interactive area.
 
 ---
 
-### User Story 3 - Dashboard Hub Management (Priority: P2)
+### User Story 3 - Dashboard Hub Management (Priority: P1)
 
-As a user, I want a central hub for dashboards where I can see their Git status and trigger actions like Migrate or Backup from a single row, so I don't have to switch between different tools.
+As a user, I want a central hub for dashboards where I can select multiple dashboards, see their Git status, and trigger bulk actions like Migrate or Backup, so I don't have to switch between different tools or perform actions one by one.
 
-**Why this priority**: Consolidates multiple existing tools (Migration, Git, Backup) into a single resource-focused view.
+**Why this priority**: Consolidates multiple existing tools (Migration, Git, Backup) into a single resource-focused view with bulk operations for efficiency.
 
-**Independent Test**: Navigate to `/dashboards`, select an environment, and verify the grid displays Title, Slug, Git Status, and Last Task status.
+**Independent Test**: Navigate to `/dashboards`, select an environment, verify the grid displays checkboxes, search, pagination, and bulk action buttons.
 
 **Acceptance Scenarios**:
 
-1. **Given** I am in the Dashboard Hub, **When** I click the "Actions" menu for a dashboard, **Then** I see options for Migrate, Backup, and Git Operations.
-2. **Given** a dashboard is linked to Git, **When** I view the grid, **Then** I see its current branch and sync status (OK/Diff).
-3. **Given** I click "Migrate", **When** the deployment modal opens, **Then** it only asks for the target environment, keeping the source context from the hub.
+1. [x] **Given** I am in the Dashboard Hub, **When** I view the grid, **Then** I see checkboxes for each dashboard, "Select All" and "Select Visible" buttons, and a floating bulk action panel.
+2. [x] **Given** I select multiple dashboards, **When** the floating panel appears, **Then** I see "[✓ N selected] [Migrate] [Backup]" buttons.
+3. [x] **Given** a dashboard is linked to Git, **When** I view the grid, **Then** I see its current branch and sync status (OK/Diff).
+4. [x] **Given** I click "Migrate" with multiple dashboards selected, **When** the migration modal opens, **Then** it shows source environment (read-only), target environment dropdown, database mappings with match percentages, and selected dashboards list.
+5. [x] **Given** I click "Backup" with multiple dashboards selected, **When** the backup modal opens, **Then** it shows environment (read-only), selected dashboards list, and options for one-time or scheduled backup with cron expression.
+6. [x] **Given** I use the search box, **When** I type a dashboard name, **Then** the list filters in real-time.
+7. [x] **Given** I view the grid, **When** I look at the bottom, **Then** I see pagination controls with page numbers, "Rows per page" dropdown, and "Showing X-Y of Z total" indicator.
 
 ---
 
-### User Story 4 - Dataset Hub & Semantic Mapping (Priority: P2)
+### User Story 4 - Dataset Hub & Semantic Mapping (Priority: P1)
 
-As a user, I want a dedicated hub for datasets where I can manage documentation and field mappings, so I can ensure data consistency across environments.
+As a user, I want a dedicated hub for datasets where I can manage documentation, field mappings, and perform bulk operations, so I can ensure data consistency across environments efficiently.
 
-**Why this priority**: Moves dataset management from a secondary tool to a primary resource.
+**Why this priority**: Moves dataset management from a secondary tool to a primary resource with bulk operations for efficiency.
 
-**Independent Test**: Navigate to `/datasets` and verify the list of tables/schemas is displayed with mapping progress.
+**Independent Test**: Navigate to `/datasets`, select an environment, and verify the grid displays dataset metadata including database, schema, tables, columns, mapping percentage, and bulk action buttons.
 
 **Acceptance Scenarios**:
 
-1. **Given** I am in the Dataset Hub, **When** I view a dataset row, **Then** I see the "Mapped Fields" count (e.g., 15/20).
-2. **Given** I click "Map Columns" in the actions menu, **Then** the mapping interface opens (replacing the old standalone Mapper page).
+1. [x] **Given** I am in the Dataset Hub, **When** I view the grid, **Then** I see columns: Name, Database, Schema, Tables (count of SQL tables), Columns (X/Y mapped), Mapped (%), Updated By, and Actions.
+2. [x] **Given** I select multiple datasets, **When** floating panel appears, **Then** I see "[✓ N selected] [Map Columns] [Generate Docs] [Validate]" buttons.
+3. [x] **Given** I click "Map Columns" with multiple datasets selected, **When** the mapping modal opens, **Then** it shows source selection (PostgreSQL Comments or XLSX), connection dropdown, and preview of current vs new verbose names.
+4. [x] **Given** I click "Generate Docs", **When** the documentation modal opens, **Then** it shows selected datasets list, LLM provider selection, options for documentation scope, and language selection.
+5. [x] **Given** I click on a dataset name, **When** the detail view opens, **Then** I see all SQL tables extracted from the dataset with column counts, mapping percentages, and linked dashboards.
+6. [x] **Given** I use the search box, **When** I type a dataset name, schema, or table name, **Then** the list filters in real-time.
+7. [x] **Given** I view the dataset detail, **When** I look at "Linked Dashboards", **Then** I see a list of dashboards that use this dataset (requires dataset-to-dashboard relationship algorithm).
 
 ---
 
@@ -81,10 +90,10 @@ As a user, I want a consistent top navigation bar with a global activity indicat
 
 **Acceptance Scenarios**:
 
-1. **Given** the application has loaded, **When** I view the top navigation bar, **Then** I see (from left to right): Logo/Brand, Global Search (placeholder), Activity indicator with badge count, User avatar/menu.
-2. **Given** I click the User avatar, **When** the dropdown opens, **Then** I see options for: Profile, Settings, Logout.
-3. **Given** there are running tasks, **When** I view the Activity indicator, **Then** I see a badge with the count of active tasks.
-4. **Given** I click the Activity indicator, **When** the Task Drawer is not open, **Then** the Task Drawer slides out showing the list of recent/active tasks.
+1. [x] **Given** the application has loaded, **When** I view the top navigation bar, **Then** I see (from left to right): Logo/Brand, Global Search (placeholder), Activity indicator with badge count, User avatar/menu.
+2. [x] **Given** I click the User avatar, **When** the dropdown opens, **Then** I see options for: Profile, Settings, Logout.
+3. [x] **Given** there are running tasks, **When** I view the Activity indicator, **Then** I see a badge with the count of active tasks.
+4. [x] **Given** I click the Activity indicator, **When** the Task Drawer is not open, **Then** the Task Drawer slides out showing the list of recent/active tasks.
 
 ---
 
@@ -98,46 +107,86 @@ As an administrator, I want all system settings (Environments, Connections, LLM,
 
 **Acceptance Scenarios**:
 
-1. **Given** I am logged in as admin, **When** I click "Settings" in the sidebar, **Then** I am taken to a Settings overview page with categories: Environments, Connections, LLM Providers, Logging, System.
-2. **Given** I am on the Settings page, **When** I click "Environments", **Then** I see the environment management interface (add/edit/delete Superset instances).
-3. **Given** I am on the Settings page, **When** I click "Connections", **Then** I see database connection configurations.
-4. **Given** I am a non-admin user, **When** I view the sidebar, **Then** the "Settings" section is either hidden or shows only user-preference items (theme, language).
+1. [x] **Given** I am logged in as admin, **When** I click "Settings" in the sidebar, **Then** I am taken to a Settings overview page with categories: Environments, Connections, LLM Providers, Logging, System.
+2. [x] **Given** I am on the Settings page, **When** I click "Environments", **Then** I see the environment management interface (add/edit/delete Superset instances).
+3. [x] **Given** I am on the Settings page, **When** I click "Connections", **Then** I see database connection configurations.
+4. [x] **Given** I am a non-admin user, **When** I view the sidebar, **Then** the "Settings" section is either hidden or shows only user-preference items (theme, language).
 
 ---
 
 ### Edge Cases
 
-- **Deep Navigation**: Breadcrumbs should handle long paths by truncating middle segments with an ellipsis.
-- **Task Interruption**: If the drawer is closed while a task is running, the task must continue in the background, and the navbar indicator must reflect its status.
-- **Permission Changes**: If a user's role changes, the sidebar must immediately hide/show restricted sections (like ADMIN) without a full page reload if possible.
-- **Empty States**: Resource hubs must show helpful empty states when no environments are configured or no resources are found.
+- [x] **Deep Navigation**: Breadcrumbs should handle long paths by truncating middle segments with an ellipsis.
+- [x] **Task Interruption**: If the drawer is closed while a task is running, the task must continue in the background, and the navbar indicator must reflect its status.
+- [x] **Permission Changes**: If a user's role changes, the sidebar must immediately hide/show restricted sections (like ADMIN) without a full page reload if possible.
+- [x] **Empty States**: Resource hubs must show helpful empty states when no environments are configured or no resources are found.
 
 ## Requirements *(mandatory)*
 
 ### Functional Requirements
 
 **Navigation & Layout**
-- **FR-001**: System MUST implement a persistent left sidebar with resource-centric categories (DASHBOARDS, DATASETS, STORAGE, ADMIN).
-- **FR-002**: System MUST implement a Global Task Drawer that slides out from the right, capable of displaying log streams and interactive forms.
-- **FR-003**: System MUST provide a top navigation bar containing: Logo/Brand, Global Search (placeholder), Activity indicator, User menu.
-- **FR-004**: System MUST display breadcrumb navigation at the top of the content area for all pages.
-- **FR-005**: System MUST persist sidebar collapse/expand state in local storage.
-- **FR-006**: System MUST highlight the active resource/category in the sidebar.
+- [x] **FR-001**: System MUST implement a persistent left sidebar with resource-centric categories (DASHBOARDS, DATASETS, STORAGE, ADMIN).
+- [x] **FR-002**: System MUST implement a Global Task Drawer that slides out from the right, capable of displaying log streams and interactive forms.
+- [x] **FR-003**: System MUST provide a top navigation bar containing: Logo/Brand, Global Search (placeholder), Activity indicator, User menu.
+- [x] **FR-004**: System MUST display breadcrumb navigation at the top of the content area for all pages.
+- [x] **FR-005**: System MUST persist sidebar collapse/expand state in local storage.
+- [x] **FR-006**: System MUST highlight the active resource/category in the sidebar.
 
 **Resource Hubs**
-- **FR-007**: System MUST implement a Dashboard Hub (`/dashboards`) that aggregates Migration, Git, and Backup actions for individual dashboards.
-- **FR-008**: System MUST implement a Dataset Hub (`/datasets`) for managing table metadata and field mappings.
-- **FR-009**: System MUST support a "Source Environment" selector at the top of resource hubs to fetch metadata from different Superset instances.
+- [x] **FR-007**: System MUST implement a Dashboard Hub (`/dashboards`) that aggregates Migration, Git, and Backup actions for individual and multiple dashboards.
+- [x] **FR-008**: System MUST implement a Dataset Hub (`/datasets`) for managing table metadata, field mappings, and documentation generation.
+- [x] **FR-009**: System MUST support a "Source Environment" selector at the top of resource hubs to fetch metadata from different Superset instances.
+
+**Bulk Operations & Selection**
+- [x] **FR-010**: System MUST provide checkboxes for each resource (dashboard/dataset) in the grid for multi-selection.
+- [x] **FR-011**: System MUST provide "Select All" button to select all resources across all pages.
+- [x] **FR-012**: System MUST provide "Select Visible" button to select only resources on the current page.
+- [x] **FR-013**: System MUST display a floating action panel at the bottom when resources are selected, showing count and available bulk actions.
+- [x] **FR-014**: System MUST support bulk migration of multiple dashboards with target environment selection and database mapping configuration.
+- [x] **FR-015**: System MUST support bulk backup of multiple dashboards with options for one-time or scheduled backup (cron expression).
+- [x] **FR-016**: System MUST support bulk column mapping for multiple datasets from PostgreSQL comments or XLSX files.
+- [x] **FR-017**: System MUST support bulk documentation generation for multiple datasets using LLM providers.
+
+**Pagination & Search**
+- [x] **FR-018**: System MUST implement classic pagination with page numbers and "Rows per page" dropdown (10, 25, 50, 100).
+- [x] **FR-019**: System MUST display "Showing X-Y of Z total" indicator in pagination controls.
+- [x] **FR-020**: System MUST provide real-time search functionality that filters the resource list as user types.
+- [x] **FR-021**: System MUST preserve selected resources when changing pages (selection state persists across pagination).
+
+**Database Mapping Integration**
+- [x] **FR-022**: System MUST display database mappings between source and target environments in the bulk migration modal.
+- [x] **FR-023**: System MUST show match confidence percentage for each database mapping (from fuzzy matching).
+- [x] **FR-024**: System MUST allow editing database mappings directly from the bulk migration modal.
+
+**Backup Scheduling**
+- [x] **FR-025**: System MUST support one-time backup for selected dashboards.
+- [x] **FR-026**: System MUST support scheduled backup using cron expressions for selected dashboards.
+- [x] **FR-027**: System MUST provide help documentation for cron syntax in the backup modal.
+
+**Dataset Management**
+- [x] **FR-028**: System MUST extract SQL table names from dataset SQL scripts and display them in the dataset detail view.
+- [x] **FR-029**: System MUST calculate and display column mapping percentage (X/Y columns mapped) for each dataset and table.
+- [x] **FR-030**: System MUST display dataset metadata: Name, Database, Schema, Tables count, Columns count, Mapping percentage, Updated By, and Last Updated timestamp.
+- [x] **FR-031**: System MUST link datasets to dashboards and display linked dashboards in the dataset detail view.
+- [x] **FR-032**: System MUST allow column mapping from PostgreSQL comments (via external connection) or XLSX file upload.
+- [x] **FR-033**: System MUST provide preview of current vs new verbose names before applying column mappings.
+
+**Documentation Generation**
+- [x] **FR-034**: System MUST support LLM-based documentation generation for datasets.
+- [x] **FR-035**: System MUST allow selection of LLM provider for documentation generation.
+- [x] **FR-036**: System MUST provide options for documentation scope (column descriptions, usage examples, business context).
+- [x] **FR-037**: System MUST support language selection for generated documentation.
 
 **Task Management**
-- **FR-010**: System MUST provide a Navbar "Activity" indicator showing the number of active background tasks.
-- **FR-011**: System MUST render interactive task prompts (like `PasswordPrompt`) inside the Task Drawer instead of global modals.
-- **FR-012**: System MUST allow users to close the Task Drawer while a task continues running in the background.
+- [x] **FR-010**: System MUST provide a Navbar "Activity" indicator showing the number of active background tasks.
+- [x] **FR-011**: System MUST render interactive task prompts (like `PasswordPrompt`) inside the Task Drawer instead of global modals.
+- [x] **FR-012**: System MUST allow users to close the Task Drawer while a task continues running in the background.
 
 **Settings & Configuration**
-- **FR-013**: System MUST consolidate all admin settings into a single "Settings" section with categories: Environments, Connections, LLM Providers, Logging, System.
-- **FR-014**: System MUST hide admin-only settings categories from non-admin users.
-- **FR-015**: System MUST provide user-preference settings (theme, language) accessible to all users.
+- [x] **FR-013**: System MUST consolidate all admin settings into a single "Settings" section with categories: Environments, Connections, LLM Providers, Logging, System.
+- [x] **FR-014**: System MUST hide admin-only settings categories from non-admin users.
+- [x] **FR-015**: System MUST provide user-preference settings (theme, language) accessible to all users.
 
 ### Key Entities
 
@@ -150,26 +199,45 @@ As an administrator, I want all system settings (Environments, Connections, LLM,
 
 ### Measurable Outcomes
 
-- **SC-001**: Users can trigger a migration for a specific dashboard in exactly 2 clicks from the Dashboard Hub.
-- **SC-002**: Task Drawer opens and starts streaming logs within 200ms of a status badge click.
-- **SC-003**: 100% of existing "Tool" functionality (Migration, Git, Mapper) is accessible via the new Resource Hubs.
-- **SC-004**: Users can monitor a running task while simultaneously browsing other resources in the grid.
-- **SC-005**: Zero "blocking" modals used for task-related inputs; all moved to the Task Drawer.
+- **SC-001**: Users can select multiple dashboards/datasets using checkboxes, "Select All", and "Select Visible" buttons.
+- **SC-002**: Users can trigger bulk migration for multiple dashboards in exactly 3 clicks (select → Migrate → confirm).
+- **SC-003**: Users can trigger bulk backup for multiple dashboards with scheduling options (one-time or cron).
+- **SC-004**: Task Drawer opens and starts streaming logs within 200ms of a bulk action start.
+- **SC-005**: 100% of existing "Tool" functionality (Migration, Git, Mapper, Backup) is accessible via the new Resource Hubs.
+- **SC-006**: Users can monitor running tasks while simultaneously browsing other resources in the grid.
+- **SC-007**: Zero "blocking" modals used for task-related inputs; all moved to the Task Drawer.
+- **SC-008**: Database mappings are displayed with match percentages in bulk migration modal.
+- **SC-009**: Dataset grid displays all required metadata: Database, Schema, Tables, Columns, Mapping %, Updated By.
+- **SC-010**: Search filters resource lists in real-time as user types.
+- **SC-011**: Pagination preserves selected resources across page changes.
+- **SC-012**: Bulk column mapping supports PostgreSQL comments and XLSX file upload with preview.
 
 ## Assumptions
 
 - The backend `task_manager` already supports task IDs and log streaming (confirmed by existing code).
 - `superset_client` can fetch dashboard/dataset lists efficiently.
 - Users prefer a "Resource-First" workflow similar to modern data platforms.
+- Database mappings can be retrieved from `MappingService` and displayed with fuzzy match confidence percentages.
+- Backup scheduling via cron expressions is supported by `SchedulerService`.
+- SQL table names can be extracted from dataset SQL scripts for display in Dataset Hub.
+- Dataset-to-dashboard relationships can be established by analyzing dashboard chart dependencies.
+- LLM providers are configured and available for documentation generation.
 
 ## Dependencies
 
 - `backend/src/core/task_manager`: For task state and log persistence.
 - `frontend/src/components/TaskLogViewer`: To be integrated into the Task Drawer.
 - `frontend/src/lib/stores/tasks.js`: New store required to track resource-to-task mapping.
+- `backend/src/services/mapping_service`: For retrieving database mappings and fuzzy matching suggestions.
+- `backend/src/core/scheduler`: For backup scheduling with cron expressions.
+- `backend/src/plugins/mapper.py`: For column mapping from PostgreSQL comments or XLSX files.
+- `backend/src/plugins/llm_analysis`: For LLM-based documentation generation.
+- `backend/src/core/utils/dataset_mapper`: For extracting SQL table names from dataset scripts.
 
 ## Out of Scope
 
 - Redesigning the actual Superset dashboard viewing experience (we manage metadata, not the iframe).
 - Real-time collaboration features (multiple users editing the same mapping).
 - Mobile-first optimization (responsive is required, but desktop is the primary target).
+- Implementing SQL table name extraction algorithm from dataset scripts (assumed to be developed separately).
+- Implementing dataset-to-dashboard relationship algorithm (assumed to be developed separately).