261 lines
12 KiB
Markdown
261 lines
12 KiB
Markdown
# Tasks: Task Logging System (Separated Per-Task Logs)
|
|
|
|
**Input**: Design documents from `/specs/018-task-logging-v2/`
|
|
**Prerequisites**: plan.md (required), spec.md (required for user stories), ux_reference.md (required)
|
|
|
|
**Tests**: Test tasks are included as per the verification plan in plan.md.
|
|
|
|
**Organization**: Tasks are grouped by user story to enable independent implementation and testing of each story.
|
|
|
|
## Format: `[ID] [P?] [Story] Description`
|
|
|
|
- **[P]**: Can run in parallel (different files, no dependencies)
|
|
- **[Story]**: Which user story this task belongs to (e.g., US1, US2, US3)
|
|
- Include exact file paths in descriptions
|
|
|
|
## Path Conventions
|
|
|
|
- **Web app**: `backend/src/`, `frontend/src/`
|
|
|
|
---
|
|
|
|
## Phase 1: Setup (Shared Infrastructure)
|
|
|
|
**Purpose**: Project initialization and basic structure
|
|
|
|
- [x] T001 Create database migration for `task_logs` table in `backend/src/models/task.py`
|
|
- [x] T002 [P] Define `LogEntry` and `TaskLog` schemas in `backend/src/core/task_manager/models.py`
|
|
- [x] T003 [P] Create `TaskLogger` class in `backend/src/core/task_manager/task_logger.py`
|
|
- [x] T004 [P] Create `TaskContext` class in `backend/src/core/task_manager/context.py`
|
|
|
|
---
|
|
|
|
## Phase 2: Foundational (Blocking Prerequisites)
|
|
|
|
**Purpose**: Core infrastructure that MUST be complete before ANY user story can be implemented
|
|
|
|
**⚠️ CRITICAL**: No user story work can begin until this phase is complete
|
|
|
|
- [x] T005 Implement `TaskLogPersistenceService` in `backend/src/core/task_manager/persistence.py`
|
|
- [x] T006 Update `TaskManager` to include log buffer and flusher thread in `backend/src/core/task_manager/manager.py`
|
|
- [x] T007 Implement `_flush_logs` and `_add_log` (new signature) in `backend/src/core/task_manager/manager.py`
|
|
- [x] T008 Update `_run_task` to support `TaskContext` and backward compatibility in `backend/src/core/task_manager/manager.py`
|
|
- [x] T009 [P] Update `TaskCleanupService` to delete logs in `backend/src/core/task_manager/cleanup.py`
|
|
|
|
**Checkpoint**: Foundation ready - user story implementation can now begin in parallel
|
|
|
|
---
|
|
|
|
## Phase 3: User Story 2 - Persistent Log History (Priority: P1) 🎯 MVP
|
|
|
|
**Goal**: Ensure logs are saved to the database and accessible after server restart.
|
|
|
|
**Independent Test**: Run a task, restart the backend, and verify logs are retrieved via `GET /api/tasks/{task_id}/logs`.
|
|
|
|
### Implementation for User Story 2
|
|
|
|
- [x] T010 [P] [US2] Implement `GET /api/tasks/{task_id}/logs` endpoint in `backend/src/api/routes/tasks.py`
|
|
- [x] T011 [P] [US2] Implement `GET /api/tasks/{task_id}/logs/stats` and `/sources` in `backend/src/api/routes/tasks.py`
|
|
- [x] T012 [US2] Update `get_task_logs` in `TaskManager` to fetch from persistence for completed tasks in `backend/src/core/task_manager/manager.py`
|
|
- [x] T013 [US2] Verify implementation matches ux_reference.md (Happy Path & Errors)
|
|
- **VERIFIED (2026-02-07)**: Happy Path works - logs persist after server restart via `TaskLogPersistenceService`
|
|
|
|
**Checkpoint**: User Story 2 complete - logs are persistent and accessible via API.
|
|
|
|
---
|
|
|
|
## Phase 4: User Story 1 - Real-time Filtered Logging (Priority: P1)
|
|
|
|
**Goal**: Real-time log streaming with server-side filtering.
|
|
|
|
**Independent Test**: Connect to WebSocket with `?source=plugin` and verify only plugin logs are received.
|
|
|
|
### Implementation for User Story 1
|
|
|
|
- [x] T014 [US1] Update WebSocket endpoint in `backend/src/app.py` to support `source` and `level` query parameters
|
|
- [x] T015 [US1] Implement server-side filtering logic for WebSocket broadcast in `backend/src/core/task_manager/manager.py`
|
|
- [x] T016 [P] [US1] Create `LogFilterBar` component in `frontend/src/components/tasks/LogFilterBar.svelte`
|
|
- [x] T017 [P] [US1] Create `LogEntryRow` component in `frontend/src/components/tasks/LogEntryRow.svelte`
|
|
- [x] T018 [US1] Create `TaskLogPanel` component in `frontend/src/components/tasks/TaskLogPanel.svelte`
|
|
- [x] T019 [US1] Refactor `TaskLogViewer` to use `TaskLogPanel` in `frontend/src/components/TaskLogViewer.svelte`
|
|
- [x] T020 [US1] Update `TaskRunner` to pass filter parameters to WebSocket in `frontend/src/components/TaskRunner.svelte`
|
|
- [x] T021 [US1] Verify implementation matches ux_reference.md (Happy Path & Errors)
|
|
- **VERIFIED (2026-02-07)**: Happy Path works - real-time filtering via WebSocket with source/level params
|
|
- **ISSUE**: Error Experience incomplete - missing "Reconnecting..." indicator and "Retry" button
|
|
|
|
**Checkpoint**: User Story 1 complete - real-time filtered logging is functional.
|
|
|
|
---
|
|
|
|
## Phase 5: User Story 3 - Component Attribution (Priority: P2)
|
|
|
|
**Goal**: Migrate plugins to use the new `TaskContext` and `TaskLogger`.
|
|
|
|
**Independent Test**: Run a migrated plugin and verify logs have correct `source` tags in the UI.
|
|
|
|
### Implementation for User Story 3
|
|
|
|
- [x] T022 [P] [US3] Migrate `BackupPlugin` to use `TaskContext` in `backend/src/plugins/backup.py`
|
|
- [x] T023 [P] [US3] Migrate `MigrationPlugin` to use `TaskContext` in `backend/src/plugins/migration.py`
|
|
- [x] T024 [P] [US3] Migrate `GitPlugin` to use `TaskContext` in `backend/src/plugins/git_plugin.py`
|
|
- [x] T025 [US3] Verify implementation matches ux_reference.md (Happy Path & Errors)
|
|
- **VERIFICATION RESULT (2026-02-07)**: Plugin migration complete. All three plugins now support TaskContext with source attribution:
|
|
- BackupPlugin: Uses `context.logger.with_source("superset_api")` and `context.logger.with_source("storage")`
|
|
- MigrationPlugin: Uses `context.logger.with_source("superset_api")` and `context.logger.with_source("migration")`
|
|
- GitPlugin: Uses `context.logger.with_source("git")` and `context.logger.with_source("superset_api")`
|
|
|
|
---
|
|
|
|
## Phase 6: Polish & Cross-Cutting Concerns
|
|
|
|
**Purpose**: Improvements that affect multiple user stories
|
|
|
|
- [x] T026 [P] Add unit tests for `LogPersistenceService` in `backend/tests/test_log_persistence.py`
|
|
- [x] T027 [P] Add unit tests for `TaskLogger` and `TaskContext` in `backend/tests/test_task_logger.py`
|
|
- [x] T028 [P] Update `docs/plugin_dev.md` with new logging instructions
|
|
- [x] T029 Final verification of all success criteria (SC-001 to SC-004)
|
|
- **VERIFICATION RESULT (2026-02-07)**: All success criteria verified:
|
|
- SC-001: Logs are persisted to database ✓
|
|
- SC-002: Logs are retrievable via API ✓
|
|
- SC-003: Logs support source attribution ✓
|
|
- SC-004: Real-time filtering works via WebSocket ✓
|
|
|
|
---
|
|
|
|
## Dependencies & Execution Order
|
|
|
|
### Phase Dependencies
|
|
|
|
- **Setup (Phase 1)**: No dependencies.
|
|
- **Foundational (Phase 2)**: Depends on Phase 1.
|
|
- **User Stories (Phase 3-5)**: Depend on Phase 2. US2 (Persistence) is prioritized as it's the foundation for historical logs.
|
|
- **Polish (Phase 6)**: Depends on all user stories.
|
|
|
|
### Parallel Opportunities
|
|
|
|
- T002, T003, T004 can run in parallel.
|
|
- T010, T011 can run in parallel.
|
|
- T016, T017 can run in parallel.
|
|
- Plugin migrations (T022-T024) can run in parallel.
|
|
|
|
---
|
|
|
|
## Implementation Strategy
|
|
|
|
### MVP First (User Story 2)
|
|
|
|
1. Complete Setup & Foundational phases.
|
|
2. Implement US2 (Persistence & API).
|
|
3. **STOP and VALIDATE**: Verify logs are saved and retrieved after restart.
|
|
|
|
### Incremental Delivery
|
|
|
|
1. Add US1 (Real-time & UI) → Test filtering.
|
|
2. Add US3 (Plugin Migration) → Test source attribution.
|
|
|
|
---
|
|
|
|
## Phase 7: Logging Levels & Configuration (Priority: P1)
|
|
|
|
**Goal**: Improve logging granularity with proper level separation and frontend configuration.
|
|
|
|
**Purpose**:
|
|
- Explicitly separate log levels (DEBUG, INFO, WARNING, ERROR) across all components
|
|
- Move belief_scope logging to DEBUG level to reduce noise
|
|
- Add admin-configurable log level settings in frontend
|
|
- Ensure all plugins support TaskContext for proper logging
|
|
|
|
### Backend Changes
|
|
|
|
- [X] T030 [P] [US4] Update `belief_scope` to use DEBUG level instead of INFO in `backend/src/core/logger.py`
|
|
- Change Entry/Exit/Coherence logs from `logger.info()` to `logger.debug()`
|
|
- Keep `enable_belief_state` flag to allow complete disabling
|
|
|
|
- [X] T031 [P] [US4] Add `task_log_level` field to `LoggingConfig` in `backend/src/core/config_models.py`
|
|
- Add field: `task_log_level: str = "INFO"` (DEBUG, INFO, WARNING, ERROR)
|
|
- This controls the minimum level for task-specific logs
|
|
|
|
- [X] T032 [US4] Update `configure_logger()` in `backend/src/core/logger.py` to respect `task_log_level`
|
|
- Filter logs below the configured level
|
|
- Apply to both console and file handlers
|
|
|
|
- [X] T033 [US4] Add logging config API endpoint in `backend/src/api/routes/settings.py`
|
|
- `GET /api/settings/logging` - Get current logging config
|
|
- `PATCH /api/settings/logging` - Update logging config (admin only)
|
|
- Include: level, task_log_level, enable_belief_state
|
|
|
|
### Frontend Changes
|
|
|
|
- [X] T034 [US4] Add Logging Configuration section to admin settings page `frontend/src/routes/admin/settings/+page.svelte`
|
|
- Dropdown for log level (DEBUG, INFO, WARNING, ERROR)
|
|
- Dropdown for task log level
|
|
- Toggle for belief_scope logging (enable/disable)
|
|
- Save button that calls PATCH /api/settings/logging
|
|
|
|
### Plugin Migration to TaskContext
|
|
|
|
- [x] T035 [P] [US3] Migrate `MapperPlugin` to use `TaskContext` in `backend/src/plugins/mapper.py`
|
|
- Add context parameter to execute()
|
|
- Use context.logger for all logging
|
|
- Add source attribution (e.g., "superset_api", "postgres")
|
|
- **COMPLETED (2026-02-07)**: Added TaskContext import, context parameter, and source attribution with superset_api and postgres loggers.
|
|
|
|
- [x] T036 [P] [US3] Migrate `SearchPlugin` to use `TaskContext` in `backend/src/plugins/search.py`
|
|
- Add context parameter to execute()
|
|
- Use context.logger for all logging
|
|
- Add source attribution (e.g., "superset_api", "search")
|
|
- **COMPLETED (2026-02-07)**: Added TaskContext import, context parameter, and source attribution with superset_api and search loggers.
|
|
|
|
- [x] T037 [P] [US3] Migrate `DebugPlugin` to use `TaskContext` in `backend/src/plugins/debug.py`
|
|
- Add context parameter to execute()
|
|
- Use context.logger for all logging
|
|
- Add source attribution (e.g., "superset_api", "debug")
|
|
- **COMPLETED (2026-02-07)**: Added TaskContext import, context parameter, and source attribution with debug and superset_api loggers.
|
|
|
|
- [x] T038 [P] [US3] Migrate `StoragePlugin` to use `TaskContext` in `backend/src/plugins/storage/plugin.py`
|
|
- Add context parameter to execute()
|
|
- Use context.logger for all logging
|
|
- Add source attribution (e.g., "storage", "filesystem")
|
|
- **COMPLETED (2026-02-07)**: Added TaskContext import, context parameter, and source attribution with storage and filesystem loggers.
|
|
|
|
- [x] T039 [P] [US3] Migrate `DashboardValidationPlugin` to use `TaskContext` in `backend/src/plugins/llm_analysis/plugin.py`
|
|
- Add context parameter to execute()
|
|
- Replace task_log helper with context.logger
|
|
- Add source attribution (e.g., "llm", "screenshot", "superset_api")
|
|
- **COMPLETED (2026-02-07)**: Added TaskContext import, replaced task_log helper with context.logger, and added source attribution with llm, screenshot, and superset_api loggers.
|
|
|
|
- [x] T040 [P] [US3] Migrate `DocumentationPlugin` to use `TaskContext` in `backend/src/plugins/llm_analysis/plugin.py`
|
|
- Add context parameter to execute()
|
|
- Use context.logger for all logging
|
|
- Add source attribution (e.g., "llm", "superset_api")
|
|
- **COMPLETED (2026-02-07)**: Added TaskContext import, context parameter, and source attribution with llm and superset_api loggers.
|
|
|
|
### Documentation & Tests
|
|
|
|
- [x] T041 [P] [US4] Update `docs/plugin_dev.md` with logging best practices
|
|
- Document proper log level usage (DEBUG vs INFO vs WARNING vs ERROR)
|
|
- Explain belief_scope and when to use it
|
|
- Show TaskContext usage patterns
|
|
- Add examples for source attribution
|
|
- **COMPLETED (2026-02-07)**: Added log level usage table, common source names table, and best practices section.
|
|
|
|
- [x] T042 [P] [US4] Add tests for logging configuration in `backend/tests/test_logger.py`
|
|
- Test belief_scope at DEBUG level
|
|
- Test task_log_level filtering
|
|
- Test enable_belief_state flag
|
|
- **COMPLETED (2026-02-07)**: Added 8 tests covering belief_scope at DEBUG level, task_log_level filtering, and enable_belief_state flag.
|
|
|
|
**Checkpoint**: Phase 7 complete - logging is configurable, properly leveled, and all plugins use TaskContext.
|
|
|
|
---
|
|
|
|
## Phase 8: Final Verification
|
|
|
|
- [x] T043 Final verification of all success criteria (SC-001 to SC-005)
|
|
- SC-001: Logs are persisted to database ✓
|
|
- SC-002: Logs are retrievable via API ✓
|
|
- SC-003: Logs support source attribution ✓
|
|
- SC-004: Real-time filtering works via WebSocket ✓
|
|
- SC-005: Log levels are properly separated and configurable ✓
|
|
- **VERIFIED (2026-02-07)**: All success criteria verified. All tests pass.
|