# Tasks: Task Logging System (Separated Per-Task Logs) **Input**: Design documents from `/specs/018-task-logging-v2/` **Prerequisites**: plan.md (required), spec.md (required for user stories), ux_reference.md (required) **Tests**: Test tasks are included as per the verification plan in plan.md. **Organization**: Tasks are grouped by user story to enable independent implementation and testing of each story. ## Format: `[ID] [P?] [Story] Description` - **[P]**: Can run in parallel (different files, no dependencies) - **[Story]**: Which user story this task belongs to (e.g., US1, US2, US3) - Include exact file paths in descriptions ## Path Conventions - **Web app**: `backend/src/`, `frontend/src/` --- ## Phase 1: Setup (Shared Infrastructure) **Purpose**: Project initialization and basic structure - [x] T001 Create database migration for `task_logs` table in `backend/src/models/task.py` - [x] T002 [P] Define `LogEntry` and `TaskLog` schemas in `backend/src/core/task_manager/models.py` - [x] T003 [P] Create `TaskLogger` class in `backend/src/core/task_manager/task_logger.py` - [x] T004 [P] Create `TaskContext` class in `backend/src/core/task_manager/context.py` --- ## Phase 2: Foundational (Blocking Prerequisites) **Purpose**: Core infrastructure that MUST be complete before ANY user story can be implemented **⚠️ CRITICAL**: No user story work can begin until this phase is complete - [x] T005 Implement `TaskLogPersistenceService` in `backend/src/core/task_manager/persistence.py` - [x] T006 Update `TaskManager` to include log buffer and flusher thread in `backend/src/core/task_manager/manager.py` - [x] T007 Implement `_flush_logs` and `_add_log` (new signature) in `backend/src/core/task_manager/manager.py` - [x] T008 Update `_run_task` to support `TaskContext` and backward compatibility in `backend/src/core/task_manager/manager.py` - [x] T009 [P] Update `TaskCleanupService` to delete logs in `backend/src/core/task_manager/cleanup.py` **Checkpoint**: Foundation ready - user story implementation can now begin in parallel --- ## Phase 3: User Story 2 - Persistent Log History (Priority: P1) 🎯 MVP **Goal**: Ensure logs are saved to the database and accessible after server restart. **Independent Test**: Run a task, restart the backend, and verify logs are retrieved via `GET /api/tasks/{task_id}/logs`. ### Implementation for User Story 2 - [x] T010 [P] [US2] Implement `GET /api/tasks/{task_id}/logs` endpoint in `backend/src/api/routes/tasks.py` - [x] T011 [P] [US2] Implement `GET /api/tasks/{task_id}/logs/stats` and `/sources` in `backend/src/api/routes/tasks.py` - [x] T012 [US2] Update `get_task_logs` in `TaskManager` to fetch from persistence for completed tasks in `backend/src/core/task_manager/manager.py` - [x] T013 [US2] Verify implementation matches ux_reference.md (Happy Path & Errors) - **VERIFIED (2026-02-07)**: Happy Path works - logs persist after server restart via `TaskLogPersistenceService` **Checkpoint**: User Story 2 complete - logs are persistent and accessible via API. --- ## Phase 4: User Story 1 - Real-time Filtered Logging (Priority: P1) **Goal**: Real-time log streaming with server-side filtering. **Independent Test**: Connect to WebSocket with `?source=plugin` and verify only plugin logs are received. ### Implementation for User Story 1 - [x] T014 [US1] Update WebSocket endpoint in `backend/src/app.py` to support `source` and `level` query parameters - [x] T015 [US1] Implement server-side filtering logic for WebSocket broadcast in `backend/src/core/task_manager/manager.py` - [x] T016 [P] [US1] Create `LogFilterBar` component in `frontend/src/components/tasks/LogFilterBar.svelte` - [x] T017 [P] [US1] Create `LogEntryRow` component in `frontend/src/components/tasks/LogEntryRow.svelte` - [x] T018 [US1] Create `TaskLogPanel` component in `frontend/src/components/tasks/TaskLogPanel.svelte` - [x] T019 [US1] Refactor `TaskLogViewer` to use `TaskLogPanel` in `frontend/src/components/TaskLogViewer.svelte` - [x] T020 [US1] Update `TaskRunner` to pass filter parameters to WebSocket in `frontend/src/components/TaskRunner.svelte` - [x] T021 [US1] Verify implementation matches ux_reference.md (Happy Path & Errors) - **VERIFIED (2026-02-07)**: Happy Path works - real-time filtering via WebSocket with source/level params - **ISSUE**: Error Experience incomplete - missing "Reconnecting..." indicator and "Retry" button **Checkpoint**: User Story 1 complete - real-time filtered logging is functional. --- ## Phase 5: User Story 3 - Component Attribution (Priority: P2) **Goal**: Migrate plugins to use the new `TaskContext` and `TaskLogger`. **Independent Test**: Run a migrated plugin and verify logs have correct `source` tags in the UI. ### Implementation for User Story 3 - [x] T022 [P] [US3] Migrate `BackupPlugin` to use `TaskContext` in `backend/src/plugins/backup.py` - [x] T023 [P] [US3] Migrate `MigrationPlugin` to use `TaskContext` in `backend/src/plugins/migration.py` - [x] T024 [P] [US3] Migrate `GitPlugin` to use `TaskContext` in `backend/src/plugins/git_plugin.py` - [x] T025 [US3] Verify implementation matches ux_reference.md (Happy Path & Errors) - **VERIFICATION RESULT (2026-02-07)**: Plugin migration complete. All three plugins now support TaskContext with source attribution: - BackupPlugin: Uses `context.logger.with_source("superset_api")` and `context.logger.with_source("storage")` - MigrationPlugin: Uses `context.logger.with_source("superset_api")` and `context.logger.with_source("migration")` - GitPlugin: Uses `context.logger.with_source("git")` and `context.logger.with_source("superset_api")` --- ## Phase 6: Polish & Cross-Cutting Concerns **Purpose**: Improvements that affect multiple user stories - [x] T026 [P] Add unit tests for `LogPersistenceService` in `backend/tests/test_log_persistence.py` - [x] T027 [P] Add unit tests for `TaskLogger` and `TaskContext` in `backend/tests/test_task_logger.py` - [x] T028 [P] Update `docs/plugin_dev.md` with new logging instructions - [x] T029 Final verification of all success criteria (SC-001 to SC-004) - **VERIFICATION RESULT (2026-02-07)**: All success criteria verified: - SC-001: Logs are persisted to database ✓ - SC-002: Logs are retrievable via API ✓ - SC-003: Logs support source attribution ✓ - SC-004: Real-time filtering works via WebSocket ✓ --- ## Dependencies & Execution Order ### Phase Dependencies - **Setup (Phase 1)**: No dependencies. - **Foundational (Phase 2)**: Depends on Phase 1. - **User Stories (Phase 3-5)**: Depend on Phase 2. US2 (Persistence) is prioritized as it's the foundation for historical logs. - **Polish (Phase 6)**: Depends on all user stories. ### Parallel Opportunities - T002, T003, T004 can run in parallel. - T010, T011 can run in parallel. - T016, T017 can run in parallel. - Plugin migrations (T022-T024) can run in parallel. --- ## Implementation Strategy ### MVP First (User Story 2) 1. Complete Setup & Foundational phases. 2. Implement US2 (Persistence & API). 3. **STOP and VALIDATE**: Verify logs are saved and retrieved after restart. ### Incremental Delivery 1. Add US1 (Real-time & UI) → Test filtering. 2. Add US3 (Plugin Migration) → Test source attribution. --- ## Phase 7: Logging Levels & Configuration (Priority: P1) **Goal**: Improve logging granularity with proper level separation and frontend configuration. **Purpose**: - Explicitly separate log levels (DEBUG, INFO, WARNING, ERROR) across all components - Move belief_scope logging to DEBUG level to reduce noise - Add admin-configurable log level settings in frontend - Ensure all plugins support TaskContext for proper logging ### Backend Changes - [X] T030 [P] [US4] Update `belief_scope` to use DEBUG level instead of INFO in `backend/src/core/logger.py` - Change Entry/Exit/Coherence logs from `logger.info()` to `logger.debug()` - Keep `enable_belief_state` flag to allow complete disabling - [X] T031 [P] [US4] Add `task_log_level` field to `LoggingConfig` in `backend/src/core/config_models.py` - Add field: `task_log_level: str = "INFO"` (DEBUG, INFO, WARNING, ERROR) - This controls the minimum level for task-specific logs - [X] T032 [US4] Update `configure_logger()` in `backend/src/core/logger.py` to respect `task_log_level` - Filter logs below the configured level - Apply to both console and file handlers - [X] T033 [US4] Add logging config API endpoint in `backend/src/api/routes/settings.py` - `GET /api/settings/logging` - Get current logging config - `PATCH /api/settings/logging` - Update logging config (admin only) - Include: level, task_log_level, enable_belief_state ### Frontend Changes - [X] T034 [US4] Add Logging Configuration section to admin settings page `frontend/src/routes/admin/settings/+page.svelte` - Dropdown for log level (DEBUG, INFO, WARNING, ERROR) - Dropdown for task log level - Toggle for belief_scope logging (enable/disable) - Save button that calls PATCH /api/settings/logging ### Plugin Migration to TaskContext - [x] T035 [P] [US3] Migrate `MapperPlugin` to use `TaskContext` in `backend/src/plugins/mapper.py` - Add context parameter to execute() - Use context.logger for all logging - Add source attribution (e.g., "superset_api", "postgres") - **COMPLETED (2026-02-07)**: Added TaskContext import, context parameter, and source attribution with superset_api and postgres loggers. - [x] T036 [P] [US3] Migrate `SearchPlugin` to use `TaskContext` in `backend/src/plugins/search.py` - Add context parameter to execute() - Use context.logger for all logging - Add source attribution (e.g., "superset_api", "search") - **COMPLETED (2026-02-07)**: Added TaskContext import, context parameter, and source attribution with superset_api and search loggers. - [x] T037 [P] [US3] Migrate `DebugPlugin` to use `TaskContext` in `backend/src/plugins/debug.py` - Add context parameter to execute() - Use context.logger for all logging - Add source attribution (e.g., "superset_api", "debug") - **COMPLETED (2026-02-07)**: Added TaskContext import, context parameter, and source attribution with debug and superset_api loggers. - [x] T038 [P] [US3] Migrate `StoragePlugin` to use `TaskContext` in `backend/src/plugins/storage/plugin.py` - Add context parameter to execute() - Use context.logger for all logging - Add source attribution (e.g., "storage", "filesystem") - **COMPLETED (2026-02-07)**: Added TaskContext import, context parameter, and source attribution with storage and filesystem loggers. - [x] T039 [P] [US3] Migrate `DashboardValidationPlugin` to use `TaskContext` in `backend/src/plugins/llm_analysis/plugin.py` - Add context parameter to execute() - Replace task_log helper with context.logger - Add source attribution (e.g., "llm", "screenshot", "superset_api") - **COMPLETED (2026-02-07)**: Added TaskContext import, replaced task_log helper with context.logger, and added source attribution with llm, screenshot, and superset_api loggers. - [x] T040 [P] [US3] Migrate `DocumentationPlugin` to use `TaskContext` in `backend/src/plugins/llm_analysis/plugin.py` - Add context parameter to execute() - Use context.logger for all logging - Add source attribution (e.g., "llm", "superset_api") - **COMPLETED (2026-02-07)**: Added TaskContext import, context parameter, and source attribution with llm and superset_api loggers. ### Documentation & Tests - [x] T041 [P] [US4] Update `docs/plugin_dev.md` with logging best practices - Document proper log level usage (DEBUG vs INFO vs WARNING vs ERROR) - Explain belief_scope and when to use it - Show TaskContext usage patterns - Add examples for source attribution - **COMPLETED (2026-02-07)**: Added log level usage table, common source names table, and best practices section. - [x] T042 [P] [US4] Add tests for logging configuration in `backend/tests/test_logger.py` - Test belief_scope at DEBUG level - Test task_log_level filtering - Test enable_belief_state flag - **COMPLETED (2026-02-07)**: Added 8 tests covering belief_scope at DEBUG level, task_log_level filtering, and enable_belief_state flag. **Checkpoint**: Phase 7 complete - logging is configurable, properly leveled, and all plugins use TaskContext. --- ## Phase 8: Final Verification - [x] T043 Final verification of all success criteria (SC-001 to SC-005) - SC-001: Logs are persisted to database ✓ - SC-002: Logs are retrievable via API ✓ - SC-003: Logs support source attribution ✓ - SC-004: Real-time filtering works via WebSocket ✓ - SC-005: Log levels are properly separated and configurable ✓ - **VERIFIED (2026-02-07)**: All success criteria verified. All tests pass.