# Feature Specification: Task Logging System (Separated Per-Task Logs) **Feature Branch**: `018-task-logging-v2` **Reference UX**: `specs/018-task-logging-v2/ux_reference.md` **Created**: 2026-02-07 **Status**: Draft **Input**: User description: "Implement a separated per-task logging system with source attribution, persistence, and frontend filtering." ## User Scenarios & Testing *(mandatory)* ### User Story 1 - Real-time Filtered Logging (Priority: P1) As a user, I want to see logs from a running task filtered by their source so that I can focus on specific component behavior without noise. **Why this priority**: This is the core value proposition—isolating component logs during execution. **Independent Test**: Start a task that emits logs from multiple sources (e.g., `plugin` and `system`), apply a source filter in the UI, and verify only matching logs are displayed in real-time. **Acceptance Scenarios**: 1. **Given** a task is running and emitting logs from "plugin" and "superset_api", **When** I select "superset_api" in the Source filter, **Then** only logs with the "superset_api" tag are visible. 2. **Given** a filter is active, **When** a new log matching the filter arrives via WebSocket, **Then** it is immediately appended to the view. --- ### User Story 2 - Persistent Log History (Priority: P1) As a user, I want my task logs to be saved permanently so that I can review them even after the server restarts. **Why this priority**: Current logs are in-memory and lost on restart, which is a major pain point for debugging past failures. **Independent Test**: Run a task, restart the backend server, and verify that the logs for that task are still accessible via the UI. **Acceptance Scenarios**: 1. **Given** a task has completed, **When** I restart the server and navigate to the task details, **Then** all logs are loaded from the database. 2. **Given** a task was deleted via the cleanup service, **When** I check the database, **Then** all associated logs are also removed. --- ### User Story 3 - Component Attribution (Priority: P2) As a developer, I want to use a dedicated logger that automatically tags my logs with the correct source. **Why this priority**: Simplifies plugin development and ensures consistent data for filtering. **Independent Test**: Update a plugin to use `context.logger.with_source("custom")` and verify that logs appear with the "custom" tag. **Acceptance Scenarios**: 1. **Given** a plugin uses the new `TaskContext`, **When** it calls `logger.info()`, **Then** the log is recorded with `source="plugin"` by default. 2. **Given** a plugin creates a sub-logger with `with_source("api")`, **When** it logs a message, **Then** the log is recorded with `source="api"`. --- ### User Story 4 - Configurable Logging Levels (Priority: P1) As an admin, I want to configure logging levels through the frontend so that I can control verbosity and filter noise during debugging. **Why this priority**: Essential for production debugging and performance tuning. **Independent Test**: Change the log level from INFO to DEBUG in admin settings, run a task, and verify that DEBUG-level logs (including belief_scope) now appear. **Acceptance Scenarios**: 1. **Given** I am an admin user, **When** I navigate to Admin Settings > Logging, **Then** I see options for log level, task log level, and belief_scope toggle. 2. **Given** belief_scope logging is enabled, **When** I set log level to DEBUG, **Then** all belief_scope Entry/Exit/Coherence logs are visible. 3. **Given** belief_scope logging is enabled, **When** I set log level to INFO, **Then** belief_scope logs are hidden (they are DEBUG level). 4. **Given** I set task_log_level to WARNING, **When** a task runs, **Then** only WARNING and ERROR logs are persisted for that task. --- ### Edge Cases - **High Volume Logs**: How does the system handle a task generating 10,000+ logs in a few seconds? (Requirement: Batching and virtual scrolling). - **Database Connection Loss**: What happens if the log flusher cannot write to the DB? (Requirement: Logs should remain in-memory as a fallback until the next flush). - **Concurrent Tasks**: Ensure logs from Task A never leak into the view or database records of Task B. ## Requirements *(mandatory)* ### Functional Requirements - **FR-001**: System MUST persist task logs in a dedicated `task_logs` database table. - **FR-002**: Each log entry MUST include: `task_id`, `timestamp`, `level`, `message`, `source`, and optional `metadata`. - **FR-003**: System MUST provide a `TaskLogger` via a `TaskContext` to all plugins. - **FR-004**: System MUST support batch-writing logs to the database (e.g., every 2 seconds) to maintain performance. - **FR-005**: Backend MUST support server-side filtering of logs by `level`, `source`, and text `search` via REST and WebSocket. - **FR-006**: Frontend MUST provide a UI for filtering and searching logs within the task details view. - **FR-007**: System MUST maintain backward compatibility for plugins using the old `execute` signature. - **FR-008**: System MUST automatically delete logs when their associated task is deleted. - **FR-009**: Task logs MUST follow the same retention policy as task records (logs are kept as long as the task record exists). - **FR-010**: The default log level filter in the UI MUST be set to INFO and above (hiding DEBUG logs by default). - **FR-011**: System MUST support filtering logs by specific keys within the `metadata` JSON (e.g., `dashboard_id`, `database_uuid`). - **FR-012**: System MUST separate log levels clearly: DEBUG (development/tracing), INFO (normal operations), WARNING (potential issues), ERROR (failures). - **FR-013**: `belief_scope` context manager MUST log at DEBUG level (not INFO) to reduce noise in production. - **FR-014**: Admin users MUST be able to configure logging levels through the frontend settings UI. - **FR-015**: System MUST support separate log level configuration for application logs vs task-specific logs. - **FR-016**: All plugins MUST support `TaskContext` for proper source attribution and log level filtering. ## Clarifications ### Session 2026-02-07 - Q: Should task logs follow the same retention period as the task records themselves? → A: Same as Task Records. - Q: What should be the default log level filter when a user opens the task log panel? → A: INFO and above. - Q: Do we need to support searching or filtering inside the JSON metadata? → A: Searchable keys (e.g., `dashboard_id` should be filterable). ### Key Entities *(include if feature involves data)* - **TaskLog**: Represents a single log message. - Attributes: `id` (PK), `task_id` (FK), `timestamp`, `level` (Enum/String), `source` (String), `message` (Text), `metadata` (JSON). - **TaskContext**: Execution context provided to plugins. - Attributes: `task_id`, `logger`, `config`. ## Success Criteria *(mandatory)* ### Measurable Outcomes - **SC-001**: Log retrieval for a task with 1,000 entries takes less than 200ms. - **SC-002**: Database write overhead for logging does not increase task execution time by more than 5%. - **SC-003**: 100% of logs generated by a task are available after a server restart. - **SC-004**: Users can filter logs by source and see results in under 100ms on the frontend. - **SC-005**: Admin users can change logging levels via frontend UI without server restart. - **SC-006**: All plugins use TaskContext for logging with proper source attribution.