ss-tools/specs/018-task-logging-v2/spec.md

Feature Specification: Task Logging System (Separated Per-Task Logs)

Feature Branch: 018-task-logging-v2
Reference UX: specs/018-task-logging-v2/ux_reference.md Created: 2026-02-07
Status: Draft
Input: User description: "Implement a separated per-task logging system with source attribution, persistence, and frontend filtering."

User Scenarios & Testing (mandatory)

User Story 1 - Real-time Filtered Logging (Priority: P1)

As a user, I want to see logs from a running task filtered by their source so that I can focus on specific component behavior without noise.

Why this priority: This is the core value proposition—isolating component logs during execution.

Independent Test: Start a task that emits logs from multiple sources (e.g., plugin and system), apply a source filter in the UI, and verify only matching logs are displayed in real-time.

Acceptance Scenarios:

1. Given a task is running and emitting logs from "plugin" and "superset_api", When I select "superset_api" in the Source filter, Then only logs with the "superset_api" tag are visible.
2. Given a filter is active, When a new log matching the filter arrives via WebSocket, Then it is immediately appended to the view.

---

User Story 2 - Persistent Log History (Priority: P1)

As a user, I want my task logs to be saved permanently so that I can review them even after the server restarts.

Why this priority: Current logs are in-memory and lost on restart, which is a major pain point for debugging past failures.

Independent Test: Run a task, restart the backend server, and verify that the logs for that task are still accessible via the UI.

Acceptance Scenarios:

1. Given a task has completed, When I restart the server and navigate to the task details, Then all logs are loaded from the database.
2. Given a task was deleted via the cleanup service, When I check the database, Then all associated logs are also removed.

---

User Story 3 - Component Attribution (Priority: P2)

As a developer, I want to use a dedicated logger that automatically tags my logs with the correct source.

Why this priority: Simplifies plugin development and ensures consistent data for filtering.

Independent Test: Update a plugin to use context.logger.with_source("custom") and verify that logs appear with the "custom" tag.

Acceptance Scenarios:

1. Given a plugin uses the new TaskContext, When it calls logger.info(), Then the log is recorded with source="plugin" by default.
2. Given a plugin creates a sub-logger with with_source("api"), When it logs a message, Then the log is recorded with source="api".

---

User Story 4 - Configurable Logging Levels (Priority: P1)

As an admin, I want to configure logging levels through the frontend so that I can control verbosity and filter noise during debugging.

Why this priority: Essential for production debugging and performance tuning.

Independent Test: Change the log level from INFO to DEBUG in admin settings, run a task, and verify that DEBUG-level logs (including belief_scope) now appear.

Acceptance Scenarios:

1. Given I am an admin user, When I navigate to Admin Settings > Logging, Then I see options for log level, task log level, and belief_scope toggle.
2. Given belief_scope logging is enabled, When I set log level to DEBUG, Then all belief_scope Entry/Exit/Coherence logs are visible.
3. Given belief_scope logging is enabled, When I set log level to INFO, Then belief_scope logs are hidden (they are DEBUG level).
4. Given I set task_log_level to WARNING, When a task runs, Then only WARNING and ERROR logs are persisted for that task.

---

Edge Cases

• High Volume Logs: How does the system handle a task generating 10,000+ logs in a few seconds? (Requirement: Batching and virtual scrolling).
• Database Connection Loss: What happens if the log flusher cannot write to the DB? (Requirement: Logs should remain in-memory as a fallback until the next flush).
• Concurrent Tasks: Ensure logs from Task A never leak into the view or database records of Task B.

Requirements (mandatory)

Functional Requirements

• FR-001: System MUST persist task logs in a dedicated task_logs database table.
• FR-002: Each log entry MUST include: task_id, timestamp, level, message, source, and optional metadata.
• FR-003: System MUST provide a TaskLogger via a TaskContext to all plugins.
• FR-004: System MUST support batch-writing logs to the database (e.g., every 2 seconds) to maintain performance.
• FR-005: Backend MUST support server-side filtering of logs by level, source, and text search via REST and WebSocket.
• FR-006: Frontend MUST provide a UI for filtering and searching logs within the task details view.
• FR-007: System MUST maintain backward compatibility for plugins using the old execute signature.
• FR-008: System MUST automatically delete logs when their associated task is deleted.
• FR-009: Task logs MUST follow the same retention policy as task records (logs are kept as long as the task record exists).
• FR-010: The default log level filter in the UI MUST be set to INFO and above (hiding DEBUG logs by default).
• FR-011: System MUST support filtering logs by specific keys within the metadata JSON (e.g., dashboard_id, database_uuid).
• FR-012: System MUST separate log levels clearly: DEBUG (development/tracing), INFO (normal operations), WARNING (potential issues), ERROR (failures).
• FR-013: belief_scope context manager MUST log at DEBUG level (not INFO) to reduce noise in production.
• FR-014: Admin users MUST be able to configure logging levels through the frontend settings UI.
• FR-015: System MUST support separate log level configuration for application logs vs task-specific logs.
• FR-016: All plugins MUST support TaskContext for proper source attribution and log level filtering.

Clarifications

Session 2026-02-07

• Q: Should task logs follow the same retention period as the task records themselves? → A: Same as Task Records.
• Q: What should be the default log level filter when a user opens the task log panel? → A: INFO and above.
• Q: Do we need to support searching or filtering inside the JSON metadata? → A: Searchable keys (e.g., dashboard_id should be filterable).

Key Entities (include if feature involves data)

• TaskLog: Represents a single log message.
  • Attributes: id (PK), task_id (FK), timestamp, level (Enum/String), source (String), message (Text), metadata (JSON).
• TaskContext: Execution context provided to plugins.
  • Attributes: task_id, logger, config.

Success Criteria (mandatory)

Measurable Outcomes

• SC-001: Log retrieval for a task with 1,000 entries takes less than 200ms.
• SC-002: Database write overhead for logging does not increase task execution time by more than 5%.
• SC-003: 100% of logs generated by a task are available after a server restart.
• SC-004: Users can filter logs by source and see results in under 100ms on the frontend.
• SC-005: Admin users can change logging levels via frontend UI without server restart.
• SC-006: All plugins use TaskContext for logging with proper source attribution.