From b2bbd734397964b1100940ef548110ecbc333ece Mon Sep 17 00:00:00 2001 From: busya Date: Wed, 28 Jan 2026 18:30:23 +0300 Subject: [PATCH] tasks ready --- .kilocode/rules/specify-rules.md | 3 +- backend/backend/auth.db | Bin 147456 -> 0 bytes backend/backend/git_repos/12 | 1 - .../checklists/data-integrity.md | 20 +++ .../checklists/reliability-security.md | 24 ++++ .../checklists/requirements.md | 34 +++++ .../checklists/ux-usability.md | 24 ++++ .../contracts/llm-api.yaml | 96 ++++++++++++++ specs/017-llm-analysis-plugin/data-model.md | 51 ++++++++ specs/017-llm-analysis-plugin/plan.md | 98 ++++++++++++++ specs/017-llm-analysis-plugin/quickstart.md | 42 ++++++ specs/017-llm-analysis-plugin/research.md | 48 +++++++ specs/017-llm-analysis-plugin/spec.md | 122 ++++++++++++++++++ specs/017-llm-analysis-plugin/tasks.md | 103 +++++++++++++++ 14 files changed, 664 insertions(+), 2 deletions(-) delete mode 100644 backend/backend/auth.db delete mode 160000 backend/backend/git_repos/12 create mode 100644 specs/017-llm-analysis-plugin/checklists/data-integrity.md create mode 100644 specs/017-llm-analysis-plugin/checklists/reliability-security.md create mode 100644 specs/017-llm-analysis-plugin/checklists/requirements.md create mode 100644 specs/017-llm-analysis-plugin/checklists/ux-usability.md create mode 100644 specs/017-llm-analysis-plugin/contracts/llm-api.yaml create mode 100644 specs/017-llm-analysis-plugin/data-model.md create mode 100644 specs/017-llm-analysis-plugin/plan.md create mode 100644 specs/017-llm-analysis-plugin/quickstart.md create mode 100644 specs/017-llm-analysis-plugin/research.md create mode 100644 specs/017-llm-analysis-plugin/spec.md create mode 100644 specs/017-llm-analysis-plugin/tasks.md diff --git a/.kilocode/rules/specify-rules.md b/.kilocode/rules/specify-rules.md index 79c3e98..d24793b 100644 --- a/.kilocode/rules/specify-rules.md +++ b/.kilocode/rules/specify-rules.md @@ -32,6 +32,7 @@ Auto-generated from all feature plans. Last updated: 2025-12-19 - Python 3.9+ (Backend), Node.js 18+ (Frontend) + FastAPI (Backend), SvelteKit + Tailwind CSS (Frontend) (015-frontend-nav-redesign) - N/A (UI reorganization and API integration) (015-frontend-nav-redesign) - SQLite (`auth.db`) for Users, Roles, Permissions, and Mappings. (016-multi-user-auth) +- SQLite (existing `tasks.db` for results, `auth.db` for permissions, `mappings.db` or new `plugins.db` for provider config/metadata) (017-llm-analysis-plugin) - Python 3.9+ (Backend), Node.js 18+ (Frontend Build) (001-plugin-arch-svelte-ui) @@ -52,9 +53,9 @@ cd src; pytest; ruff check . Python 3.9+ (Backend), Node.js 18+ (Frontend Build): Follow standard conventions ## Recent Changes +- 017-llm-analysis-plugin: Added Python 3.9+ (Backend), Node.js 18+ (Frontend) - 016-multi-user-auth: Added Python 3.9+ (Backend), Node.js 18+ (Frontend) - 015-frontend-nav-redesign: Added Python 3.9+ (Backend), Node.js 18+ (Frontend) + FastAPI (Backend), SvelteKit + Tailwind CSS (Frontend) -- 014-file-storage-ui: Added Python 3.9+ (Backend), Node.js 18+ (Frontend) + FastAPI (Backend), SvelteKit (Frontend) diff --git a/backend/backend/auth.db b/backend/backend/auth.db deleted file mode 100644 index 1f6ffe005f33a4915a6c6c20d8f2a07d261a4285..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 147456 zcmeI*Uu+vkeg|+;l1cp|DaWVdCC**w*lwgYW6$jWwJ-JQ6vj~{ONk_&V-yEsc6V0d zSt1pZ@|~{_2iccD>4OXTj%?Cj?^yWE*sifeB#)!L#IHk#YKUD8uisdPH^l~O5{O3lds zSLFZh<2m_cr29htPWLOFQ)XJ`-b{MDQcQJf zO&!;#s+ul!h!F-padcl0gZjevg&yi&z=Q2ty;IdKvAZ(2_|97W_Wjn^?rA%B-(GGs zgt_!qQ?CWi-G#M!YiVn1eRpkPYxCXP+iyKs+NrM`QB|GZF!XD>=~Bb&sj4OyJ2$9% z&2X6Im}>risPI9m@S*&H2LvDh0SG_<0uX=z1Rwwb2tWV=r$Jz5xROpX3^Me}aAk}q z7shf`zw z{r{h)3jbdC>BlM^nIQlH2tWV=5P$##AOHafKmY;|I3)t3!_(0az|iP$aV+}KLG=0m z!mp$XesUh%ZMvX z`NsnsOLLiL37xV^OEl-wBbB1bJi&jQ1smhb~F%ClOYdtE<2Qn@(oc`nsy;1M$r zOlC0|aRfDqM*|)j0r%wBnC)!sZq({6mFecxNM*LD*_JSM77)+XI5B)fJ_}_99jbHY zG3sbuxS#1pOWttIH$+s)8wHtp+GByQc|s$uEx*@IeyJKGo+d)#x*9Vn6S`+=@yu$& zPJX0vb(-rI^)=TZwz}on&_bWMzC($wdv0jzn&En`s6VJR8})5bZ|_$*Cv#r(1L~TA z77(FPPv(^0#^yM>K}@;#Ha95uP2SmRAiUj;@;l}E%6j>=Nm=EKPUvaeW-jr$Cy}@* zE6?N|+LmUU+;k1ivdwsvJK5cdk;27$gjJzWVJ1ycq%J1IAcuHj6fV<3EuQKySc|yZV@;<)NMgT z=*Zk&Aa4`bgltHjWre=$i%_smy|?+!j%apr-x?jMkc%d@b<1@EqB9!EThcvSl7%`6 zT-T(bZkZZ$ySZx{^49WNqu$+oR&Acjm@p$Wi5AF^8J;1VBhx}6_t~bt9Z7vBXhau2BEyY-L!0(n+CE`8nT^mYFa@ccRvs!`aJuMmT0$Y^^N@_;l?@D z$U@(;WMn-O7(z7^<`b@&CNVY1C#z*>?xD<4BO4j1TrN6+Asd8d6Dk~eE4~(5g!`5h z2z<-5eAAIx_cz>ZY>EA?+QTyEG<5=7wq6PBdeS>|O~M_Y5<9SY;Cd#rZL23|v`KmQ z|K%ccsclG7VlgILr*50-YYrI+h2h8un1PJNe#TwdmG5j)eiGpFwESQ|=yO|sEuxCE z;c|n>=*VU+D{uLhE5k&4a;iq$>2j)B}_8TkWmSfj47N`36l&Uj6`)zFkUdM5+)fRi0=PSG8~}h|5HAg|1Rwwb2tWV=5P$##AOHafoLB(!|0gCx1_(d^0uX=z1Rwwb2tWV= z5P-m$6F~p}%!LdUh5!U0009U<00Izz00bZa0SKH}AeVhDHI;ckmH(f@pXA=i|Ng}9 zzqAB`?&+nM*XuZ=v)+#UMEk#g#vhW1jwow_go^*nAppDC`9 zbgEVl#J5`SZq?eN%6Hq1=yi3Y)~+^tQsXHEAOHafKwyx-UTG>@ynH$RDBI@Vme>)^ z?OLlO`hETTHj&g^O)o+lrD-9$RvhuBX`O z_3Yj^$0Fp%cOvAdoao1R3^`hKOzTtY2SV>}xzlJB@eXgbzS(F7)lJ^o?8Zi|kk?k5 z)p$Eri)*c_YQYag>8mR%OXay`wa{-0-WEZXw@a_fgJpf;PFXG9;;nXdtFck5#}*$E z)$!w^8trhEMZYET+!qJug8p=%|QpnfZeRJ z0V~EHEuG628B2e6J|b3^*0M=G5Y4LJsE4(UR{xSr47Pv$$1>e%HXhUh(LC@x6p>@8 zcVwIFH1~erK!kjEOZFVZPNP+8H<}M)X;m9;wRwBDb+Fi-J!seu_*N}Y&Hf3Edp?sX z-erTL({EK}{mZhC8pBac-W!WzA_Gwu(HluOCjCoBVlnAoACJkvHXd8iezzAbz_jpUZ+@(}Y51*={oShR=Z@MGSFQwPZZ_HJe)=Tr1 z)z$LydR3it*4F3lycHF^6Gf)?gmip{{)1|MdZvU7X4(%ItGtLaB`QP_G< zQN7QHVxj6?^ZyA_&+!loo>UCv$+5<_TGatpF8vJlP^P$ebx`Z=t!kC^z8aR7C!^{T zC71or)cN5*P8EKf`lKXT_BlS%M^00d5z zz~1DgZ1MK>^xpOEl`ape8_mYYBK3 zZE2>vXm*dklr3Jro__RhbUy7aKiFd5f|p}w*1pxpoL;+WyYEA>Q>p4wmsQVhrjvKJv|4_nEZ@53%WK`fR@CR4D`kq6>x0g?-D(}|a+UQhy8P&cFJ_B` zq#tiYOwks5x~_*`mtshLYh_2bQ|BWsJ}NkoNT;Y*d-~EcUvycOTOk+3#V0^z2J9(T|p^_7=Sw@cw(8 zbL_pjQ^>1P%Q?!glOSfHmhCGX_UmjnmRc?An&~fQinlKfYLn5nqU8I%w0(`2*wAgUwAzIh3I&A@9R;-4sYP_>*r(9 zJG?f2&gknO2C^URHMs0i%NTdAfMe$ld6_6XQ9Z{>UlC2Q-H@{zJ=bylU6-wf&$p_w zcXQBb?@#0JHk+zv>@|7akEc@&p!KlsR}b}XoajJ|HMP#c**S^N^zL8ZQG2j&v6iF+ z<2}gEp=cM6ySQxg)_vJDd^tKHgWCJ*3y;r#KDw@cPecLjUDx}(7zZaK>MM{_*JN zvc=ii^m~g@K;vuUucl)`jjuZ@tsFb-#p?b-2J$7km*9C!WE?Z@|AfAlT-+~d{fqRX`Jtwqu7 zOVGFALM*0ztDkmQqtSX9zv}MpfH&rj;BiSh@sUY-hX{{qVaFvAbvaymFvg++`J|)t z(`pyy`Al))@}T3p=K`t9dR{!I{{G)kAxss1SN^~Q0uX=z1Rwwb2tWV=5P$##AOL}n zA+R_!T&Zx+FqsG`38-xo!{L-TmTnPW1Ue0=9qL>zvutkbE-{6s5yNyDafRu~)RyKl z&k{Oii)mFDbzGmS+%A=?!@wtw?h9g2U--VzL){DH9UWW<-6TR6lo+n76NlN12-7ly zKo4!XT7CY1s327TUwn)mL{fzmwN;f1mrc+~VZ_%Efp<00Izz00bZa0SG_<0ucD@1QsWv z+0Q3WlX{*@bsBiY3 z$0WAT0>hw|@B?pgEMZ}xa2(H}f_QSSmKdJq5zk^8q1+UD;Am9N`!0?qEX-qpuX#cv zt}UmQ4Z~o>(?m#IS7Rn+LibEwXW=TMz9k5NbS!o^I& z!nkfxUvmv&tC?p*3w`4H4kfzoxuL0RhU>YD=Moks0*8mXEr=d%n^KR-sdh`VO>Vk|X4&TAaKge&YU`Hk1Vm>vkW==a zO&llGN#MFB4Ry=Zn7cp!pI~m^cA0SG%slZ;Lk2+2)^koZbvpoKuI_N|hn@NVCm#V+ zGBYDIi5AEx7@i>;Ez?3G3$m$i2b6NE=Km)qe<1t+xxbnG*Fv!%@;}a#-1jGckpHFJ z{AYKVW6KbL00bZa0SG_<0uX?}Gb^w^gPNcNm891{k+`%Z-Tv{!r6uY2k0mZGNymRQ zacN0<{@KK(CF%NS5|@^w?|&|FX-PW&BZ*5(()%AyTw0Rue|P@>lj;Af`Tx|^Gkada zjza(f5P$##AOHafKmY;|fB*!Z2>~_#kN*EN5o+ut1Rwwb2tWV=5P$##AOHafJi`L$ e|3AaQ$F4&F0uX=z1Rwwb2tWV=5P-llBk+GNzQ9%h diff --git a/backend/backend/git_repos/12 b/backend/backend/git_repos/12 deleted file mode 160000 index f467724..0000000 --- a/backend/backend/git_repos/12 +++ /dev/null @@ -1 +0,0 @@ -Subproject commit f46772443ad76fb5f35d2e478cf71078389dd2b7 diff --git a/specs/017-llm-analysis-plugin/checklists/data-integrity.md b/specs/017-llm-analysis-plugin/checklists/data-integrity.md new file mode 100644 index 0000000..63d925d --- /dev/null +++ b/specs/017-llm-analysis-plugin/checklists/data-integrity.md @@ -0,0 +1,20 @@ +# Checklist: Data Integrity Requirements Quality + +**Purpose**: Validate requirement quality for data generation and persistence. +**Feature**: `017-llm-analysis-plugin` +**Created**: 2026-01-28 + +## Requirement Completeness +- [ ] CHK001 Are requirements defined for mapping generated documentation to the correct metadata fields? [Completeness, Spec §FR-017] +- [ ] CHK002 Are requirements specified for handling schema changes during documentation generation? [Completeness, Gap] +- [ ] CHK003 Are requirements defined for validating the format of the generated commit message? [Completeness, Gap] +- [ ] CHK004 Are requirements specified for concurrent documentation updates? [Completeness, Gap] + +## Requirement Clarity +- [ ] CHK005 Is "structured description" defined with specific markdown or text constraints? [Clarity, Spec §SC-004] +- [ ] CHK006 Are "recent commit history" requirements defined by count or time window? [Clarity, Spec §FR-012] + +## Edge Case Coverage +- [ ] CHK007 Are requirements defined for when the LLM generates hallucinated column names? [Edge Case, Gap] +- [ ] CHK008 Are rollback requirements defined if a metadata update fails partially? [Edge Case, Gap] +- [ ] CHK009 Are requirements defined for handling empty or null schema inputs? [Edge Case, Gap] \ No newline at end of file diff --git a/specs/017-llm-analysis-plugin/checklists/reliability-security.md b/specs/017-llm-analysis-plugin/checklists/reliability-security.md new file mode 100644 index 0000000..21375e2 --- /dev/null +++ b/specs/017-llm-analysis-plugin/checklists/reliability-security.md @@ -0,0 +1,24 @@ +# Checklist: Reliability & Security Requirements Quality + +**Purpose**: Validate requirement quality for LLM integration reliability and security. +**Feature**: `017-llm-analysis-plugin` +**Created**: 2026-01-28 + +## Requirement Completeness +- [ ] CHK001 Are retry strategies (count, backoff) defined for all external LLM API calls? [Completeness, Spec §FR-018] +- [ ] CHK002 Are timeout thresholds specified for long-running validation tasks? [Completeness, Gap] +- [ ] CHK003 Are encryption requirements defined for storing API keys at rest? [Completeness, Spec §FR-002] +- [ ] CHK004 Are masking requirements defined for displaying API keys in the UI? [Completeness, Gap] +- [ ] CHK005 Is the fallback behavior defined when the primary screenshot method (Headless) fails? [Completeness, Spec §FR-016] +- [ ] CHK006 Are requirements defined for handling rate limits from LLM providers? [Completeness, Gap] +- [ ] CHK007 Are data privacy requirements specified regarding what dashboard data (screenshots, logs) is sent to the LLM? [Completeness, Gap] + +## Requirement Clarity +- [ ] CHK008 Is "securely store" quantified with specific encryption standards (e.g., AES-256)? [Clarity, Spec §FR-002] +- [ ] CHK009 Are "recent execution logs" defined by specific time window or line count? [Clarity, Spec §FR-006] +- [ ] CHK010 Is "automatic retry logic" defined with specific backoff parameters? [Clarity, Spec §FR-018] + +## Edge Case Coverage +- [ ] CHK011 Are requirements defined for scenarios where the LLM provider is completely unreachable? [Edge Case, Gap] +- [ ] CHK012 Are requirements defined for handling "token limit exceeded" errors with large logs? [Edge Case, Gap] +- [ ] CHK013 Are requirements defined for invalid/expired API keys during task execution? [Edge Case, Gap] \ No newline at end of file diff --git a/specs/017-llm-analysis-plugin/checklists/requirements.md b/specs/017-llm-analysis-plugin/checklists/requirements.md new file mode 100644 index 0000000..776ec4b --- /dev/null +++ b/specs/017-llm-analysis-plugin/checklists/requirements.md @@ -0,0 +1,34 @@ +# Specification Quality Checklist: LLM Analysis & Documentation Plugins + +**Purpose**: Validate specification completeness and quality before proceeding to planning +**Created**: 2026-01-28 +**Feature**: [specs/017-llm-analysis-plugin/spec.md](../spec.md) + +## Content Quality + +- [x] No implementation details (languages, frameworks, APIs) +- [x] Focused on user value and business needs +- [x] Written for non-technical stakeholders +- [x] All mandatory sections completed + +## Requirement Completeness + +- [x] No [NEEDS CLARIFICATION] markers remain +- [x] Requirements are testable and unambiguous +- [x] Success criteria are measurable +- [x] Success criteria are technology-agnostic (no implementation details) +- [x] All acceptance scenarios are defined +- [x] Edge cases are identified +- [x] Scope is clearly bounded +- [x] Dependencies and assumptions identified + +## Feature Readiness + +- [x] All functional requirements have clear acceptance criteria +- [x] User scenarios cover primary flows +- [x] Feature meets measurable outcomes defined in Success Criteria +- [x] No implementation details leak into specification + +## Notes + +- Items marked incomplete require spec updates before `/speckit.clarify` or `/speckit.plan` \ No newline at end of file diff --git a/specs/017-llm-analysis-plugin/checklists/ux-usability.md b/specs/017-llm-analysis-plugin/checklists/ux-usability.md new file mode 100644 index 0000000..87cf575 --- /dev/null +++ b/specs/017-llm-analysis-plugin/checklists/ux-usability.md @@ -0,0 +1,24 @@ +# Checklist: UX & Usability Requirements Quality + +**Purpose**: Validate requirement quality for user interaction and workflows. +**Feature**: `017-llm-analysis-plugin` +**Created**: 2026-01-28 + +## Requirement Completeness +- [ ] CHK001 Are requirements defined for the "Validate" button state during active execution (loading/disabled)? [Completeness, Gap] +- [ ] CHK002 Are feedback requirements defined for successful/failed connection tests in Settings? [Completeness, Gap] +- [ ] CHK003 Are requirements specified for viewing historical validation reports? [Completeness, Spec §US1] +- [ ] CHK004 Are requirements defined for the notification content layout (summary vs. link)? [Completeness, Spec §FR-015] +- [ ] CHK005 Are requirements defined for the "Generate Commit" interaction flow (modal vs. inline)? [Completeness, Spec §US4] + +## Requirement Clarity +- [ ] CHK006 Is "visual representation" defined with resolution or format constraints? [Clarity, Spec §FR-005] +- [ ] CHK007 Are "structured analysis" outputs defined with specific UI presentation requirements? [Clarity, Spec §SC-003] + +## Consistency +- [ ] CHK008 Do the new "Validate" and "Generate Docs" actions follow existing UI patterns (icons, placement)? [Consistency, Gap] +- [ ] CHK009 Are error messages consistent with the application's existing error handling standards? [Consistency, Gap] + +## Edge Case Coverage +- [ ] CHK010 Are requirements defined for when a dashboard cannot be rendered (e.g., 404)? [Edge Case, Gap] +- [ ] CHK011 Are requirements defined for when the generated commit message is empty or invalid? [Edge Case, Gap] \ No newline at end of file diff --git a/specs/017-llm-analysis-plugin/contracts/llm-api.yaml b/specs/017-llm-analysis-plugin/contracts/llm-api.yaml new file mode 100644 index 0000000..bc82cb8 --- /dev/null +++ b/specs/017-llm-analysis-plugin/contracts/llm-api.yaml @@ -0,0 +1,96 @@ +openapi: 3.0.0 +info: + title: LLM Analysis Plugin API + version: 1.0.0 +paths: + /api/settings/llm/providers: + get: + summary: List configured LLM providers + responses: + '200': + description: List of providers + content: + application/json: + schema: + type: array + items: + $ref: '#/components/schemas/LLMProviderConfig' + post: + summary: Create or update a provider + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/LLMProviderConfigInput' + responses: + '200': + description: Created/Updated provider + content: + application/json: + schema: + $ref: '#/components/schemas/LLMProviderConfig' + + /api/git/generate-message: + post: + summary: Generate commit message from diff + requestBody: + required: true + content: + application/json: + schema: + type: object + properties: + diff: + type: string + files: + type: array + items: + type: string + recent_commits: + type: array + items: + type: string + responses: + '200': + description: Generated message + content: + application/json: + schema: + type: object + properties: + message: + type: string + +components: + schemas: + LLMProviderConfig: + type: object + properties: + id: + type: string + format: uuid + provider_type: + type: string + enum: [openai, openrouter, kilo] + name: + type: string + base_url: + type: string + default_model: + type: string + is_active: + type: boolean + LLMProviderConfigInput: + type: object + properties: + provider_type: + type: string + name: + type: string + base_url: + type: string + api_key: + type: string + default_model: + type: string \ No newline at end of file diff --git a/specs/017-llm-analysis-plugin/data-model.md b/specs/017-llm-analysis-plugin/data-model.md new file mode 100644 index 0000000..97229f1 --- /dev/null +++ b/specs/017-llm-analysis-plugin/data-model.md @@ -0,0 +1,51 @@ +# Data Model: LLM Analysis Plugin + +**Feature**: `017-llm-analysis-plugin` + +## Entities + +### LLMProviderConfig +Stores configuration for different LLM providers. + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| id | UUID | Yes | Unique identifier | +| provider_type | Enum | Yes | `openai`, `openrouter`, `kilo` | +| name | String | Yes | Display name (e.g., "Production GPT-4") | +| base_url | String | Yes | API Endpoint URL | +| api_key | String | Yes | Encrypted API Key | +| default_model | String | Yes | Model identifier (e.g., `gpt-4o`) | +| is_active | Boolean | Yes | Whether this provider is currently enabled | + +### ValidationResult +Stores the outcome of a dashboard validation task. + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| id | UUID | Yes | Unique identifier | +| dashboard_id | String | Yes | Reference to the dashboard | +| timestamp | DateTime | Yes | When the validation ran | +| status | Enum | Yes | `PASS`, `WARN`, `FAIL` | +| screenshot_path | String | No | Path to the captured screenshot (if stored) | +| issues | JSON | Yes | List of detected issues `[{severity, message, location}]` | +| raw_response | Text | No | Full LLM response for debugging | + +### TaskConfiguration (Extension) +Extends existing Task model to support validation-specific params. + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| task_type | String | Yes | `dashboard_validation` | +| parameters | JSON | Yes | `{dashboard_id, provider_id, screenshot_strategy}` | + +## API Contracts + +See `contracts/` directory for OpenAPI specifications. + +### Key Interactions + +1. **Configure Provider**: `POST /api/settings/llm/providers` +2. **Trigger Validation**: `POST /api/tasks/dispatch` (payload: `{type: "dashboard_validation", ...}`) +3. **Get Results**: `GET /api/tasks/{task_id}/result` (or via WebSocket stream) +4. **Generate Documentation**: `POST /api/tasks/dispatch` (payload: `{type: "dataset_documentation", ...}`) +5. **Generate Commit**: `POST /api/git/generate-message` \ No newline at end of file diff --git a/specs/017-llm-analysis-plugin/plan.md b/specs/017-llm-analysis-plugin/plan.md new file mode 100644 index 0000000..bbdbb65 --- /dev/null +++ b/specs/017-llm-analysis-plugin/plan.md @@ -0,0 +1,98 @@ +# Implementation Plan: LLM Analysis & Documentation Plugins + +**Branch**: `017-llm-analysis-plugin` | **Date**: 2026-01-28 | **Spec**: [specs/017-llm-analysis-plugin/spec.md](spec.md) +**Input**: Feature specification from `/specs/017-llm-analysis-plugin/spec.md` + +**Note**: This template is filled in by the `/speckit.plan` command. See `.specify/templates/commands/plan.md` for the execution workflow. + +## Summary + +This feature implements two new plugins for the `ss-tools` platform: `DashboardValidationPlugin` and `DocumentationPlugin`. The `DashboardValidationPlugin` uses multimodal LLMs to analyze dashboard health by combining screenshots (via headless browser or API) and execution logs. The `DocumentationPlugin` automates the generation of metadata descriptions for datasets and columns using LLMs. Additionally, the feature integrates LLM capabilities into the Git plugin for commit message generation and adds a configuration interface for managing multiple LLM providers (OpenAI, OpenRouter, Kilo). + +## Technical Context + +**Language/Version**: Python 3.9+ (Backend), Node.js 18+ (Frontend) +**Primary Dependencies**: +- Backend: `openai` (or compatible SDK), `playwright` (for headless screenshots), `fastapi` (existing), `pydantic` (existing), `sqlalchemy` (existing) +- Frontend: `svelte` (existing), `tailwindcss` (existing) +**Storage**: SQLite (existing `tasks.db` for results, `auth.db` for permissions, `mappings.db` or new `plugins.db` for provider config/metadata) +**Testing**: `pytest` (Backend), `vitest` (Frontend) +**Target Platform**: Linux server (existing environment) +**Project Type**: Web application (FastAPI + SvelteKit) +**Performance Goals**: Dashboard validation < 90s, Documentation generation < 30s per asset +**Constraints**: Must integrate with existing `PluginBase` and `TaskManager`. Secure storage for API keys. +**Scale/Scope**: Support for configurable LLM providers, scheduled tasks, and notification integration. + +## Constitution Check + +*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.* + +- [x] **I. Semantic Protocol Compliance**: Will follow `[DEF]`, `@RELATION`, and file structure standards. +- [x] **II. Causal Validity**: Contracts/Interfaces will be defined before implementation. +- [x] **III. Immutability of Architecture**: No changes to core immutable layers planned; extending via Plugin architecture. +- [x] **IV. Design by Contract**: All new plugins and services will use DbC. +- [x] **V. Belief State Logging**: Will use standard logging for belief state. +- [x] **VI. Fractal Complexity Limit**: Plugins will be modular and adhere to complexity limits. +- [x] **VII. Everything is a Plugin**: Implemented as `DashboardValidationPlugin` and `DocumentationPlugin`. +- [x] **VIII. Unified Frontend Experience**: Will use existing Svelte components and i18n. +- [x] **IX. Security & Access Control**: Will define permissions (e.g., `plugin:llm:validate`, `plugin:llm:configure`). + +## Project Structure + +### Documentation (this feature) + +```text +specs/017-llm-analysis-plugin/ +├── plan.md # This file (/speckit.plan command output) +├── research.md # Phase 0 output (/speckit.plan command) +├── data-model.md # Phase 1 output (/speckit.plan command) +├── quickstart.md # Phase 1 output (/speckit.plan command) +├── contracts/ # Phase 1 output (/speckit.plan command) +└── tasks.md # Phase 2 output (/speckit.tasks command - NOT created by /speckit.plan) +``` + +### Source Code (repository root) + +```text +backend/ +├── src/ +│ ├── plugins/ +│ │ ├── llm_analysis/ # New Plugin Directory +│ │ │ ├── __init__.py +│ │ │ ├── plugin.py # Implements DashboardValidationPlugin & DocumentationPlugin +│ │ │ ├── service.py # LLM interaction logic +│ │ │ └── models.py # Pydantic models for LLM config/results +│ │ └── git/ # Existing Git Plugin +│ │ └── ... # Update for commit message generation +│ ├── api/ +│ │ └── routes/ +│ │ └── llm.py # New routes for LLM config/proxy if needed +│ └── services/ +│ └── llm_provider.py # Shared service for LLM provider management +└── tests/ + └── plugins/ + └── test_llm_analysis.py + +frontend/ +├── src/ +│ ├── components/ +│ │ ├── llm/ +│ │ │ ├── ProviderConfig.svelte # Configuration UI +│ │ │ └── ValidationReport.svelte # Result viewer +│ │ └── git/ +│ │ └── CommitMessageGen.svelte # UI component for commit generation +│ └── routes/ +│ └── admin/ +│ └── settings/ +│ └── llm/ # Settings page route +``` + +**Structure Decision**: Web application structure with backend plugins and frontend components, consistent with existing project layout. + +## Complexity Tracking + +> **Fill ONLY if Constitution Check has violations that must be justified** + +| Violation | Why Needed | Simpler Alternative Rejected Because | +|-----------|------------|-------------------------------------| +| N/A | | | diff --git a/specs/017-llm-analysis-plugin/quickstart.md b/specs/017-llm-analysis-plugin/quickstart.md new file mode 100644 index 0000000..74d1859 --- /dev/null +++ b/specs/017-llm-analysis-plugin/quickstart.md @@ -0,0 +1,42 @@ +# Quickstart: LLM Analysis Plugin + +## Prerequisites + +1. An API Key from OpenAI, OpenRouter, or Kilo. +2. (Optional) Playwright browsers installed for high-fidelity screenshots: + ```bash + playwright install chromium + ``` + +## Configuration + +1. Navigate to **Settings > LLM Providers**. +2. Click **"Add Provider"**. +3. Select type (e.g., `OpenRouter`). +4. Enter your API Key and Base URL (e.g., `https://openrouter.ai/api/v1`). +5. Set a default model (e.g., `anthropic/claude-3.5-sonnet`). +6. Click **Save**. + +## Usage + +### Dashboard Validation + +1. Go to **Environments**. +2. Select a dashboard from the list. +3. Click **"Validate"**. +4. Wait for the task to complete and view the report in the **Task History** panel. + +### Dataset Documentation + +1. Go to **Tools > Documentation**. +2. Select a dataset. +3. Click **"Generate Docs"**. +4. Review the generated descriptions and changes will be applied to the metadata. + +### Git Commit Generation + +1. Make changes to files in the repository. +2. Go to the **Git** plugin tab. +3. Stage your files. +4. Click the **"✨" (Magic)** button next to the commit message box. +5. Review and edit the suggested message before committing. \ No newline at end of file diff --git a/specs/017-llm-analysis-plugin/research.md b/specs/017-llm-analysis-plugin/research.md new file mode 100644 index 0000000..e19d120 --- /dev/null +++ b/specs/017-llm-analysis-plugin/research.md @@ -0,0 +1,48 @@ +# Research: LLM Analysis & Documentation Plugins + +**Feature**: `017-llm-analysis-plugin` + +## 1. LLM Provider Integration + +**Decision**: Use a unified `LLMProviderService` that abstracts OpenAI-compatible APIs. +**Rationale**: OpenRouter, Kilo, and OpenAI all support the standard OpenAI API format. This simplifies implementation by using the `openai` Python SDK and changing the `base_url` and `api_key` dynamically based on configuration. +**Alternatives Considered**: LangChain (too heavy/complex for this specific scope), custom HTTP requests (reinventing the wheel). + +## 2. Dashboard Screenshot Capture + +**Decision**: Implement a `ScreenshotService` with a strategy pattern supporting `Playwright` (primary) and `Superset API` (fallback). +**Rationale**: +- **Playwright**: Provides the most accurate "user-view" render, handling JS-heavy charts that API thumbnails might miss or render poorly. Requires a browser binary. +- **Superset API**: Faster, lightweight, but relies on Superset's internal thumbnail cache which can be stale. +**Implementation Detail**: The service will check configuration. If 'Headless' is selected, it launches a Playwright context, logs in (using a service account or session cookie), navigates to the dashboard, waits for network idle, and captures. + +## 3. Multimodal Analysis Prompting + +**Decision**: Use a structured prompt template that accepts base64-encoded images and text logs. +**Rationale**: Models like GPT-4o and Claude 3.5 Sonnet (via OpenRouter) support this natively. +**Prompt Structure**: +- System: "You are a Data Observability Expert..." +- User Image: [Base64 Screenshot] +- User Text: "Recent Logs: \n[Log Snippets]..." +- Output Format: JSON (Status, Issues[], Recommendations[]) + +## 4. Documentation Persistence + +**Decision**: Update `Dataset` and `Column` models in the existing metadata database (likely `mappings.db` or the main application DB if integrated). +**Rationale**: Keeps documentation co-located with the assets. +**Mechanism**: The `DocumentationPlugin` will fetch schema, generate markdown, and execute an `UPDATE` operation on the relevant tables/fields. + +## 5. Git Commit Integration + +**Decision**: Add a REST endpoint `/api/git/generate-message` used by the frontend Git component. +**Rationale**: Keeps the heavy lifting (LLM call, diff processing) on the backend. The frontend simply sends the list of staged files and a diff summary (truncated if necessary). + +## 6. Security & Storage + +**Decision**: Encrypt API keys at rest using the existing `Fernet` or similar encryption mechanism used for database credentials. +**Rationale**: API keys are sensitive. They should not be stored in plain text in `config.json` or the DB. + +## 7. Retry Logic + +**Decision**: Use `tenacity` library for decorators on LLM service methods. +**Rationale**: Standard, robust, declarative retry logic (exponential backoff) as required by FR-018. \ No newline at end of file diff --git a/specs/017-llm-analysis-plugin/spec.md b/specs/017-llm-analysis-plugin/spec.md new file mode 100644 index 0000000..ef0a88c --- /dev/null +++ b/specs/017-llm-analysis-plugin/spec.md @@ -0,0 +1,122 @@ +# Feature Specification: LLM Analysis & Documentation Plugins + +**Feature Branch**: `017-llm-analysis-plugin` +**Created**: 2026-01-28 +**Status**: Draft +**Input**: User description: "LLM Dashboard Validation Plugin для интеграции LLM в ss-tools. Плагин должен поддерживать анализ корректности работы дашбордов через мультимодальную LLM (скриншот + логи). Второй плагин должен заниматься документированием датасетов и дашбордов, используя LLM. Возможно включение в создание текста коммитов в плагине Git Поддержка провайдеров: OpenRouter, Kilo Provider, OpenAI API. Интеграция с существующей PluginBase архитектурой, Task Manager, WebSocket логами." + +## Clarifications + +### Session 2026-01-28 +- Q: Notification Content Strategy → A: **Summary with Link**: Include status (Pass/Fail), key issues count, and a direct link to the full report in the UI. +- Q: Dashboard Screenshot Source → A: **Hybrid (Configurable)**: Support both Headless Browser (accurate) and API/Thumbnail (fast) methods, allowing admin configuration. +- Q: Dataset Documentation Output Format → A: **Direct Object Update**: Update the description fields of the dataset and its columns directly within the dataset object (persisted to backend/metadata). +- Q: Git Commit Message Context → A: **Diff + Recent History**: Send diff, file names, and the last 3 commit messages to match style. +- Q: LLM Failure Handling → A: **Retry then Fail**: Automatically retry 3 times with exponential backoff before failing. + +## User Scenarios & Testing *(mandatory)* + +### User Story 1 - Dashboard Health Analysis (Priority: P1) + +As a Data Engineer, I want to automatically analyze a dashboard's status using visual and log data directly from the Environments interface so that I can identify rendering issues or data errors without manual inspection. + +**Why this priority**: Core value proposition of the feature. Enables automated quality assurance. + +**Independent Test**: Can be tested by selecting a dashboard in the Environment list and clicking "Validate", or by scheduling a validation task. + +**Acceptance Scenarios**: + +1. **Given** I am on the Environments page (Dashboard list), **When** I select a dashboard and click "Validate", **Then** the system triggers a validation task with the dashboard's context. +2. **Given** the validation task is running, **When** it completes, **Then** I see the analysis report (visual + logs) in the task output history. +3. **Given** I want regular checks, **When** I configure a schedule for the validation task (similar to Backup plugin), **Then** the system runs the check automatically at the specified interval. +4. **Given** a validation issue is found, **When** the task completes, **Then** the system sends a notification (Email/Pulse) if configured. + +--- + +### User Story 2 - Automated Dataset Documentation (Priority: P1) + +As a Data Steward, I want to generate documentation for datasets and dashboards using LLMs so that I can maintain up-to-date metadata with minimal manual effort. + +**Why this priority**: significantly reduces maintenance overhead for data governance. + +**Independent Test**: Can be tested by selecting a dataset and triggering the documentation task, then checking if a description is generated. + +**Acceptance Scenarios**: + +1. **Given** a dataset identifier, **When** I run the Documentation task, **Then** the system fetches the dataset's schema and metadata. +2. **Given** the metadata is fetched, **When** sent to the LLM, **Then** a structured description/documentation text is returned. +3. **Given** the documentation is generated, **When** the task completes, **Then** the result is available for review (e.g., in the task log or saved to a file/db). + +--- + +### User Story 3 - LLM Provider Configuration (Priority: P1) + +As an Administrator, I want to configure different LLM providers (OpenAI, OpenRouter, Kilo) so that I can switch between models based on cost or capability. + +**Why this priority**: Prerequisite for any LLM functionality. + +**Independent Test**: Can be tested by entering API keys in settings and verifying a connection/test call. + +**Acceptance Scenarios**: + +1. **Given** the Settings page, **When** I select an LLM provider (e.g., OpenAI) and enter an API key, **Then** the system saves the configuration. +2. **Given** a configured provider, **When** I run an analysis task, **Then** the system uses the selected provider for API calls. + +--- + +### User Story 4 - Git Commit Message Suggestion (Priority: P3) + +As a Developer, I want the system to suggest commit messages based on changes directly within the Git plugin interface so that I can maintain consistent history with minimal effort. + +**Why this priority**: Enhances the existing Git workflow and improves commit quality. + +**Independent Test**: Can be tested by staging files in the Git plugin and clicking the "Generate Message" button. + +**Acceptance Scenarios**: + +1. **Given** staged changes in the Git plugin, **When** I click "Generate Message", **Then** the system analyzes the diff using the configured LLM and populates the commit message field with a suggested summary. + +### Edge Cases + +- What happens when the LLM provider API is down or times out? (System should retry or fail gracefully with a clear error message). +- What happens if the dashboard screenshot cannot be generated? (System should proceed with logs only or fail depending on configuration). +- What happens if the context (logs/metadata) exceeds the LLM's token limit? (System should truncate or summarize input). +- How does the system handle missing API keys? (Task should fail immediately with a configuration error). + +## Requirements *(mandatory)* + +### Functional Requirements + +- **FR-001**: System MUST allow configuration of multiple LLM providers, specifically supporting OpenAI API, OpenRouter, and Kilo Provider. +- **FR-002**: System MUST securely store API keys for these providers. +- **FR-003**: System MUST implement a `DashboardValidationPlugin` that integrates with the existing `PluginBase` architecture. +- **FR-004**: `DashboardValidationPlugin` MUST accept a dashboard identifier as input. +- **FR-005**: `DashboardValidationPlugin` MUST be capable of retrieving a visual representation (screenshot) of the dashboard. +- **FR-016**: System MUST support configurable screenshot strategies: 'Headless Browser' (default, high accuracy) and 'API Thumbnail' (fallback/fast). +- **FR-006**: `DashboardValidationPlugin` MUST retrieve recent execution logs associated with the dashboard. +- **FR-007**: `DashboardValidationPlugin` MUST combine visual and text data to prompt a Multimodal LLM for analysis. +- **FR-008**: System MUST implement a `DocumentationPlugin` (or similar) for documenting datasets and dashboards. +- **FR-009**: `DocumentationPlugin` MUST retrieve schema and metadata for the target asset. +- **FR-017**: `DocumentationPlugin` MUST apply generated descriptions directly to the target object's metadata fields (dataset description, column descriptions). +- **FR-010**: All LLM interactions MUST be executed as asynchronous tasks via the Task Manager. +- **FR-018**: System MUST implement automatic retry logic (3 attempts with exponential backoff) for failed LLM API calls. +- **FR-011**: Task execution logs and results MUST be streamed via the existing WebSocket infrastructure. +- **FR-012**: System SHOULD expose an interface to generate text summaries for Git diffs, utilizing the diff, file list, and recent commit history as context. +- **FR-013**: System MUST support scheduling of validation tasks for dashboards (leveraging existing scheduler architecture). +- **FR-014**: System SHOULD support notification dispatch (Email, Pulse) upon validation failure or completion. +- **FR-015**: Notifications MUST contain a summary of results (Status, Issue Count) and a direct link to the full report, avoiding sensitive full details in the message body. + +### Key Entities + +- **LLMProviderConfig**: Stores provider type (OpenAI, etc.), base URL, model name, and API key. +- **ValidationResult**: Stores the analysis output, timestamp, and reference to the dashboard. +- **AutoDocumentation**: Stores the generated documentation text for an asset. + +## Success Criteria *(mandatory)* + +### Measurable Outcomes + +- **SC-001**: Users can successfully configure and validate a connection to at least one LLM provider. +- **SC-002**: A dashboard validation task completes within 90 seconds (assuming standard LLM latency). +- **SC-003**: The system successfully processes a multimodal prompt (image + text) and returns a structured analysis. +- **SC-004**: Generated documentation for a standard dataset contains descriptions for at least 80% of the columns (based on LLM capability, but pipeline must support it). \ No newline at end of file diff --git a/specs/017-llm-analysis-plugin/tasks.md b/specs/017-llm-analysis-plugin/tasks.md new file mode 100644 index 0000000..41b0a7d --- /dev/null +++ b/specs/017-llm-analysis-plugin/tasks.md @@ -0,0 +1,103 @@ +# Tasks: LLM Analysis & Documentation Plugins + +**Feature**: `017-llm-analysis-plugin` +**Status**: Pending +**Spec**: [specs/017-llm-analysis-plugin/spec.md](spec.md) + +## Dependencies + +- **Phase 1 (Setup)**: Must run first. +- **Phase 2 (Foundational)**: Must complete before any User Stories. +- **Phase 3 (US1 - Dashboard Analysis)**: Depends on Phase 2. +- **Phase 4 (US2 - Documentation)**: Depends on Phase 2. Parallel with Phase 3. +- **Phase 5 (US3 - Configuration)**: Depends on Phase 2. Recommended before Phase 3/4 for easier testing, but technically parallelizable if config is mocked. +- **Phase 6 (US4 - Git Integration)**: Independent of other stories, depends on Phase 2. + +## Phase 1: Setup + +**Goal**: Initialize project structure and dependencies. + +- [ ] T001 Install backend dependencies (openai, playwright, tenacity) in `backend/requirements.txt` +- [ ] T002 Create plugin directory structure `backend/src/plugins/llm_analysis/` +- [ ] T003 Initialize `backend/src/plugins/llm_analysis/__init__.py` +- [ ] T004 Create `backend/src/plugins/llm_analysis/models.py` for Pydantic models (LLMProviderConfig, ValidationResult) +- [ ] T005 Update `backend/src/core/plugin_loader.py` to recognize new plugin type if necessary +- [ ] T006 Create `backend/src/api/routes/llm.py` placeholder +- [ ] T007 Register new route in `backend/src/app.py` + +## Phase 2: Foundational + +**Goal**: Implement core services and shared infrastructure. + +- [ ] T008 Implement `LLMProviderService` in `backend/src/services/llm_provider.py` (CRUD for providers, encryption) +- [ ] T009 Implement `ScreenshotService` in `backend/src/plugins/llm_analysis/service.py` (Playwright + API strategies) +- [ ] T010 Implement `LLMClient` in `backend/src/plugins/llm_analysis/service.py` (OpenAI SDK wrapper, retry logic with tenacity) +- [ ] T011 Create `backend/src/plugins/llm_analysis/plugin.py` with `PluginBase` implementation stubs +- [ ] T012 Define database schema updates for `LLMProviderConfig` in `backend/src/models/llm.py` (or appropriate location) +- [ ] T013 Run migration to create new tables (if using SQLAlchemy/Alembic) or update SQLite schema + +## Phase 3: Dashboard Health Analysis (US1) + +**Goal**: Enable automated dashboard validation with multimodal analysis. + +- [ ] T014 [US1] Implement `validate_dashboard` task logic in `backend/src/plugins/llm_analysis/plugin.py` +- [ ] T015 [US1] Implement log retrieval logic (fetch recent logs) in `backend/src/plugins/llm_analysis/plugin.py` +- [ ] T016 [US1] Construct multimodal prompt (image + text) in `backend/src/plugins/llm_analysis/service.py` +- [ ] T017 [US1] Implement result parsing and persistence (ValidationResult) in `backend/src/plugins/llm_analysis/plugin.py` +- [ ] T018 [US1] Add `validate` endpoint trigger in `backend/src/api/routes/tasks.py` (or reuse existing dispatch) +- [ ] T019 [US1] Implement notification dispatch (Email/Pulse) on failure in `backend/src/plugins/llm_analysis/plugin.py` +- [ ] T020 [US1] Create `frontend/src/components/llm/ValidationReport.svelte` for viewing results +- [ ] T021 [US1] Add "Validate" button to `frontend/src/components/DashboardGrid.svelte` (or Environments list) +- [ ] T022 [US1] Enable scheduling for validation tasks in `backend/src/core/scheduler.py` (if not automatic via TaskManager) + +## Phase 4: Automated Dataset Documentation (US2) + +**Goal**: Generate and persist dataset documentation. + +- [ ] T023 [US2] Implement `generate_documentation` task logic in `backend/src/plugins/llm_analysis/plugin.py` +- [ ] T024 [US2] Implement metadata fetching (schema, columns) in `backend/src/plugins/llm_analysis/plugin.py` +- [ ] T025 [US2] Construct documentation prompt in `backend/src/plugins/llm_analysis/service.py` +- [ ] T026 [US2] Implement metadata update logic (write back to DB) in `backend/src/services/mapping_service.py` (or relevant service) +- [ ] T027 [US2] Add "Generate Docs" button to `frontend/src/components/tools/MapperTool.svelte` (or Dataset view) +- [ ] T028 [US2] Create feedback/preview UI component `frontend/src/components/llm/DocPreview.svelte` (optional but recommended) + +## Phase 5: LLM Provider Configuration (US3) + +**Goal**: Manage LLM providers via UI. + +- [ ] T029 [US3] Implement CRUD endpoints for providers in `backend/src/api/routes/llm.py` +- [ ] T030 [US3] Create `frontend/src/components/llm/ProviderConfig.svelte` form +- [ ] T031 [US3] Create `frontend/src/routes/admin/settings/llm/+page.svelte` settings page +- [ ] T032 [US3] Implement "Test Connection" button logic in frontend/backend +- [ ] T033 [US3] Ensure API keys are masked in frontend responses + +## Phase 6: Git Commit Message Suggestion (US4) + +**Goal**: AI-assisted commit messages. + +- [ ] T034 [US4] Implement `generate_commit_message` logic in `backend/src/plugins/git_plugin.py` (or `llm_analysis` if preferred, but Git context lives in Git plugin) +- [ ] T035 [US4] Create endpoint `POST /api/git/generate-message` in `backend/src/api/routes/git.py` +- [ ] T036 [US4] Construct commit generation prompt (Diff + History) in `backend/src/services/git_service.py` +- [ ] T037 [US4] Add "Generate" button to `frontend/src/components/git/CommitModal.svelte` +- [ ] T038 [US4] Wire up frontend to call generation endpoint and populate textarea + +## Phase 7: Polish & Cross-Cutting + +**Goal**: Finalize and verify. + +- [ ] T039 Verify all permissions (`plugin:llm:validate`, etc.) are registered and enforceable +- [ ] T040 Verify i18n strings for all new UI components +- [ ] T041 Run full end-to-end test of Dashboard Validation flow +- [ ] T042 Run full end-to-end test of Documentation flow +- [ ] T043 Update `README.md` with new plugin capabilities + +## Implementation Strategy + +1. **MVP**: Focus on Phase 1, 2, 3, and 5. This delivers the core value (Dashboard Validation) and the necessary configuration. +2. **Increment 1**: Add Phase 4 (Documentation). +3. **Increment 2**: Add Phase 6 (Git Integration). + +## Parallel Execution Examples + +- **Backend/Frontend Split**: T020 (Frontend Report) can be built while T014-T017 (Backend Logic) are in progress, using mock data. +- **Story Parallelism**: US2 (Documentation) and US4 (Git) are largely independent and can be worked on simultaneously by different developers (or sequentially without blocking). \ No newline at end of file