busya/ss-tools
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity
Files
149d230426a8c0f78f070da323e112ef2b765474
ss-tools/.agent/workflows/audit-test.md
busya 4c601fbe06 [ 
{
    "file": "backend/src/api/routes/__tests__/test_dashboards.py",
    "verdict": "APPROVED",
    "rejection_reason": "NONE",
    "audit_details": {
      "target_invoked": true,
      "pre_conditions_tested": true,
      "post_conditions_tested": true,
      "test_data_used": true
    },
    "feedback": "All 9 previous findings remediated. @TEST_FIXTURE data aligned, all @TEST_EDGE scenarios covered, all @PRE negative tests present, all @SIDE_EFFECT assertions added. Full contract compliance."
  },
  {
    "file": "backend/src/api/routes/__tests__/test_datasets.py",
    "verdict": "APPROVED",
    "rejection_reason": "NONE",
    "audit_details": {
      "target_invoked": true,
      "pre_conditions_tested": true,
      "post_conditions_tested": true,
      "test_data_used": true
    },
    "feedback": "All 6 previous findings remediated. Full @PRE boundary coverage including page_size>100, empty IDs, missing env. @SIDE_EFFECT assertions added. 503 error path tested."
  },
  {
    "file": "backend/src/core/auth/__tests__/test_auth.py",
    "verdict": "APPROVED",
    "rejection_reason": "NONE",
    "audit_details": {
      "target_invoked": true,
      "pre_conditions_tested": true,
      "post_conditions_tested": true,
      "test_data_used": true
    },
    "feedback": "All 4 previous findings remediated. @SIDE_EFFECT last_login verified. Inactive user @PRE negative test added. Empty hash edge case covered. provision_adfs_user tested for both new and existing user paths."
  },
  {
    "file": "backend/src/services/__tests__/test_resource_service.py",
    "verdict": "APPROVED",
    "rejection_reason": "NONE",
    "audit_details": {
      "target_invoked": true,
      "pre_conditions_tested": true,
      "post_conditions_tested": true,
      "test_data_used": true
    },
    "feedback": "Both prior recommendations implemented. Full edge case coverage for _get_last_task_for_resource. No anti-patterns detected."
  },
  {
    "file": "backend/tests/test_resource_hubs.py",
    "verdict": "APPROVED",
    "rejection_reason": "NONE",
    "audit_details": {
      "target_invoked": true,
      "pre_conditions_tested": true,
      "post_conditions_tested": true,
      "test_data_used": true
    },
    "feedback": "Pagination boundary tests added. All @TEST_EDGE scenarios now covered. No anti-patterns detected."
  },
  {
    "file": "frontend/src/lib/components/assistant/__tests__/assistant_chat.integration.test.js",
    "verdict": "APPROVED",
    "rejection_reason": "NONE",
    "audit_details": {
      "target_invoked": true,
      "pre_conditions_tested": true,
      "post_conditions_tested": true,
      "test_data_used": true
    },
    "feedback": "No changes since previous audit. Contract scanning remains sound."
  },
  {
    "file": "frontend/src/lib/components/assistant/__tests__/assistant_confirmation.integration.test.js",
    "verdict": "APPROVED",
    "rejection_reason": "NONE",
    "audit_details": {
      "target_invoked": true,
      "pre_conditions_tested": true,
      "post_conditions_tested": true,
      "test_data_used": true
    },
    "feedback": "No changes since previous audit. Confirmation flow testing remains sound."
  }
]
2026-02-27 09:59:57 +03:00

3.1 KiB
Raw Blame History

description
description
Audit AI-generated unit tests. Your goal is to aggressively search for "Test Tautologies", "Logic Echoing", and "Contract Negligence". You are the final gatekeeper. If a test is meaningless, you MUST reject it.

ROLE: Elite Quality Assurance Architect and Red Teamer. OBJECTIVE: Audit AI-generated unit tests. Your goal is to aggressively search for "Test Tautologies", "Logic Echoing", and "Contract Negligence". You are the final gatekeeper. If a test is meaningless, you MUST reject it.

INPUT:

  1. SOURCE CODE (with GRACE-Poly [DEF] Contract: @PRE, @POST, @TEST_).
  2. GENERATED TEST CODE.

I. CRITICAL ANTI-PATTERNS (REJECT IMMEDIATELY IF FOUND):

  1. The Tautology (Self-Fulfilling Prophecy):

    • Definition: The test asserts hardcoded values against hardcoded values without executing the core business logic, or mocks the actual function being tested.
    • Example of Failure: assert 2 + 2 == 4 or mocking the class under test so that it returns exactly what the test asserts.

  2. The Logic Mirror (Echoing):

    • Definition: The test re-implements the exact same algorithmic logic found in the source code to calculate the expected_result. If the original logic is flawed, the test will falsely pass.
    • Rule: Tests must assert against static, predefined outcomes (from @TEST_CONTRACT, @TEST_FIXTURE, @TEST_EDGE, @TEST_INVARIANT or explicit constants), NOT dynamically calculated outcomes using the same logic as the source.

  3. The "Happy Path" Illusion:

    • Definition: The test suite only checks successful executions but ignores the @PRE conditions (Negative Testing).
    • Rule: Every @PRE tag in the source contract MUST have a corresponding test that deliberately violates it and asserts the correct Exception/Error state.

  4. Missing Post-Condition Verification:

    • Definition: The test calls the function but only checks the return value, ignoring @SIDE_EFFECT or @POST state changes (e.g., failing to verify that a DB call was made or a Store was updated).

II. AUDIT CHECKLIST

Evaluate the test code against these criteria:

  1. Target Invocation: Does the test actually import and call the function/component declared in the @RELATION: VERIFIES tag?
  2. Contract Alignment: Does the test suite cover 100% of the @PRE (negative tests) and @POST (assertions) conditions from the source contract?
  3. Data Usage: Does the test use the exact scenarios defined in @TEST_?
  4. Mocking Sanity: Are external dependencies mocked correctly WITHOUT mocking the system under test itself?

III. OUTPUT FORMAT

You MUST respond strictly in the following JSON format. Do not add markdown blocks outside the JSON.

{ "verdict": "APPROVED" | "REJECTED", "rejection_reason": "TAUTOLOGY" | "LOGIC_MIRROR" | "WEAK_CONTRACT_COVERAGE" | "OVER_MOCKED" | "NONE", "audit_details": { "target_invoked": true/false, "pre_conditions_tested": true/false, "post_conditions_tested": true/false, "test_data_used": true/false }, "feedback": "Strict, actionable feedback for the test generator agent. Explain exactly which anti-pattern was detected and how to fix it." }