Implementation Plan: ID Synchronization and Cross-Filter Recovery

Branch: 022-sync-id-cross-filters | Date: 2026-02-25 | Spec: spec.md Input: Feature specification from /specs/022-sync-id-cross-filters/spec.md

Summary

This feature implements a mechanism to recover native cross-filtering links when migrating Apache Superset dashboards. It introduces an IdMappingService to track Superset's unstable Integer IDs across environments using stable UUIDs. During migration, a new fix_cross_filters option enables a workflow that intercepts the export ZIP, dynamically translates Source environment Integer IDs to Target environment Integer IDs in the Dashboard's json_metadata, and performs a dual-import sequence if the Chart objects don't yet exist on the target.

Additionally, it provides a Settings UI allowing administrators to configure the background synchronization interval via a Cron string and to inspect the internal mapping catalog, showing real names (resource_name) alongside their UUIDs and IDs across different environments.

Technical Context

Language/Version: Python 3.11+ Primary Dependencies: FastAPI, Pydantic, APScheduler, SQLAlchemy (or existing SQLite interface) Storage: Local mappings.db (SQLite) Testing: pytest Target Platform: Linux / Docker Project Type: Backend Web Service (ss-tools API) Performance Goals: ID translations must complete in < 5 seconds. Background sync must paginate smoothly to avoid overwhelming Target Superset APIs. Constraints: Requires admin API access to Superset instances. Scale/Scope: Supports hundreds of dashboards and charts per environment.

Constitution Check

GATE: Passed

I. Library-First: The IdMappingService will be strictly isolated as a core concept within backend/src/core/mapping_service (or similar). It does NOT depend on web-layer code.
II. CLI Interface: N/A for this internal daemon, though its behavior is exposed over the API.
III. Test-First (NON-NEGOTIABLE): We will write targeted pytest unit tests for the ZIP manifest patching logic within MigrationEngine using the fixtures provided in the spec.
IV. Integration Testing: A mock Superset API environment will be required to integration test the "Dual-Import" capability safely.

Project Structure

Documentation (this feature)

specs/022-sync-id-cross-filters/
├── plan.md              # This file
├── research.md          # Phase 0 output
├── data-model.md        # Phase 1 output
├── quickstart.md        # Phase 1 output
├── contracts/           # Phase 1 output
└── tasks.md             # Phase 2 output (Pending)

Source Code

backend/
├── src/
│   ├── core/
│   │   ├── migration_engine.py  # (MODIFIED) Add YAML patch layer
│   │   └── mapping_service.py   # (NEW) IdMappingService + scheduling
│   ├── models/
│   │   └── mappings.py          # (NEW) SQLAlchemy/Pydantic models for ResourceMapping
│   └── api/
│       └── routes/
│           ├── migration.py     # (MODIFIED) Add fix_cross_filters config flag
│           └── settings.py      # (MODIFIED/NEW) Add cron config and mapping GET endpoints
└── tests/
    └── core/
        ├── test_migration_engine.py # (NEW/MODIFIED)
        └── test_mapping_service.py  # (NEW)

frontend/
├── src/
│   ├── routes/
│   │   └── settings/            # (MODIFIED) Add UI for cron and mapping table
│   └── components/
│       └── mappings/
│           └── MappingTable.svelte # (NEW) Table to view current UUID <-> ID catalogs

Structure Decision: Extending the existing Backend FastApi structure. The logic touches API routing, the core MigrationEngine, and introduces a new standalone mapping_service.py to handle the scheduler and DB layer.

Complexity Tracking

No major constitutional violations. The only added complexity is the "Dual-Import" mechanism in MigrationEngine, which is strictly necessary because Superset assigns IDs unconditionally on creation.

Test Data Reference

Component	TIER	Fixture Name	Location
`IdMappingService`	CRITICAL	`resource_mapping_record`	spec.md#test-data-fixtures
`MigrationEngine`	CRITICAL	`missing_target_scenario`	spec.md#test-data-fixtures

4.4 KiB Raw Blame History