Skip to content

Import Framework

Audience: developers extending the import system — adding a new preset for an existing domain, adding a new domain entirely, or modifying framework behaviour.

This document covers the generic import framework (packages/app/src/components/import/*, packages/app/src/lib/import/typing/*, packages/app/src/services/import/*) and explains where the payments domain layers on top of it. The split is enforced by code organisation, not just convention: framework code never imports from a domain layer.


At a glance

The framework is a polymorphic wizard that takes a CSV file and a per-domain configuration object (ImportFlowConfig<TExtras, TPreset>) and produces a typed ImportRow[] plus a per-domain context payload. The wizard hands them to a domain-owned TS commit worker (Server Action under services/<domain>/) which calls the existing per-table service helpers, writes the run + per-row outcomes to import_runs, and returns a summary the wizard displays.

Today Payments is the only domain. The architecture is designed so a second domain (Budgets, Transactions, …) ships entirely in new directories with zero changes to framework code or the shared DB tables — only its own commit worker.

Files & directories

packages/app/src/
├── constants/import.ts              FRAMEWORK
├── constants/payment.ts             PAYMENTS DOMAIN
├── types/import.ts                  FRAMEWORK
├── types/payment.ts                 PAYMENTS DOMAIN
├── schemas/importMapping.schema.ts  BOTH (framework schemas + paymentsCreate refinement)
│
├── lib/import/
│   ├── typing/index.ts              FRAMEWORK: coerceValueByKind, getCellsForDestination
│   ├── presets/
│   │   ├── generic.ts               FRAMEWORK: universal `generic` preset
│   │   └── index.ts
│   └── payments/                    PAYMENTS DOMAIN
│       ├── flowConfig.tsx
│       ├── domainConfig.ts
│       ├── presets/                 bankGeneric, cardEquals, cardRevolut
│       ├── review/classifyReviewRow.ts
│       └── typing/                  classifyPresetFilter, coercion, applyPaymentsMapping
│
├── components/import/               FRAMEWORK
│   ├── ImportWizard/
│   ├── ImportStepUpload/
│   ├── ImportStepProvider/
│   ├── ImportStepMapping/
│   ├── ImportSaveCustomDialog/
│   ├── ImportLogo/
│   └── ImportExampleDownloadButton/
│
├── components/payments/import/      PAYMENTS DOMAIN
│   ├── ImportStepType/              TYPE step body (bank vs card)
│   ├── ImportStepPaymentMethod/     TARGET step body
│   ├── ImportStepResolve/           RESOLVE step (per-card decisions)
│   ├── ImportStepReview/            REVIEW step (per-row classifier display)
│   ├── ImportCreatePaymentMethodDialog/
│   └── PaymentsImportTargetSlot/
│
├── services/import/                 FRAMEWORK
│   ├── base.ts                      CRUD + getBankMappingForPaymentMethod (payments-tinted)
│   ├── server.ts
│   └── actions.ts
│
└── hooks/queries/useImportQuery.ts  FRAMEWORK

Database

public.import_mappings    framework — domain-polymorphic, opaque `domain_config` JSONB
public.import_fields      framework — destination-field catalog per entity_type
public.import_runs        framework — per-import audit log (written by each domain's TS commit worker)

See docs/architecture/database/FINANCIAL.md for full column definitions, indexes and RLS policies.


The wizard, step by step

The framework wizard renders a configurable sequence of steps. Default sequence:

UPLOAD → PROVIDER → MAPPING → REVIEW

A domain can insert its own steps and reorder via ImportFlowConfig.steps. Payments inserts TYPE and TARGET (and a RESOLVE step for the card flow):

TYPE → TARGET → UPLOAD → PROVIDER → MAPPING → RESOLVE → REVIEW

The bank flow drops PROVIDER and RESOLVE (the preset is auto-picked at TARGET and there's nothing to disambiguate per-row).

Step responsibilities

Step Owned by What it does
TYPE Domain Domain-defined entry split. Payments asks "bank or card?" and stamps extras.statementType. Domains without a split skip this step entirely
TARGET Domain Pick the target entity for the import (Payments: the payment method). Optional — used when the domain has a 1-to-many concept that must be resolved before mapping
UPLOAD Framework Read the file → tokenise into headers + raw rows. Validates against ImportFlowConfig.acceptedFormats. Stores uploadedText for session persistence + uploadedHeaders for the Provider step to score against
PROVIDER Framework Render a tile per matching import_mappings row (per-domain built-ins + per-project customs + the universal generic fallback). Tiles are scored against the file's headers via preset.requiredHeaders.intersection
MAPPING Framework Show one row per import_fields destination for the active entity_type. Each row holds a single-select (or multi-select when allow_multiple=true) over the file's headers. Required gating enforced before NEXT
RESOLVE Domain Per-row decisions the framework can't make. Payments uses it to pair each distinct card_last_four with an existing PM or a "skip" decision (card flow only)
REVIEW Domain Final per-row preview before commit. Domain renders the table inside the generic Review shell. The wizard calls nextLabel(REVIEW, …) to compute the commit-button text (Payments: "Process N transactions")

State

The wizard's generic state lives in ImportWizardGenericState:

{
  presetKey:        string | null;
  savedMappingId:   string | null;
  uploadedHeaders:  string[] | null;
  uploadedText:     string | null;     // for hydrate
  uploadedFile:     {name, size} | null;
  columnMapping:    ColumnMapping;     // file-header-first map
  parsedRows:       ParsedRow[] | null;
}

Plus a typed extras: TExtras blob — each domain declares its own. The wizard serialises a subset of extras to URL query params (config.serializeExtras / parseExtras) and the full blob to sessionStorage (so a refresh restores the user's place).

Hydration on refresh

The wizard persists state to sessionStorage keyed by (projectId, entityType). On mount:

  1. Read snapshot → if uploadedText is missing, bounce to UPLOAD.
  2. Re-run preset.parse(uploadedText) to rebuild parsedRows.
  3. Restore columnMapping, presetKey, savedMappingId, and the per-domain extras blob.

parsedRows is never persisted — the snapshot would balloon for large statements. The framework recomputes on hydrate.


Configuration: ImportFlowConfig<TExtras, TPreset>

The domain config object the page-client passes to <ImportWizard>. Required fields:

{
  entityType:               string,                            // matches import_mappings.entity_type
  acceptedFormats:          ImportFileFormatKey[],             // ['csv'] today
  steps:                    ImportStepKey[] | (extras) => ,   // step sequence
  customSteps:              Record<ImportStepKey, renderer>,   // domain-step bodies
  initialExtras:            TExtras,                            // stamped on mount
  reviewContent:            (props) => ReactNode,              // REVIEW body
  commit:                   { buildContext: (props) => Record<string, unknown> },
  canCreateCustomMappings:  boolean,                            // Provider-step "generic" tile + Save-as-custom UI
}

Optional hooks:

{
  canAdvance(step, props)         : boolean | undefined,
  stepTitle(step, props)          : string | undefined,
  nextLabel(step, props)          : string | undefined,        // override the "Next" button
  serializeExtras(extras)         : Record<string, string|null>,
  parseExtras(searchParams)       : Partial<TExtras>,
  presetMatcher                   : ImportPresetMatcher,       // Provider-step scoring override
  buildCreatePayload(input)       : CreateImportMapping,       // save-as-custom payload builder
}

The framework's <ImportWizard> accepts two top-level callbacks for cross-cutting domain hooks:

<ImportWizard
  config={paymentsImportFlowConfig}
  filterMappings={(mappings, {extras}) => }   // narrow Provider tiles per extras
  filterFields={(fields, {extras}) => }       // hide destinations per extras
/>

Payments uses filterMappings to narrow per statementType (bank vs card) and filterFields to hide card_last_four from the bank flow.


How a preset works

A preset is a small object satisfying:

interface ImportPreset {
  key: ImportPresetKey;
  headerSignature: (headers: string[]) => boolean; // does this file look like our shape?
  requiredHeaders: readonly string[]; // for Provider step scoring
  parse: (text: string) => PresetParseResult;
  reviewColumns?: readonly ReviewColumnSpec[]; // optional REVIEW column declarations
}

Domains can extend ImportPreset with their own typed shape. Payments declares:

interface PaymentsImportPreset extends ImportPreset {
  categoryColumn?: string;
  amountColumns?: readonly string[];
  currencyColumn?: string;
  typeColumn?: string;
  includedTypes?: readonly string[];
  cashTypes?: readonly string[];
  statusColumn?: string;
  includedStatuses?: readonly string[];
  provider?: string; // narrows the Resolve "Pick existing" PM picker
  groupColumn?: string; // pairs Fee rows with their main row
}

The framework never reads these. They're consumed exclusively by payments-domain code (classifyPresetFilter, classifyReviewRow, useResolveCardSummaries).

What a preset does (Payments examples)

  • bankGenericPreset — permissive headerSignature (requires only Date and Amount). No type / status / cash gates. Every kept row commits against the single PM picked at TARGET.
  • cardEqualsPreset — strict 7-column headerSignature (both Created date (UTC) — the purchase date the mapping stores — and Completed date (UTC) are required, so pre-created-date legacy exports deliberately fail detection). Filters by Type ∈ ['Card'], Status ∈ ['Complete']. Pairs Fee rows with their main via groupColumn: 'Group transaction ID'. Cash short-circuit on Type='Cash withdrawal'.
  • cardRevolutPreset — strict headerSignature matching Revolut Business "Transaction statement". Filters by Type ∈ ['CARD_PAYMENT']. Carries provider: 'Revolut' so the Resolve picker narrows to Revolut-issued cards.

The ParsedRow boundary

A preset's only job is to tokenise. No domain coercion happens in parse — the output ParsedRow.cells is Record<header, raw string>. Date format / amount sign / masked-PAN extraction / description concatenation all live in the per-domain typing layer (lib/import/payments/typing/index.ts), keyed off import_fields.value_kind + per-preset coercion config.

This is the key invariant that lets two domains share a preset when the on-the-wire format matches. It also keeps the universal generic preset honest — it splits on commas and that's it.


The DB tables, in one paragraph each

import_mappings

One row per available mapping: the universal generic fallback, per-domain seeded built-ins, and user-saved customs. Discriminated by entity_type (NULL for the universal generic, 'payments' for payments rows). Domain-specific values like Payments' payment_method_type and payment_method_id live on the opaque domain_config JSONB column — the framework never reads inside it. is_custom=true rows are project-scoped and editable; is_custom=false rows are global templates seeded by migration and immutable from the app.

import_fields

Catalog of destination fields per entity_type. The Mapping step renders one row per record (filtered by the domain's filterFields callback). value_kind is a closed framework-level coercion contract (date | number | text | currency_code | external_id); domain-specific tweaks like Payments' masked-PAN extraction live in per-domain typing code keyed off field.key.

import_runs

Per-import audit log. One row per import run, written by the per-domain TS commit worker. Stores the source attachment + mapping, row-count tallies (total_rows = rows in the file before preset filter; processed_rows = rows the worker tried; created_rows / updated_rows / skipped_rows / errored_rows), per-row outcomes (row_outcomes JSONB), the run status (success / partial / failed), and the user who ran it. The status is computed by the framework's runImport lifecycle helper based on errored_rows vs processed_rows: SUCCESS when errored_rows = 0, PARTIAL when 0 < errored_rows < processed_rows, FAILED when the worker throws OR errored_rows >= processed_rows.

row_outcomes is an opaque array of domain-shaped per-row entries (payments uses paymentsImportRunOutcomeSchema). To list "which file lines were skipped" filter the array for entries whose action === 'skipped' — the framework doesn't denormalise this because it's a one-liner client-side.

Commit workers

There is no SQL dispatcher. Each domain ships its own TS commit worker — a Server Action under services/<domain>/actions.ts — that:

  1. Receives the typed ImportRow[] plus the domain context payload (extras.commit.buildContext(...)).
  2. Inserts an import_runs row to open the audit.
  3. Calls the existing per-table service helpers (createPayment, updatePayment, …) to apply the domain's commit logic.
  4. Updates the import_runs row with counts + per-row outcomes.
  5. Returns a summary the wizard renders.

The framework owns parsing, mapping, the file-hash dedupe, and the run audit table; domains own writes.


Runbook: add a new preset to an existing domain

Example: shipping a "HSBC card statement" preset for Payments.

1. Create the parser

packages/app/src/lib/import/payments/presets/cardHsbc.ts
import { Alignment } from '@/constants/common';
import { ImportReviewColumnFormat } from '@/constants/import';
import {
  PaymentsImportPresetKey,
  PaymentsReviewColumnTransform,
} from '@/constants/payment';
import type { ParsedRow, PresetParseResult } from '@/types/import';
import type { PaymentsImportPreset } from '@/types/payment';

const REQUIRED_HEADERS = [
  'Date',
  'Type',
  'Description',
  'Amount',
  'Card Number',
];

export const cardHsbcPreset: PaymentsImportPreset = {
  key: PaymentsImportPresetKey.CARD_HSBC,
  requiredHeaders: REQUIRED_HEADERS,

  // Domain metadata — payments-only, framework ignores
  typeColumn: 'Type',
  includedTypes: ['Purchase'],
  cashTypes: ['Cash advance'],
  statusColumn: 'Status',
  includedStatuses: ['Posted'],
  amountColumns: ['Amount'],
  currencyColumn: 'Currency',
  categoryColumn: 'Type',
  provider: 'HSBC',

  // REVIEW columns — declared in display order
  reviewColumns: [
    {
      header: 'Date',
      source: { kind: 'rawHeader', rawHeader: 'Date' },
      format: ImportReviewColumnFormat.DATE,
      size: 140,
    },
    {
      header: 'Card Identifier',
      source: { kind: 'rawHeader', rawHeader: 'Card Number' },
      transform: PaymentsReviewColumnTransform.CARD_LAST_FOUR,
      size: 130,
    },
    {
      header: 'Type',
      source: { kind: 'rawHeader', rawHeader: 'Type' },
      size: 140,
    },
    {
      header: 'Description',
      source: { kind: 'destination', destination: 'description' },
      size: 260,
    },
    {
      header: 'Amount',
      source: { kind: 'destination', destination: 'amount' },
      format: ImportReviewColumnFormat.CURRENCY,
      align: Alignment.RIGHT,
      size: 140,
    },
  ],

  headerSignature: (headers) => {
    const set = new Set(headers.map((h) => h.trim()));
    return REQUIRED_HEADERS.every((h) => set.has(h));
  },

  parse: (csvText: string): PresetParseResult => {
    // … use papaparse to tokenise, return {rows, headers, warnings}
  },
};

2. Register the preset

Add the new key to the enum:

// packages/app/src/constants/payment.ts
export enum PaymentsImportPresetKey {
  BANK_GENERIC = 'bank.generic',
  CARD_EQUALS = 'card.equals',
  CARD_REVOLUT = 'card.revolut',
  CARD_HSBC = 'card.hsbc', // new
}

Register the preset:

// packages/app/src/lib/import/payments/presets/index.ts
import { cardHsbcPreset } from './cardHsbc';

const PAYMENTS_PRESETS = {
  [PaymentsImportPresetKey.BANK_GENERIC]: bankGenericPreset,
  [PaymentsImportPresetKey.CARD_EQUALS]: cardEqualsPreset,
  [PaymentsImportPresetKey.CARD_REVOLUT]: cardRevolutPreset,
  [PaymentsImportPresetKey.CARD_HSBC]: cardHsbcPreset, // new
};

const PAYMENTS_DETECTION_ORDER = [
  PaymentsImportPresetKey.CARD_REVOLUT,
  PaymentsImportPresetKey.CARD_EQUALS,
  PaymentsImportPresetKey.CARD_HSBC, // before BANK_GENERIC (specific first)
  PaymentsImportPresetKey.BANK_GENERIC,
];

3. Ship the seed migration

cd packages/database
npx supabase migration new add_hsbc_card_import_preset

Insert one import_mappings row for the built-in tile (optional logo + example CSV via attachments):

INSERT INTO public.import_mappings (
    id, project_id, entity_type, source_preset, name, description, provider,
    is_custom, domain_config, mapping_json,
    logo_attachment_id, example_attachment_id,
    created_by_user_id, updated_by_user_id
) VALUES (
    gen_random_uuid(),
    NULL, 'payments', 'card.hsbc', 'HSBC Card',
    'HSBC business card statement export.',
    'HSBC', false,
    '{"payment_method_type": "Card"}'::jsonb,
    '{
        "scheduled_date":  ["Date"],
        "amount":          ["Amount"],
        "currency_code":   ["Currency"],
        "description":     ["Description"],
        "card_last_four":  ["Card Number"]
     }'::jsonb,
    NULL,  -- logo (add when shipped)
    NULL,  -- example CSV (add when shipped)
    (SELECT id FROM public.users WHERE is_internal = true LIMIT 1),
    (SELECT id FROM public.users WHERE is_internal = true LIMIT 1)
);

4. Tests

  • Unit test the parse() body against a sample row.
  • Unit test headerSignature accepts a real HSBC export header line and rejects others.
  • Add a cardHsbcPreset test fixture mirroring cardEqualsPreset.test.ts patterns.

That's it. The Provider step picks up the new tile automatically; the Mapping step reads its mapping_json and walks the user through the destination list.


Runbook: add a new domain

Example: shipping 'budgets' as a second import domain.

1. Pick keys and shape

Decide your entity_type discriminator ('budgets'), the destination field list, and the shape of your domain_config JSONB if you need per-row discriminators.

2. DB seed

cd packages/database
npx supabase migration new add_budgets_csv_import

Add:

  • N rows in import_fields for the budget destination keys (period_id, category_id, amount, description, …) with appropriate value_kind.
  • One or more rows in import_mappings for built-in budget presets. entity_type='budgets', is_custom=false, project_id=NULL, optional domain_config per your shape.

Optional: a partial unique index on import_mappings for any uniqueness rule specific to your domain (mirroring uq_import_mappings_bank_per_pm), reading via domain_config->>'<your_key>'.

3. Commit worker (Server Action)

Add the domain's commit worker under services/budgets/actions.ts:

'use server';

import { withRequestContext } from '@/lib/context/requestContext';
import * as budgetsServer from '@/services/budgets/server';
import * as importServer from '@/services/import/server';
import type { ServiceResponse } from '@/types/database';
import type { ImportRow } from '@/types/import';
import type { BudgetImportSummary } from '@/types/budget';

export async function processBudgetImportAction(payload: {
  projectId: string;
  rows: ImportRow[];
  context: { period_id: string };
  // …source preset, statement type, mapping id, attachment id, etc.
}): Promise<ServiceResponse<BudgetImportSummary>> {
  return withRequestContext(async () => {
    const run = await importServer.createImportRun({
      project_id: payload.projectId,
      entity_type: 'budgets',
      // …
    });
    const outcomes: ImportRunOutcome[] = [];
    for (const row of payload.rows) {
      // Apply your domain's commit logic via existing service helpers
      // (createBudgetItem, updateBudgetItem, …) and push an outcome.
    }
    return importServer.finalizeImportRun(run.id, {
      processed_rows: /* rows the worker tried */,
      created_rows: /* … */,
      updated_rows: /* … */,
      skipped_rows: /* … */,
      errored_rows: /* … */,
      row_outcomes: outcomes,
      status: 'success', // 'partial' if 0 < errored < processed; 'failed' if errored >= processed
    });
  });
}

There's no SQL dispatcher — each domain wires its own action. The framework only owns createImportRun + finalizeImportRun helpers (in services/import/) so the audit shape stays uniform.

4. App-side types

Add a domain-level extras shape + (optional) preset-extension type:

// packages/app/src/types/budget.ts
export interface BudgetsImportExtras {
  periodId: string | null;
  // … whatever else the flow needs
}

export interface BudgetsImportPreset extends ImportPreset {
  // domain-specific preset metadata, if any
}

Add domain constants if needed:

// packages/app/src/constants/budget.ts
export enum BudgetsImportPresetKey {
  GENERIC_BUDGET = 'budgets.generic',
}

5. App-side directories

Create a new directory tree mirroring the payments pattern:

packages/app/src/lib/import/budgets/
├── flowConfig.tsx               your ImportFlowConfig<BudgetsImportExtras, BudgetsImportPreset>
├── presets/
│   ├── index.ts                 your getBudgetsPresetByKey, detectBudgetsPreset
│   └── budgetsGeneric.ts        starter preset (or several)
└── typing/                      coercion helpers + applyBudgetsMapping

packages/app/src/components/budgets/import/
└── (whatever domain-specific step bodies you need)

6. Schema narrowing (optional)

If your domain_config has typed shape rules, ship a domain refinement schema mirroring paymentsCreateImportMappingSchema:

// packages/app/src/schemas/importMapping.schema.ts
export const budgetsCreateImportMappingSchema = createImportMappingSchema
  .extend({ domain_config: budgetsImportDomainConfigSchema })
  .refine();

The framework's createImportMappingSchema already accepts your domain_config as opaque — the refined schema is only needed if you want bank-vs-card-style shape invariants enforced at the wizard boundary.

7. Mount the wizard

// packages/app/src/app/.../budgets/import/page-client.tsx
import { ImportWizard } from '@/components/import/ImportWizard';
import { budgetsImportFlowConfig, resolveBudgetsImportPreset } from '@/lib/import/budgets/flowConfig';

export function ImportBudgetsPageClient() {
  return (
    <ImportWizard
      projectId={projectId}
      config={budgetsImportFlowConfig}
      presetByKey={resolveBudgetsImportPreset}
      onClose={}
    />
  );
}

8. What you do NOT touch

The whole point of the framework split is that adding a new domain is additive. You do not edit:

  • components/import/* (the wizard shell + framework steps)
  • services/import/* (mapping CRUD + import-run audit helpers)
  • hooks/queries/useImportQuery.ts
  • lib/import/typing/*
  • lib/import/presets/generic.ts
  • schemas/importMapping.schema.ts framework portion (importMappingSchema, createImportMappingSchema, updateImportMappingSchema)
  • The import_mappings / import_fields / import_runs table definitions

If you find yourself needing to edit any of these to support your domain, that's a signal a domain concept has leaked into the framework. Stop and refactor it into a domain hook on ImportFlowConfig (the way filterFields / filterMappings / nextLabel / serializeExtras already work).


Common patterns

Hiding fields per flow

The Mapping step queries the full import_fields list for the active entity_type. The page-client narrows via the filterFields callback:

const filterFields = (fields, { extras }) => {
  if (extras.statementType === ImportStatementType.BANK) {
    return fields.filter((f) => !BANK_HIDDEN_FIELD_KEYS.has(f.key));
  }
  return fields;
};

<ImportWizard config={} filterFields={filterFields} />

Auto-resolving a preset

Some flows can skip the Provider step entirely by stamping a preset from a domain step. Use setPresetSelection from ImportStepRenderProps:

customSteps: {
  [ImportStepKey.TARGET]: (props) => (
    <TargetPicker
      onChange={(pickedId) => {
        props.setExtras({ ...props.extras, paymentMethodId: pickedId });
        const saved = props.mappings.find(
          (m) => getPaymentsImportDomainConfig(m)?.payment_method_id === pickedId,
        );
        props.setPresetSelection(
          saved
            ? { presetKey: saved.source_preset, savedMappingId: saved.id }
            : { presetKey: PaymentsImportPresetKey.BANK_GENERIC, savedMappingId: null },
        );
      }}
    />
  ),
}

Then drop PROVIDER from the step list when this flow runs:

steps: (extras) =>
  extras.statementType === ImportStatementType.BANK
    ? [TYPE, TARGET, UPLOAD, MAPPING, REVIEW] // PROVIDER + RESOLVE dropped
    : [TYPE, TARGET, UPLOAD, PROVIDER, MAPPING, RESOLVE, REVIEW];

Restoring per-row decisions after a refresh

The wizard mirrors state.uploadedText, columnMapping, presetKey, uploadedFile.name, and the full extras blob to sessionStorage. The hydrate effect on mount re-parses the file and restores everything. URL-derived extras (via parseExtras) take precedence over the snapshot so a freshly-edited URL still wins.

If you want a field to survive refresh, put it on extras. Don't reach for separate state.


Layering — what's NOT generic

Two things in the current codebase have payment-specific names but live in framework-shaped files:

  1. services/import/base.ts → getBankMappingForPaymentMethod(projectId, paymentMethodId) — the function body reads from domain_config->>'payment_method_id' correctly, but the name is payments-flavoured. A second domain wanting the same lookup pattern could:
  2. Reuse the function via findOne directly with a different column key, or
  3. Rename to getMappingByDomainConfigKey(entityType, key, value) once a second use case exists.

  4. hooks/queries/useImportQuery.ts → useBankImportMappingQuery — same story. A second domain doesn't get hurt (it just doesn't use this hook), but the name is payments-specific.

Both are deliberate today: there's no second consumer, and renaming pre-emptively adds churn without benefit. Document this as a known wrinkle.