Import Framework¶
Audience: developers extending the import system — adding a new preset for an existing domain, adding a new domain entirely, or modifying framework behaviour.
This document covers the generic import framework (packages/app/src/components/import/*, packages/app/src/lib/import/typing/*, packages/app/src/services/import/*) and explains where the payments domain layers on top of it. The split is enforced by code organisation, not just convention: framework code never imports from a domain layer.
At a glance¶
The framework is a polymorphic wizard that takes a CSV file and a per-domain configuration object (ImportFlowConfig<TExtras, TPreset>) and produces a typed ImportRow[] plus a per-domain context payload. The wizard hands them to a domain-owned TS commit worker (Server Action under services/<domain>/) which calls the existing per-table service helpers, writes the run + per-row outcomes to import_runs, and returns a summary the wizard displays.
Today Payments is the only domain. The architecture is designed so a second domain (Budgets, Transactions, …) ships entirely in new directories with zero changes to framework code or the shared DB tables — only its own commit worker.
Files & directories¶
packages/app/src/
├── constants/import.ts FRAMEWORK
├── constants/payment.ts PAYMENTS DOMAIN
├── types/import.ts FRAMEWORK
├── types/payment.ts PAYMENTS DOMAIN
├── schemas/importMapping.schema.ts BOTH (framework schemas + paymentsCreate refinement)
│
├── lib/import/
│ ├── typing/index.ts FRAMEWORK: coerceValueByKind, getCellsForDestination
│ ├── presets/
│ │ ├── generic.ts FRAMEWORK: universal `generic` preset
│ │ └── index.ts
│ └── payments/ PAYMENTS DOMAIN
│ ├── flowConfig.tsx
│ ├── domainConfig.ts
│ ├── presets/ bankGeneric, cardEquals, cardRevolut
│ ├── review/classifyReviewRow.ts
│ └── typing/ classifyPresetFilter, coercion, applyPaymentsMapping
│
├── components/import/ FRAMEWORK
│ ├── ImportWizard/
│ ├── ImportStepUpload/
│ ├── ImportStepProvider/
│ ├── ImportStepMapping/
│ ├── ImportSaveCustomDialog/
│ ├── ImportLogo/
│ └── ImportExampleDownloadButton/
│
├── components/payments/import/ PAYMENTS DOMAIN
│ ├── ImportStepType/ TYPE step body (bank vs card)
│ ├── ImportStepPaymentMethod/ TARGET step body
│ ├── ImportStepResolve/ RESOLVE step (per-card decisions)
│ ├── ImportStepReview/ REVIEW step (per-row classifier display)
│ ├── ImportCreatePaymentMethodDialog/
│ └── PaymentsImportTargetSlot/
│
├── services/import/ FRAMEWORK
│ ├── base.ts CRUD + getBankMappingForPaymentMethod (payments-tinted)
│ ├── server.ts
│ └── actions.ts
│
└── hooks/queries/useImportQuery.ts FRAMEWORK
Database¶
public.import_mappings framework — domain-polymorphic, opaque `domain_config` JSONB
public.import_fields framework — destination-field catalog per entity_type
public.import_runs framework — per-import audit log (written by each domain's TS commit worker)
See docs/architecture/database/FINANCIAL.md for full column definitions, indexes and RLS policies.
The wizard, step by step¶
The framework wizard renders a configurable sequence of steps. Default sequence:
UPLOAD → PROVIDER → MAPPING → REVIEW
A domain can insert its own steps and reorder via ImportFlowConfig.steps. Payments inserts TYPE and TARGET (and a RESOLVE step for the card flow):
TYPE → TARGET → UPLOAD → PROVIDER → MAPPING → RESOLVE → REVIEW
The bank flow drops PROVIDER and RESOLVE (the preset is auto-picked at TARGET and there's nothing to disambiguate per-row).
Step responsibilities¶
| Step | Owned by | What it does |
|---|---|---|
TYPE |
Domain | Domain-defined entry split. Payments asks "bank or card?" and stamps extras.statementType. Domains without a split skip this step entirely |
TARGET |
Domain | Pick the target entity for the import (Payments: the payment method). Optional — used when the domain has a 1-to-many concept that must be resolved before mapping |
UPLOAD |
Framework | Read the file → tokenise into headers + raw rows. Validates against ImportFlowConfig.acceptedFormats. Stores uploadedText for session persistence + uploadedHeaders for the Provider step to score against |
PROVIDER |
Framework | Render a tile per matching import_mappings row (per-domain built-ins + per-project customs + the universal generic fallback). Tiles are scored against the file's headers via preset.requiredHeaders.intersection |
MAPPING |
Framework | Show one row per import_fields destination for the active entity_type. Each row holds a single-select (or multi-select when allow_multiple=true) over the file's headers. Required gating enforced before NEXT |
RESOLVE |
Domain | Per-row decisions the framework can't make. Payments uses it to pair each distinct card_last_four with an existing PM or a "skip" decision (card flow only) |
REVIEW |
Domain | Final per-row preview before commit. Domain renders the table inside the generic Review shell. The wizard calls nextLabel(REVIEW, …) to compute the commit-button text (Payments: "Process N transactions") |
State¶
The wizard's generic state lives in ImportWizardGenericState:
{
presetKey: string | null;
savedMappingId: string | null;
uploadedHeaders: string[] | null;
uploadedText: string | null; // for hydrate
uploadedFile: {name, size} | null;
columnMapping: ColumnMapping; // file-header-first map
parsedRows: ParsedRow[] | null;
}
Plus a typed extras: TExtras blob — each domain declares its own. The wizard serialises a subset of extras to URL query params (config.serializeExtras / parseExtras) and the full blob to sessionStorage (so a refresh restores the user's place).
Hydration on refresh¶
The wizard persists state to sessionStorage keyed by (projectId, entityType). On mount:
- Read snapshot → if
uploadedTextis missing, bounce toUPLOAD. - Re-run
preset.parse(uploadedText)to rebuildparsedRows. - Restore
columnMapping,presetKey,savedMappingId, and the per-domainextrasblob.
parsedRows is never persisted — the snapshot would balloon for large statements. The framework recomputes on hydrate.
Configuration: ImportFlowConfig<TExtras, TPreset>¶
The domain config object the page-client passes to <ImportWizard>. Required fields:
{
entityType: string, // matches import_mappings.entity_type
acceptedFormats: ImportFileFormatKey[], // ['csv'] today
steps: ImportStepKey[] | (extras) => …, // step sequence
customSteps: Record<ImportStepKey, renderer>, // domain-step bodies
initialExtras: TExtras, // stamped on mount
reviewContent: (props) => ReactNode, // REVIEW body
commit: { buildContext: (props) => Record<string, unknown> },
canCreateCustomMappings: boolean, // Provider-step "generic" tile + Save-as-custom UI
}
Optional hooks:
{
canAdvance(step, props) : boolean | undefined,
stepTitle(step, props) : string | undefined,
nextLabel(step, props) : string | undefined, // override the "Next" button
serializeExtras(extras) : Record<string, string|null>,
parseExtras(searchParams) : Partial<TExtras>,
presetMatcher : ImportPresetMatcher, // Provider-step scoring override
buildCreatePayload(input) : CreateImportMapping, // save-as-custom payload builder
}
The framework's <ImportWizard> accepts two top-level callbacks for cross-cutting domain hooks:
<ImportWizard
config={paymentsImportFlowConfig}
filterMappings={(mappings, {extras}) => …} // narrow Provider tiles per extras
filterFields={(fields, {extras}) => …} // hide destinations per extras
/>
Payments uses filterMappings to narrow per statementType (bank vs card) and filterFields to hide card_last_four from the bank flow.
How a preset works¶
A preset is a small object satisfying:
interface ImportPreset {
key: ImportPresetKey;
headerSignature: (headers: string[]) => boolean; // does this file look like our shape?
requiredHeaders: readonly string[]; // for Provider step scoring
parse: (text: string) => PresetParseResult;
reviewColumns?: readonly ReviewColumnSpec[]; // optional REVIEW column declarations
}
Domains can extend ImportPreset with their own typed shape. Payments declares:
interface PaymentsImportPreset extends ImportPreset {
categoryColumn?: string;
amountColumns?: readonly string[];
currencyColumn?: string;
typeColumn?: string;
includedTypes?: readonly string[];
cashTypes?: readonly string[];
statusColumn?: string;
includedStatuses?: readonly string[];
provider?: string; // narrows the Resolve "Pick existing" PM picker
groupColumn?: string; // pairs Fee rows with their main row
}
The framework never reads these. They're consumed exclusively by payments-domain code (classifyPresetFilter, classifyReviewRow, useResolveCardSummaries).
What a preset does (Payments examples)¶
bankGenericPreset— permissiveheaderSignature(requires onlyDateandAmount). No type / status / cash gates. Every kept row commits against the single PM picked at TARGET.cardEqualsPreset— strict 7-columnheaderSignature(bothCreated date (UTC)— the purchase date the mapping stores — andCompleted date (UTC)are required, so pre-created-date legacy exports deliberately fail detection). Filters byType ∈ ['Card'],Status ∈ ['Complete']. Pairs Fee rows with their main viagroupColumn: 'Group transaction ID'. Cash short-circuit onType='Cash withdrawal'.cardRevolutPreset— strictheaderSignaturematching Revolut Business "Transaction statement". Filters byType ∈ ['CARD_PAYMENT']. Carriesprovider: 'Revolut'so the Resolve picker narrows to Revolut-issued cards.
The ParsedRow boundary¶
A preset's only job is to tokenise. No domain coercion happens in parse — the output ParsedRow.cells is Record<header, raw string>. Date format / amount sign / masked-PAN extraction / description concatenation all live in the per-domain typing layer (lib/import/payments/typing/index.ts), keyed off import_fields.value_kind + per-preset coercion config.
This is the key invariant that lets two domains share a preset when the on-the-wire format matches. It also keeps the universal generic preset honest — it splits on commas and that's it.
The DB tables, in one paragraph each¶
import_mappings¶
One row per available mapping: the universal generic fallback, per-domain seeded built-ins, and user-saved customs. Discriminated by entity_type (NULL for the universal generic, 'payments' for payments rows). Domain-specific values like Payments' payment_method_type and payment_method_id live on the opaque domain_config JSONB column — the framework never reads inside it. is_custom=true rows are project-scoped and editable; is_custom=false rows are global templates seeded by migration and immutable from the app.
import_fields¶
Catalog of destination fields per entity_type. The Mapping step renders one row per record (filtered by the domain's filterFields callback). value_kind is a closed framework-level coercion contract (date | number | text | currency_code | external_id); domain-specific tweaks like Payments' masked-PAN extraction live in per-domain typing code keyed off field.key.
import_runs¶
Per-import audit log. One row per import run, written by the per-domain TS commit worker. Stores the source attachment + mapping, row-count tallies (total_rows = rows in the file before preset filter; processed_rows = rows the worker tried; created_rows / updated_rows / skipped_rows / errored_rows), per-row outcomes (row_outcomes JSONB), the run status (success / partial / failed), and the user who ran it. The status is computed by the framework's runImport lifecycle helper based on errored_rows vs processed_rows: SUCCESS when errored_rows = 0, PARTIAL when 0 < errored_rows < processed_rows, FAILED when the worker throws OR errored_rows >= processed_rows.
row_outcomes is an opaque array of domain-shaped per-row entries (payments uses paymentsImportRunOutcomeSchema). To list "which file lines were skipped" filter the array for entries whose action === 'skipped' — the framework doesn't denormalise this because it's a one-liner client-side.
Commit workers¶
There is no SQL dispatcher. Each domain ships its own TS commit worker — a Server Action under services/<domain>/actions.ts — that:
- Receives the typed
ImportRow[]plus the domain context payload (extras.commit.buildContext(...)). - Inserts an
import_runsrow to open the audit. - Calls the existing per-table service helpers (
createPayment,updatePayment, …) to apply the domain's commit logic. - Updates the
import_runsrow with counts + per-row outcomes. - Returns a summary the wizard renders.
The framework owns parsing, mapping, the file-hash dedupe, and the run audit table; domains own writes.
Runbook: add a new preset to an existing domain¶
Example: shipping a "HSBC card statement" preset for Payments.
1. Create the parser¶
packages/app/src/lib/import/payments/presets/cardHsbc.ts
import { Alignment } from '@/constants/common';
import { ImportReviewColumnFormat } from '@/constants/import';
import {
PaymentsImportPresetKey,
PaymentsReviewColumnTransform,
} from '@/constants/payment';
import type { ParsedRow, PresetParseResult } from '@/types/import';
import type { PaymentsImportPreset } from '@/types/payment';
const REQUIRED_HEADERS = [
'Date',
'Type',
'Description',
'Amount',
'Card Number',
];
export const cardHsbcPreset: PaymentsImportPreset = {
key: PaymentsImportPresetKey.CARD_HSBC,
requiredHeaders: REQUIRED_HEADERS,
// Domain metadata — payments-only, framework ignores
typeColumn: 'Type',
includedTypes: ['Purchase'],
cashTypes: ['Cash advance'],
statusColumn: 'Status',
includedStatuses: ['Posted'],
amountColumns: ['Amount'],
currencyColumn: 'Currency',
categoryColumn: 'Type',
provider: 'HSBC',
// REVIEW columns — declared in display order
reviewColumns: [
{
header: 'Date',
source: { kind: 'rawHeader', rawHeader: 'Date' },
format: ImportReviewColumnFormat.DATE,
size: 140,
},
{
header: 'Card Identifier',
source: { kind: 'rawHeader', rawHeader: 'Card Number' },
transform: PaymentsReviewColumnTransform.CARD_LAST_FOUR,
size: 130,
},
{
header: 'Type',
source: { kind: 'rawHeader', rawHeader: 'Type' },
size: 140,
},
{
header: 'Description',
source: { kind: 'destination', destination: 'description' },
size: 260,
},
{
header: 'Amount',
source: { kind: 'destination', destination: 'amount' },
format: ImportReviewColumnFormat.CURRENCY,
align: Alignment.RIGHT,
size: 140,
},
],
headerSignature: (headers) => {
const set = new Set(headers.map((h) => h.trim()));
return REQUIRED_HEADERS.every((h) => set.has(h));
},
parse: (csvText: string): PresetParseResult => {
// … use papaparse to tokenise, return {rows, headers, warnings}
},
};
2. Register the preset¶
Add the new key to the enum:
// packages/app/src/constants/payment.ts
export enum PaymentsImportPresetKey {
BANK_GENERIC = 'bank.generic',
CARD_EQUALS = 'card.equals',
CARD_REVOLUT = 'card.revolut',
CARD_HSBC = 'card.hsbc', // new
}
Register the preset:
// packages/app/src/lib/import/payments/presets/index.ts
import { cardHsbcPreset } from './cardHsbc';
const PAYMENTS_PRESETS = {
[PaymentsImportPresetKey.BANK_GENERIC]: bankGenericPreset,
[PaymentsImportPresetKey.CARD_EQUALS]: cardEqualsPreset,
[PaymentsImportPresetKey.CARD_REVOLUT]: cardRevolutPreset,
[PaymentsImportPresetKey.CARD_HSBC]: cardHsbcPreset, // new
};
const PAYMENTS_DETECTION_ORDER = [
PaymentsImportPresetKey.CARD_REVOLUT,
PaymentsImportPresetKey.CARD_EQUALS,
PaymentsImportPresetKey.CARD_HSBC, // before BANK_GENERIC (specific first)
PaymentsImportPresetKey.BANK_GENERIC,
];
3. Ship the seed migration¶
cd packages/database
npx supabase migration new add_hsbc_card_import_preset
Insert one import_mappings row for the built-in tile (optional logo + example CSV via attachments):
INSERT INTO public.import_mappings (
id, project_id, entity_type, source_preset, name, description, provider,
is_custom, domain_config, mapping_json,
logo_attachment_id, example_attachment_id,
created_by_user_id, updated_by_user_id
) VALUES (
gen_random_uuid(),
NULL, 'payments', 'card.hsbc', 'HSBC Card',
'HSBC business card statement export.',
'HSBC', false,
'{"payment_method_type": "Card"}'::jsonb,
'{
"scheduled_date": ["Date"],
"amount": ["Amount"],
"currency_code": ["Currency"],
"description": ["Description"],
"card_last_four": ["Card Number"]
}'::jsonb,
NULL, -- logo (add when shipped)
NULL, -- example CSV (add when shipped)
(SELECT id FROM public.users WHERE is_internal = true LIMIT 1),
(SELECT id FROM public.users WHERE is_internal = true LIMIT 1)
);
4. Tests¶
- Unit test the
parse()body against a sample row. - Unit test
headerSignatureaccepts a real HSBC export header line and rejects others. - Add a
cardHsbcPresettest fixture mirroringcardEqualsPreset.test.tspatterns.
That's it. The Provider step picks up the new tile automatically; the Mapping step reads its mapping_json and walks the user through the destination list.
Runbook: add a new domain¶
Example: shipping 'budgets' as a second import domain.
1. Pick keys and shape¶
Decide your entity_type discriminator ('budgets'), the destination field list, and the shape of your domain_config JSONB if you need per-row discriminators.
2. DB seed¶
cd packages/database
npx supabase migration new add_budgets_csv_import
Add:
- N rows in
import_fieldsfor the budget destination keys (period_id,category_id,amount,description, …) with appropriatevalue_kind. - One or more rows in
import_mappingsfor built-in budget presets.entity_type='budgets',is_custom=false,project_id=NULL, optionaldomain_configper your shape.
Optional: a partial unique index on import_mappings for any uniqueness rule specific to your domain (mirroring uq_import_mappings_bank_per_pm), reading via domain_config->>'<your_key>'.
3. Commit worker (Server Action)¶
Add the domain's commit worker under services/budgets/actions.ts:
'use server';
import { withRequestContext } from '@/lib/context/requestContext';
import * as budgetsServer from '@/services/budgets/server';
import * as importServer from '@/services/import/server';
import type { ServiceResponse } from '@/types/database';
import type { ImportRow } from '@/types/import';
import type { BudgetImportSummary } from '@/types/budget';
export async function processBudgetImportAction(payload: {
projectId: string;
rows: ImportRow[];
context: { period_id: string };
// …source preset, statement type, mapping id, attachment id, etc.
}): Promise<ServiceResponse<BudgetImportSummary>> {
return withRequestContext(async () => {
const run = await importServer.createImportRun({
project_id: payload.projectId,
entity_type: 'budgets',
// …
});
const outcomes: ImportRunOutcome[] = [];
for (const row of payload.rows) {
// Apply your domain's commit logic via existing service helpers
// (createBudgetItem, updateBudgetItem, …) and push an outcome.
}
return importServer.finalizeImportRun(run.id, {
processed_rows: /* rows the worker tried */,
created_rows: /* … */,
updated_rows: /* … */,
skipped_rows: /* … */,
errored_rows: /* … */,
row_outcomes: outcomes,
status: 'success', // 'partial' if 0 < errored < processed; 'failed' if errored >= processed
});
});
}
There's no SQL dispatcher — each domain wires its own action. The framework only owns createImportRun + finalizeImportRun helpers (in services/import/) so the audit shape stays uniform.
4. App-side types¶
Add a domain-level extras shape + (optional) preset-extension type:
// packages/app/src/types/budget.ts
export interface BudgetsImportExtras {
periodId: string | null;
// … whatever else the flow needs
}
export interface BudgetsImportPreset extends ImportPreset {
// domain-specific preset metadata, if any
}
Add domain constants if needed:
// packages/app/src/constants/budget.ts
export enum BudgetsImportPresetKey {
GENERIC_BUDGET = 'budgets.generic',
}
5. App-side directories¶
Create a new directory tree mirroring the payments pattern:
packages/app/src/lib/import/budgets/
├── flowConfig.tsx your ImportFlowConfig<BudgetsImportExtras, BudgetsImportPreset>
├── presets/
│ ├── index.ts your getBudgetsPresetByKey, detectBudgetsPreset
│ └── budgetsGeneric.ts starter preset (or several)
└── typing/ coercion helpers + applyBudgetsMapping
packages/app/src/components/budgets/import/
└── (whatever domain-specific step bodies you need)
6. Schema narrowing (optional)¶
If your domain_config has typed shape rules, ship a domain refinement schema mirroring paymentsCreateImportMappingSchema:
// packages/app/src/schemas/importMapping.schema.ts
export const budgetsCreateImportMappingSchema = createImportMappingSchema
.extend({ domain_config: budgetsImportDomainConfigSchema })
.refine(…);
The framework's createImportMappingSchema already accepts your domain_config as opaque — the refined schema is only needed if you want bank-vs-card-style shape invariants enforced at the wizard boundary.
7. Mount the wizard¶
// packages/app/src/app/.../budgets/import/page-client.tsx
import { ImportWizard } from '@/components/import/ImportWizard';
import { budgetsImportFlowConfig, resolveBudgetsImportPreset } from '@/lib/import/budgets/flowConfig';
export function ImportBudgetsPageClient() {
return (
<ImportWizard
projectId={projectId}
config={budgetsImportFlowConfig}
presetByKey={resolveBudgetsImportPreset}
onClose={…}
/>
);
}
8. What you do NOT touch¶
The whole point of the framework split is that adding a new domain is additive. You do not edit:
components/import/*(the wizard shell + framework steps)services/import/*(mapping CRUD + import-run audit helpers)hooks/queries/useImportQuery.tslib/import/typing/*lib/import/presets/generic.tsschemas/importMapping.schema.tsframework portion (importMappingSchema,createImportMappingSchema,updateImportMappingSchema)- The
import_mappings/import_fields/import_runstable definitions
If you find yourself needing to edit any of these to support your domain, that's a signal a domain concept has leaked into the framework. Stop and refactor it into a domain hook on ImportFlowConfig (the way filterFields / filterMappings / nextLabel / serializeExtras already work).
Common patterns¶
Hiding fields per flow¶
The Mapping step queries the full import_fields list for the active entity_type. The page-client narrows via the filterFields callback:
const filterFields = (fields, { extras }) => {
if (extras.statementType === ImportStatementType.BANK) {
return fields.filter((f) => !BANK_HIDDEN_FIELD_KEYS.has(f.key));
}
return fields;
};
<ImportWizard config={…} filterFields={filterFields} />
Auto-resolving a preset¶
Some flows can skip the Provider step entirely by stamping a preset from a domain step. Use setPresetSelection from ImportStepRenderProps:
customSteps: {
[ImportStepKey.TARGET]: (props) => (
<TargetPicker
onChange={(pickedId) => {
props.setExtras({ ...props.extras, paymentMethodId: pickedId });
const saved = props.mappings.find(
(m) => getPaymentsImportDomainConfig(m)?.payment_method_id === pickedId,
);
props.setPresetSelection(
saved
? { presetKey: saved.source_preset, savedMappingId: saved.id }
: { presetKey: PaymentsImportPresetKey.BANK_GENERIC, savedMappingId: null },
);
}}
/>
),
}
Then drop PROVIDER from the step list when this flow runs:
steps: (extras) =>
extras.statementType === ImportStatementType.BANK
? [TYPE, TARGET, UPLOAD, MAPPING, REVIEW] // PROVIDER + RESOLVE dropped
: [TYPE, TARGET, UPLOAD, PROVIDER, MAPPING, RESOLVE, REVIEW];
Restoring per-row decisions after a refresh¶
The wizard mirrors state.uploadedText, columnMapping, presetKey, uploadedFile.name, and the full extras blob to sessionStorage. The hydrate effect on mount re-parses the file and restores everything. URL-derived extras (via parseExtras) take precedence over the snapshot so a freshly-edited URL still wins.
If you want a field to survive refresh, put it on extras. Don't reach for separate state.
Layering — what's NOT generic¶
Two things in the current codebase have payment-specific names but live in framework-shaped files:
services/import/base.ts → getBankMappingForPaymentMethod(projectId, paymentMethodId)— the function body reads fromdomain_config->>'payment_method_id'correctly, but the name is payments-flavoured. A second domain wanting the same lookup pattern could:- Reuse the function via
findOnedirectly with a different column key, or -
Rename to
getMappingByDomainConfigKey(entityType, key, value)once a second use case exists. -
hooks/queries/useImportQuery.ts → useBankImportMappingQuery— same story. A second domain doesn't get hurt (it just doesn't use this hook), but the name is payments-specific.
Both are deliberate today: there's no second consumer, and renaming pre-emptively adds churn without benefit. Document this as a known wrinkle.
Related documentation¶
- Database — Import tables — column-level schema for
import_mappings,import_fields,import_runs - Database — Functions — import-related DB functions
- Coding standards — TS / Zod patterns the import schemas follow
- Service patterns — the
services/import/layer's base / server / actions split - Testing standards — how to write tests for new presets and domain code