Synthetic data: how a sandbox tests without real customers

The loop synthetic data breaks

A start-up with a lending product needs customer, account and transaction data to show its scoring works. A bank will not hand over real records to a firm which is not yet licensed, and a regulator will not admit a firm without evidence. Each side waits for the other.

The FCA’s answer was to make its Digital Sandbox permanent on 1 August 2023, with an operator chosen by tender in April 2023. At launch it offered more than 200 synthetic, public or anonymised data sets and over 1,000 application interfaces, covering payments and transactions data, social media data and credit data, so a firm builds and tests before the regulatory process begins. The idea has spread because it removes a barrier without lowering a standard.

What makes synthetic data useful

Random numbers are not enough. A useful data set holds together. A customer record links to accounts, accounts link to transactions, and totals reconcile. Generators should follow business logic, so income relates to credit limits and arrears appear in plausible proportions. They should also produce clean and messy versions, because production data is untidy and a product which only works on tidy data has not been tested.

Coverage matters as much as structure. Where a data set stands for a bank, it should represent the products the bank offers and no others. A postal bank without a credit licence should not have loans in its synthetic customer base. Inventing products defeats the point of realism.

Safety is something you check

The first risk is leakage. A generator trained or seeded on real records reproduces them. The second is coincidence. Invented values sometimes look like real identifiers, such as an identity number or a phone number, and a reviewer cannot tell the difference. The draft Data Protection Bill defines personal data broadly, including identifiers and the singling out of an individual (section 1), so any dataset which resembles real people deserves care.

The answer is a routine. Scan each data set for patterns shaped like Namibian identity numbers, phone numbers and email addresses. Record the result as an attestation attached to the data set. Keep real data out of the sandbox entirely, and keep each participant’s workspace separate so one firm’s test data and code are never visible to another. Confidentiality of intellectual property is a known barrier to joining sandboxes elsewhere, and isolation answers it structurally.

What synthetic data does not prove

Realism has a ceiling. Synthetic data built from published product ranges reproduces the shape of an institution’s offer. It does not reproduce how real customers behave, how fraud occurs or what volumes look like at peak. A model which scores well on invented fraud has shown only its ability to find invented fraud.

The honest labelling is simple. State what the data was built from, which checks were run, and what is out of scope. Where a supervisor later allows tests with real consumers under caps, this is a separate decision with separate protections, and it needs different evidence.

A specification for useful data

A buyer or a regulator needs a way to judge a synthetic data set before relying on it. Ask for six properties in writing. The first is referential integrity: every transaction belongs to an account, every account to a customer, and balances reconcile to the transactions beneath them. The second is business logic: relationships such as income to credit limit, or salary to spending pattern, follow stated rules. The third is distributional realism: age, region, income and product holdings follow published statistics, with the source named. The fourth is controlled messiness: late payments, duplicate records, missing fields and unusual descriptions appear at stated rates, so a product is tested on the untidy data it will meet.

The fifth is repeatability. A participant who changes a parameter should be able to regenerate the data set and compare results, and a reset of one workspace should rebuild its data and record the reset. The sixth is scope: the data set states which institutions and products it represents, and which it leaves out. A short specification of these six items, with evidence for each, is more valuable than a large file.

Why the data stays in the sandbox

Synthetic data is cheap to make and easy to leak into places it was not meant to go. Two rules reduce the risk. First, data used for testing never leaves the participant’s workspace, and the workspace is deleted at exit with a record of what was removed. Second, the sandbox operator holds no production personal data in any test area, and a scan runs before any new data set is released. Where both rules hold, a breach of the test environment exposes invented people.

Isolation also protects the participants. A firm testing a new credit model does not want a competitor to see its code or its results. Elsewhere, participants have raised confidentiality of their intellectual property as a barrier to joining a sandbox. Separate workspaces, with no shared filesystem and no route from one tenant to another, answer the concern structurally and not by promise.

What a supervisor learns from it

For a supervisor, synthetic data changes what evidence is worth asking for. Instead of a business plan and a promise, a firm supplies a record: this product, run against this data set, with these faults injected, produced these outcomes. The supervisor reads the same record a second time and asks whether it would be different with a messier data set or a different seed.

The limit remains the limit. A record from synthetic testing shows a product behaves as designed against a model. It does not show the product will earn customer trust, withstand a determined attacker or meet capital rules. Synthetic evidence supports a decision. The decision, and the responsibility for it, stay with the supervisor.

Channels, not only customers

A bank’s customers reach it through different channels, and each fails in its own way. A branch journey fails when a form is wrong. A mobile journey fails when a session times out or a consent expires. A USSD journey fails when a menu drops mid-confirmation on a feature phone. A synthetic data set which holds customers and accounts but ignores channels tests the part of a product which rarely breaks. Good specifications therefore describe the channel mix alongside the customer mix, and the fault scenarios which attach to each channel: timeout, maintenance window, expired consent, rate limit, fraud alert and duplicate request.

What this means for organisations in Namibia

A firm should ask any sandbox operator for the attestation behind each data set, the list of products represented, and the distributions used. A firm with no answer to these questions has no basis for its test results.

A bank or a regulator contributing real-world knowledge should review distributions and product lists for realism, without handing over records. Review of the shape is enough, and real data is not required.