Skip to main content
Schedule a demo

Provably complete transformation for complex semi-structured data

EDI, JSON, XML, Avro
for analytics and AI.

The data isn't the problem.The approach is.

So we invented a new one, purpose-built and patented.

“We looked everywhere. Nothing comes close to what DataPancake delivers.”

— C.H. Robinson, a Fortune 200 logistics company
  1. Discover and Shape

    DataPancake deterministically discovers 100% of what's actually in your data, including every variation and what nobody documented. For X12, start from one of our licensed libraries, and the scan confirms what your data actually contains.

  2. Configure and Generate

    You shape and configure what it finds. DataPancake then generates the relational model and the SQL that builds it.

  3. Materialize and Reconcile

    Your Dynamic Tables and views are created in your own Snowflake account, and run without us, then reconciled to the source, proving no data was lost.

Ready in hours, not months.

  • No hand-built models
  • No manual mapping
  • No forcing the data to fit
C.H. Robinson (opens in a new tab)

“We looked everywhere. Nothing comes close to what DataPancake delivers.”

— C.H. Robinson, a Fortune 200 logistics company
  • $1M in savings.

    Combined compute and engineering time savings (Does not include data quality improvement and downstream business impact).

  • 3B rows in 25 minutes.

    One table containing 3 billion rows of complex Kafka messages was scanned in 25 minutes for $600 in Snowflake compute cost to do the initial discovery of the entire schema.

Semi-structured data was built to move data,
not analyze or reason over it.

For decades, we've developed powerful platforms for structured data

to connect, move, store, transform, and analyze information, and they've become remarkably good at it.

For semi-structured data it worked, at a cost.

Semi-structured data is built to be flexible. Structured tables are built to be fixed. So we model the tables first, from documentation or a sample of the data, and map the source into them. Define the structure. Map the fields. Write the code. Maintain it as the data changes.

The more the structure varied, the more time, code, and maintenance it took.

Whatever the documentation missed, or the sample didn't contain, was left behind, with nothing to say it was missing. And when a field changed type, data products that worked one minute broke the next.

We kept improving the tools. We never changed the approach.

We accepted the gaps, assuming downstream checks would catch them. But those checks covered only the fields someone expected.

AI raises the stakes.

It answers just as confidently either way, and it can't see what never arrived.

DataPancake® started with a different question: how do you benefit from both, the flexibility of semi-structured data and everything built for structured data?

That question led us to a new approach.

Discover what's in the source, and let that discovery determine the relational structure. Then reconcile the result to prove what was preserved.

?
It parsed. It loaded. An unknown field or nested array never arrived or one with PII/PHI did, and nothing said so.

Model. Map. Fix. Repeat. And hope no one asks for more data.

Not anymore.Let the data build the model.And say yes to more.

100% of your data,
without hand-built models or mapping.

DataPancake flips the order.

Discover all of it first.

The difference goes beyond saving time. It changes what's captured, what technical debt you take on, what you can verify, and ultimately, how much you can trust the data powering your business and AI.

A complete, recursive scan

DataPancake reads every record and descends every level

Illustrative example.

Approach Comparison

CurrentHand-coded pipelines, integration and transformation tools, schema inference, and AI-generated code

vs

DataPancakeDiscover, configure, generate

What comes first

Current approaches

The model comes first, built from documentation or a sample.Teams design it and map the source into it by hand. Tools that generate it automatically usually build it from a sample, because scanning everything takes too long, and rebuild it when the source changes. Either way, the model is never finished: each new field, type variation, or downstream request for data nobody modeled means extending it again.

The DataPancake approach

The data comes first, all of it.Discover 100% of the source, deterministically, today and as it changes. A full scan is efficient enough to be the default, not the exception: at one customer, 3 billion rows were scanned in 25 minutes for about $600 in Snowflake compute. Shape and configure the output once you know what's actually there. Every field is delivered from day one, so downstream teams have nothing new to request.
What gets captured

Current approaches

Without looking at every record, there's no way to know every variation.A single document can hold a nested array carrying JSON inside a string, or an XML element that repeats where nobody expected it. Whatever the model didn't anticipate can be missed, forced into the wrong type, or separated from the record it belongs to.

The DataPancake approach

Every attribute, and every type it arrives as, in its own correctly typed column.Nested arrays at any depth become child tables linked to their parents. JSON hidden inside strings is discovered and structured like everything else. And shaping rules let you reshape what was found before anything is generated.
What happens when the data changes

Current approaches

Missing fields and changed types usually surface downstream, when someone notices a report or a data product is wrong and reports it.Then comes inspection, diagnosis, engineering, and testing for each fix, which can take a two-week sprint per issue, while the data stays wrong.

The DataPancake approach

Changes are detected and alerted at the source, before anyone downstream finds them.Every update is additive: new fields and new types arrive as new Immutably Typed Derived Columns (ITDC), and existing columns stay exactly as they were, so nothing downstream breaks. You choose whether code regenerates and deploys automatically, or waits while someone reviews the metadata first. No new technical debt either way.
What you can prove

Current approaches

The usual evidence is row counts, a few checks written for the fields someone expected, and the absence of pipeline errors.That proves the pipeline ran, not that everything in the source reached the tables.

The DataPancake approach

A full reconciliation of the source against the materialized tables, with metrics across every validation lane, including foreign key uniqueness and materialization coverage.Every column traces back to its source path, so you can show what made it through and prove nothing was left behind.
What it takes to build and maintain over time

Current approaches

A large initial build for every source.EDI often has to be converted to XML or JSON before it can become tables, which adds a step and more compute. Security policies are a separate project, often for a separate team. After launch, engineers have to stay on hand to repair or extend the pipeline as the data changes, without breaking any downstream consumer that depends on it. And if AI writes the code, completeness is uncertain from the start.

The DataPancake approach

A data source can be production-ready in a day, at a price known before you build.Security is configured in the same metadata, not a separate project. With the MCP server, an agent can make metadata changes for you, such as aliases and virtual attributes. EDI goes straight from raw to tables, with no conversion to XML or JSON first, so it takes less compute. And there are no mappings to maintain.

The model used to be the input. Now it's the output.

A state energy regulator waited six years for an affordable way to analyze its energy output data. With DataPancake, it finally has one.

How it works

DataPancake scans 100% of your data, with a standards library such as X12 834 or without one, so the whole structure is found. You shape and configure what it finds; it materializes your data into tables, or Dynamic Tables that refresh continuously as new data arrives, and proves them against the source.

  1. Step 01Data LoaderInventory Source Data with Optional Load/Stage
  2. Step 02OptionalMetadata LibraryApply one of our pre-configured standards libraries, such as X12 834, prior to scanning.
  3. Step 03Schema DiscoveryDiscover every attribute path, nested array regardless of depth, stringified JSON, and type variation.
  4. Step 04Schema ShapingIteratively refine the discovered schema, such as converting objects to arrays, or consolidating the schema due to key uniqueness.
  5. Step 05Pipeline DesignConfigure schema metadata including security policy integration, foreign keys, aliases, transformation expressions, virtual attributes and much more.
  6. Step 06Code GenerationGenerate SQL DDL/DML for Dynamic Tables and views with security policy integration.
  7. Step 07DeployDeploy the generated objects through our automated deployment system or use a git integration with your own CI/CD process.
  8. Step 08ReconcileExecute DML reconciliation SQL to produce reconciliation metrics to verify multiple validation lanes including materialization coverage.
  9. Step 09Data Dictionary (LLM Assisted)Generate definitions, synthetic sample values, and synonyms for each dataset, array (child table), and attribute (field).
  10. Step 10Semantic ModelGenerate Cortex-ready YAML or Semantic Views for Natural Language Queries in Snowflake.
Continuous Control LayerSchema Drift MonitoringMonitor new paths, changed types, and structural variation as source data evolves.

Steps 3 and 4 repeat as an iterative cycle, and so do steps 5 to 8. Schema Drift Monitoring leads back to step 3, a continuing cycle.

A deterministic component for agentic AI

Agentic AI speed.
Deterministic certainty.
The best of both.

Through DataPancake's MCP server, an AI agent such as Snowflake CoCo can run all of DataPancake for you: admin setup, data loading, discovery, configuration, generation, deployment, and drift alerts. DataPancake still does the deterministic work, so every result is the same, and reconciled against the source. Now in private preview.

Illustrative example, condensed from a DataPancake demo in CoCo.

Demo to Proof of value (POV), See it on your own data, in your own Snowflake account, to Production-readyAll in one day.footnote 1

It seems like magic. It's actually deterministic, provable, and patented.

1For one dataset, with your data already in Snowflake or cloud storage.

See everything you have.
Then see what you couldn't before.

Answer today's questions with completeness you can verify. Then connect the dots, spot the patterns, and ask the questions that complete data makes possible.

HealthcareCare that already happened can still show up as an open gap.

Find clinical evidence your quality reports couldn't see.

  1. The momentA patient receives a screening, and the health plan receives the claim and clinical record documenting it.
  2. What happens todayOnce it arrives, it still has to be mapped and prepared, which can add months before it reaches quality reporting. Details the mappings weren't designed for, like an extension or a field in an unexpected place, may never arrive at all. Coordinators work from outdated or incomplete reports and contact patients who've already received care.
  3. Why it mattersQuality measures affect reimbursement and incentives. Care that never reaches reporting isn't reflected in how the plan is evaluated.
  4. What's now possibleEvery detail in the claim and clinical record is discovered as it arrives and reaches reporting once it's configured, not coded, and reconciliation confirms nothing was left out. Coordinators see who's already been cared for and reach the patients who still need it.
State agenciesThe data is yours. The answers shouldn't take months.

Answer the questions you're accountable for.

  1. The momentA legislator asks how much the state paid its managed care plans last year for members whose coverage should have ended.
  2. What happens todayThe transactions that answer the question exist, but they sit in vendor-operated systems, and existing reports often weren't built for that question. The agency waits weeks for an answer it can't fully verify.
  3. Why it mattersAgencies are accountable for payment accuracy even when vendors and plans run the systems. Medicaid improper payments reached $37 billion in fiscal year 2025, most of them because the documentation behind the payment was missing or insufficient, and 18 federal audits since 2016 have identified roughly $289 million in capitation payments made for deceased enrollees.
  4. What's now possibleOnce the agency's enrollment, payment, and encounter data is in complete, reconciled tables in its own Snowflake account, an analyst can answer questions like this in days, not months, and reconciliation shows the tables match what the plans and the state actually sent. The same works for any agency that collects nested reports, from energy filings to federal child welfare reporting.
InsuranceA new partner shouldn't mean new technical debt.

Bring on the next partner without a new mapping project.

  1. The momentAn employer group changes its enrollment file, or an MGA adds a coverage field to its ACORD feed.
  2. What happens todayNothing errors. The new field falls outside the mapping and waits in a backlog for a partner-specific update, while underwriting and claims work without data the partner already sent. Each partner's mapping becomes one more thing to maintain.
  3. Why it mattersMore than half of insurance executives surveyed say most of their IT budget goes to keeping existing systems running, and regulators and auditors ask insurers to prove nothing was lost in transformation.
  4. What's now possibleEvery partner's variation is discovered across 100% of the data, including fields nobody documented, and flagged when it arrives. It reaches the tables once it's configured, not coded, and reconciliation shows auditors nothing was dropped along the way.
BankingWhat your monitoring never saw is still your responsibility.

Your archive has the original fields. Can you prove your systems got them too?

  1. The momentA bank examiner asks a simple question: can you demonstrate that the parties in your cross-border payment messages, including the ultimate debtor and ultimate creditor when provided, made it into the systems monitoring those transactions?
  2. What happens todayThe bank retains the original ISO 20022 messages, but its payment and monitoring systems work from extracts built around predefined fields. Optional fields like the ultimate parties and structured addresses arrive so inconsistently that Swift postponed its 2026 structured address deadline, and when a message carries details the mappings didn't anticipate, they're left out. The payment may have processed successfully, even though the downstream tables don't contain everything the bank received. Answering the examiner means returning to archived XML and reconstructing what arrived.
  3. Why it mattersRetaining the original message is one thing. Demonstrating what reached the systems responsible for monitoring is another. New York requires the banks and other institutions it regulates to validate that complete and accurate data flows into transaction monitoring, with an annual certification signed by the board or a senior officer, and only 2 of the world's 31 largest banks fully meet the global standard for risk data.
  4. What's now possibleDataPancake discovers every structure and variation across the bank's retained payment messages, generates relational tables from what it finds in the bank's own Snowflake account, and reconciles them against the retained messages. That gives the bank a complete, verifiable reference. Comparing it with downstream records shows missing fields, discrepancies and dropped messages, and supports lookbacks across years of history, without changing production payment systems.
ManufacturingWithout lot details, containment covers more than the defect.

Trace a defect to the shipments it actually touched.

  1. The momentA quality problem surfaces at an OEM, and the supplier needs to know which lots and shipments are affected.
  2. What happens todayThe supplier's ship notices go out in each OEM's own format (X12, EDIFACT, or VDA), with lot and serial details nested several levels deep. When those records aren't fully captured in the supplier's own tables, the link between lots and shipments is missing, so containment has to cover everything that might be affected.
  3. Why it mattersUS manufacturers paid $29.2 billion in warranty claims in 2024, and every shipment pulled unnecessarily adds to the cost of a defect.
  4. What's now possibleEvery OEM's version of each document is discovered, including unusual packing hierarchies and plant-level variations, and brought into one consistent set of tables. The lot and serial details are already there, so the team limits containment to the shipments the defect actually touched.
Transportation and logisticsA charge is hard to dispute when the events behind it are missing.

Dispute a charge with the events that prove it.

  1. The momentA carrier bills detention on a freight invoice, saying the truck waited hours at the dock.
  2. What happens todayThe arrival and departure events come in as EDI status messages that vary by carrier, then get reshaped into internal event streams such as Kafka. A carrier-specific status code that doesn't fit the mapping can be dropped at either step, nothing flags it, and the charge gets paid because the evidence isn't there.
  3. Why it mattersBilling disputes, freight recoveries, and on-time performance depend on what actually happened, and charges on US trucking invoices have to be contested within 180 days.
  4. What's now possibleBoth the raw EDI and the event streams built from it become complete tables, with every status code, charge code, and accessorial actually sent, so the two can be compared field by field. The audit team checks the charge against what actually happened before the window to contest it closes.
Retail and consumer goodsDeductions get written off when the evidence surfaces too late.

Win back a deduction while the dispute window is open.

  1. The momentA retailer short-pays an invoice and sends a remittance with a deduction reason code.
  2. What happens todayShip notices and invoices go out, and remittances come back, as X12 EDI in each retailer's own variation, with the reason code several levels deep in the remittance. When it falls outside the mapping, the deduction shows up unexplained, and matching it to the shipment behind it takes weeks.
  3. Why it mattersDeductions and chargebacks cost suppliers revenue, and retailers set deadlines for disputing them.
  4. What's now possibleEach retailer's version of every ship notice, invoice, remittance, and adjustment is discovered, down to the nested reason codes, and brought into one consistent set of tables. Finance sees the reason code beside the ship notice and invoice behind it and disputes it while the dispute window is still open.

These are seven of many.

One question, three EDI feeds, one answer.

How much did we pay our plans for members whose coverage should have ended?

Illustrative example: enrollment, payment, and encounter tables, each reconciled to its EDI source, then joined.

Use your platforms and tools
for what they do best.

Gateways and API services connect your trading partners and applications. Integration platforms and data loading services move the data. Transformation tools shape it once it's in tables. Each does its job well, but most of their transformation capabilities were designed to work best with structured data. For semi-structured data, someone still has to model and map it first and then maintain it. DataPancake removes that step and delivers complete, correctly typed tables from your EDI, JSON, XML, and Avro data. That's the foundation your existing platforms and tools need to build trusted data products.

Iceberg fits the same pattern. VARIANT and shredding make JSON (not EDI, Avro, or XML) data faster to store and query, but someone still has to find the fields and write the SQL to turn it into tables. DataPancake connects to the VARIANT column directly, finds every field, and builds tables you can prove are complete.

EDIJSONXMLAvro

  1. 01 · Receive, Verify, LoadLoading/StagingData lands in native Snowflake tables, made available as Iceberg tables, or moved to cloud storageSemi-structured, as it lands
  2. 02 · Inventory, Materialize, and ReconcileDataPancakeThe relational data foundationComplete, correctly typed tables
  3. 03 · Build Data ProductsTransformation toolsBuild downstream data products on a reconciled foundation

Trusted data products

At one enterprise customer, a critical operational function was running on a pipeline that had been quietly missing
part of its data. No errors, no alerts.
With DataPancake, it now runs on all of it.

Where are you starting from?

  • Data sitting on the shelf

    Data you own but can't use, because refining it costs too much or takes too long.

  • Pipelines already in production

    Existing pipelines that would benefit from an independent check that nothing is missing.

  • The architecture is changing anyway

    Existing approaches that no longer scale, a migration to Snowflake, or a move to Iceberg, where the semi-structured layer gets rebuilt regardless.

Deterministic for structure.
AI for meaning.

  • Can't AI just write the code?AI is probabilistic and works from what it's shown, so it can't report what it never saw, and it can give a different answer each time. DataPancake's discovery is deterministic: it reads every record, and the same data always produces the same structure. At one enterprise customer, teams tried AI coding tools for a year; nothing except DataPancake produced complete results.
  • Where AI belongs.Once the structure is proven complete, AI can recommend configuration, such as clearer column names and transformation expressions, for your team to accept or change. Review every AI recommendation in Excel before anything changes, then pick up the conversation with your agent where you left off. It also helps your governance team build a data dictionary, so more business users can create semantic models on top of it. AI also generates synonyms and synthetic sample values, so nothing is copied from your data.
  • A deterministic component for agentic AI.It's why an agent can run DataPancake for you: AI can propose the configuration, and DataPancake generates the same tables from it every time, proven against the source.
Illustrative example.

“As we moved from Microsoft SQL Server to Snowflake, our biggest challenge was rebuilding the pipelines that continuously turned ACORD XML from multiple carriers into relational tables.

DataPancake was the one platform we found that could do it and give us relational tables we could confidently use.

With 1.5 engineers in a few weeks, we used DataPancake to generate thousands of lines of production-ready code that would have taken a full team months to write by hand.

Without it, this migration would not have been economically feasible or completed on time.”

— Jon Edwards, VP Carrier Data Acquisition and Integration, Quility

Secure by architecture. Governed by design.

  • Your data never moves.DataPancake is a Snowflake Native App that runs inside your own Snowflake account, under your access controls. No egress, no external processing, no vendor copy. It can also be installed in Snowflake's government cloud.
  • Security you can audit.Masking and row access you configure, down to nested and rare fields. Policies are set as metadata, attribute by attribute, and generated into secure views. Every policy is documented, reviewable, and reapplied whenever the code is regenerated. For regulated data, alert and review holds new attributes until their protection is set.

DataPancake has passed information security reviews at state government agencies, healthcare organizations, financial institutions, and technology companies, including organizations held to SOC 2 Type II, HIPAA, and other standards.

See how security works

TDAA provider account

Application package

Code security scan by Snowflake, no reported vulnerabilities

Snowflake Marketplace

Private listing

Approved by Snowflake

Installed by you

From the reference architecture on docs.datapancake.com.
View the full diagram

DataPancake reference architecture: data security

DataPancake's reference architecture for data security. TDAA's provider account holds the DataPancake application package, code-scanned by Snowflake with no reported CVEs. It is offered as a private listing on Snowflake Marketplace, approved by Snowflake and installed by the customer. Inside the customer's Snowflake account: the DataPancake Native App, a Streamlit app in Python with no external API calls; the customer grants select and reference privileges on the objects to scan; DataPancake creates tasks that start scans on the customer's virtual warehouses, calling Snowpark Python procedures in Snowflake's secure sandbox, and logs to the app database and events table; generated metadata obfuscates sample values for discovered string attributes; the customer manages the security policies, row access and column masking, which DataPancake's security policy metadata applies to the generated views; and the customer deploys the generated objects, relational Dynamic Tables, policy-infused views and semantic models, with their security policies.

You own all of it.
Even the proof.

  • Ordinary Snowflake objects.Tasks, Tables, Dynamic Tables, and secure views, in your own account. Any Snowflake engineer can read them, change them, and run them. No black box.
  • They run without us.Once deployed, your tables keep refreshing and your reconciliation keeps running without DataPancake. You come back to DataPancake to discover new structure, change configuration, and regenerate. If you ever stop using it, nothing you deployed stops working.
  • A price you know before you build.Measure your data's structural complexity before the build tools are turned on, or price by named dataset or library. The price is fixed for the year and never based on data volume, rows, or executions. Snowflake compute is still yours, and with incremental refresh it may even go down, because discovery and refresh are optimized and enrichment happens in a single pass.

See pricing and total cost of ownership

Illustrative example.

What could your analytics and AI do with data you can prove is complete?

Ready to try a new approach?

Demo to proof of value to production-ready in a day.