The Technology Behind Format Conversion

At first glance, digital document exchange between companies seems primarily to be a transport issue: an invoice, order, or packing slip must be securely transferred from system A to system B. In practice, however, much of the complexity lies not in the transport itself, but in the data format.

An SAP system may generate an invoice as an IDoc; a construction company uses DICO SALES005; a supplier sends UBL 2.1; an international trading partner uses EDIFACT; and another party still submits a PDF or CSV file. Meanwhile, the receiving ERP systems each expect their own data structure.

Format conversion is therefore a critical layer in electronic document exchange. Not by making every system understand all possible formats, but by centrally recognizing, interpreting, validating, and converting documents into the format required by both the sender and the recipient.

A format is more than just XML, PDF, or CSV

When discussing document formats, broad terms like XML and PDF are often used. Technically, that doesn’t mean much.

XML, for example, primarily describes how data can be recorded hierarchically. Two XML files can use completely different structures, field names, and rules.

For example, a UBL invoice contains elements such as Invoice, AccountingSupplierParty, TaxTotal, and LegalMonetaryTotal. An SAP IDoc uses segments and fields according to an entirely different structure. DICO SALES005, in turn, has its own XML schemas, namespaces, and business rules for transaction messages within the construction and engineering sectors.

Even within a single family, there are different versions and profiles. Organizations may encounter, among others:

• OASIS UBL 2.0, 2.1, 2.2, and 2.3
• Peppol BIS Billing 3.0 based on UBL 2.1
• UN/CEFACT Cross Industry Invoice, including CII D16B
• UN/EDIFACT INVOIC, for example D96A or D16B
• SAP IDoc INVOIC01 and INVOIC02
• DICO INSBOU004 and SALES005
• cXML InvoiceDetail
• xCBL
• Factur-X and ZUGFeRD
• XRechnung
• Supplier- and customer-specific XML and JSON formats
• CSV, Excel, and fixed-width files
• PDF, whether or not combined with structured XML data

The .xml extension therefore says little about actual interoperability. Two systems can both “speak” XML and still not understand each other.

EN 16931: First the Same Meaning, Then the Same Syntax

EN 16931 plays a central role for electronic invoices within Europe. An important distinction must be made here: EN 16931 is not an XML format, but at its core a semantic data model for electronic invoices.

The standard describes what business information an electronic invoice contains and what that information means. Examples include invoice number, invoice date, supplier, customer, VAT category, payment terms, invoice line items, and totals.

That same meaning can then be expressed in various technical syntaxes. Within the European standard, the following syntaxes, among others, play a role:

• OASIS UBL 2.1
• UN/CEFACT Cross Industry Invoice D16B
• UN/EDIFACT INVOIC D16B

This distinction between semantics and syntax is essential for format conversion.

Suppose an incoming SAP IDoc contains a field for the supplier’s VAT identification number. During conversion, an XML tag is not simply replaced by another XML tag. First, the meaning of the source field must be determined. Then, it is determined which data element in the target model that information should be mapped to.

The conversion layer thus essentially operates according to the principle:

source field → meaning → target field

This makes it possible to link many different source formats to a single agreed-upon financial data model.

UBL 2.1, Peppol, and Other Implementation Profiles

Even when both parties use UBL, conversion or normalization may still be necessary.

UBL—Universal Business Language— is a comprehensive OASIS standard for electronic business documents. In addition to Invoice and CreditNote, UBL includes structures for Order, OrderResponse, and DespatchAdvice, for example.

Furthermore, there are different versions of UBL. For example, an organization might use UBL 2.3 internally, while a network or recipient requires UBL 2.1.

Peppol BIS Billing 3.0 is a good example of this. For invoices and credit memos, Peppol uses UBL 2.1 and adds additional semantic and business rules on top of it. Peppol BIS Billing is therefore not simply UBL, but a specific usage profile that complies with EN 16931.

A randomly selected, technically valid UBL 2.1 invoice is therefore not automatically a valid Peppol invoice.

A document must, among other things, comply with:

• the XML structure of UBL 2.1;
• the semantic rules of EN 16931;
• the additional rules of Peppol BIS Billing;
• any applicable national or sector-specific rules.

As a result, format conversion may involve, for example:

company-specific UBL 2.3 → canonical invoice model → Peppol BIS Billing 3.0 / UBL 2.1

The difference, therefore, lies not only in XML tags, but primarily in the rules that determine how those tags should be used.

From SAP IDoc, EDIFACT, and DICO to a common model

In large organizations, the diversity is often even greater.

SAP has been using IDocs—Intermediate Documents— for decades to exchange data between SAP and external systems. For invoicing, INVOIC01 and INVOIC02 are among the well-known formats. An IDoc consists of segments with fields that have specific technical and functional meanings within SAP.

An SAP IDoc is neither an EN 16931 invoice nor a UBL document. The relevant segments must therefore be mapped in terms of content to the desired target model.

In large-scale B2B environments, UN/EDIFACT is also still widely used. For example, an INVOIC message may use segments such as:

• UNH for the message header;
• BGM for document information;
• DTM for dates;
• NAD for lots;
• LIN for invoice lines;
• MOA for amounts;
• TAX for tax information.

Within the Dutch construction and engineering sector, DICO plays a similar sector-specific role. DICO includes standardized messages for, among other things, quotes, orders, order confirmations, shipping notifications, and invoices. Within SALES005, XML structures are used for this purpose that are specifically designed to meet the information needs of that sector.

As a result, a supplier can, for example, generate a DICO SALES005 invoice, even if the customer uses UBL internally or prefers to receive the invoice via Peppol.

This can result in conversion chains such as:

DICO SALES005 → canonical model → UBL 2.1 / Peppol BIS Billing

SAP INVOIC02 → canonical model → DICO SALES005

EDIFACT INVOIC D96A → canonical model → UBL 2.1

A central or canonical model prevents the need to build a separate converter for every combination. With ten source formats and ten target formats, a point-to-point approach could otherwise result in dozens of separate mappings.

Conversion also involves interpretation and validation

A good converter does more than just move data. During conversion, differences in data type, structure, encoding, and business logic must also be resolved.

For example, a source format might use NL, NLD, or 528 to refer to the Netherlands. The target format may only allow an ISO 3166-1 alpha-2 code. A local VAT code in the source system may correspond semantically to a specific EN 16931 VAT category, but must not be passed on without a mapping.

The same applies to:

• currency codes;
• units;
• payment methods;
• VAT categories;
• document types;
• identification schemes;
• country codes;
• item and supplier identifiers.

Financial calculations also require attention. An invoice may contain separate values for net line item amount, discount, surcharge, VAT base, VAT amount, subtotal, prepaid amount, rounding, and amount due.

The target model may prescribe different calculation and rounding rules than the source system. As a result, an XML document can be technically correct yet financially invalid.

Validation therefore usually takes place at multiple levels. First, an XML document can be checked against an XSD schema to determine whether the technical structure is correct. Subsequently, content-based and business-related checks can be performed using, for example, Schematron rules.

A conversion process thus typically consists of:

recognize → read → normalize → map → convert → validate → deliver

When data is missing or cannot be converted unambiguously, this must be flagged. A converter must not invent missing financial facts on its own to make a document technically compliant.

One format internally, many formats externally

It is not very appealing for organizations to have to repeatedly adapt their ERP system every time a supplier, customer, industry standard, network, or government agency requires a different format.

A central conversion layer reverses that principle.

For example, the ERP system can always produce and receive the same agreed-upon XML format. The exchange layer translates this externally to UBL 2.1, Peppol BIS Billing, CII D16B, DICO SALES005, SAP IDoc, EDIFACT, cXML, JSON, CSV, or a customer-specific format.

Unstructured documents can also be incorporated into this process. A PDF can be parsed using document recognition and then translated into the same central data model. In the case of hybrid formats such as Factur-X and ZUGFeRD, a human-readable PDF and structured XML data are combined.

This makes format conversion much more than a technical translation function. It becomes a normalization layer between trading partners, networks, and ERP systems.

On the outside, the world can remain complex: UBL 2.1, CII D16B, INVOIC02, DICO SALES005, EDIFACT, PDF, CSV, and future standards can coexist. On the inside, a financial system doesn’t need to be aware of all that complexity.

Ultimately, that is the most important function of format conversion: not teaching every system to understand every format, but ensuring that the same financial meaning is reliably preserved, regardless of the technical language in which the document is received or sent.