Data Products in SAP Lakehouse
Data Products is the fundamental building block of SAP Business Data Cloud and it serves as a standardized and efficient way to share and consume data across applications and domains. They enable analytic scenarios, AI applications, and facilitate data integration while being optimized for intensive reads. Managed with a product mindset, they are supported by high-quality metadata and governed by decentralized ownership principles.
Purpose of Data Products
Customers currently face challenges when accessing and using data for various tasks such as building new applications, integrating applications, creating analytical models, designing dashboards, and training AI applications. The process of finding relevant data is difficult due to the lack of consistency and clarity in the APIs that expose data and metadata. The metadata often only describes technical aspects, making it hard for business users to utilize, and some necessary data isn't available via APIs. To improve this, the consistent metadata exposed by Data Products will allow for a landscape-specific Data Product to be directly exposed in every customer landscape. This will be facilitated by the catalog, and will promote ODM compliant Data Products, making it easier to combine data from different applications, and automate scenarios like validation, discovery, and integration.
In SAP, a Data Product is a dataset exposed for consumption outside the boundaries of the producing application via APIs and described by high quality metadata that can be accessed through the Data Product Catalog.
- It is a business dataset and consists of one or more business object entities, related objects, analytical data etc
- It is consumable via APIs or events. Supported API types are SQL (incl. SQL interface views), Event, REST, (oData)
- It is described with metadata that is of high quality and provided via Open Resource Discovery (ORD), following ORD schema for Data Product
- It is discoverable via the Data Product Directory, a service of UCL that aggregates metadata of all Data Products to make them discoverable in a landscape. Within the SAP Lakehouse, Data Products are additionally governed and shared through the SAP Enterprise Catalog powered by Apache Polaris.
Key Characteristic of Data Product
-
Efficient Data Sharing Across Domains: Data Products facilitate replication-free integration and consumption by SAP, customer, and partner applications via APIs and Events.
-
Efficient Data Storage: Data Products are processed and curated in a modern object-store based data lake architecture with highly scalable Spark processing.
-
High-Quality Metadata: Metadata includes business semantics and is accessible through directories like the Data Product Directory and ORD Aggregators, enhancing discoverability.
-
Optimized for Analytics: Data sets are curated for intensive reads and consumed in a read-only fashion, ensuring efficiency and reliability.
-
Supporting Analytical and AI Applications: Data Products provide foundational data for dashboards, analytical models, and AI applications.
-
Decentralized Ownership: Inspired by Data Mesh principles, Data Products are owned and managed by domain experts or teams responsible for operational data.
-
Lifecycle Management: Data Products undergo needs analysis, design, and delivery phases, ensuring their accuracy and relevance.
Key Attributes of Data Products
| Attribute | Description |
|---|---|
| Type | Defines the type (primary, derived). |
| Category | Categorizes the data set (e.g., business-object, analytical). |
| Visibility | Specifies exposure (public, internal, private). |
| Input Ports | Integration dependencies describing data inputs for lineage purposes. |
| Output Ports | APIs or Events through which the data can be accessed. |
| Integration Dependencies | Relationships with external resources. |
| Responsible | Organization/team responsible for lifecycle management. |
Types of Data Products
Within the SAP Lakehouse, data products span four governed categories, where each tier builds on the ones beneath it.
- Primary Data Products (SAP): Foundation Services produces these automatically from SAP Line of Business applications, written in UniForm and registered in Open Resource Discovery.
- Primary Data Products (Non-SAP): Dremio federation publishes non-SAP data in place, without duplication, as custom Primary Data Products.
- Unified & 360 Data Products: Reltio resolves source master data into Golden Master records and cross-system key maps, which the Lakehouse joins into unified views across sources.
- Derived Data Products: Engines combine any of the preceding categories into use-case-specific outputs, such as a weighted sales pipeline across SAP S/4HANA and Salesforce.
Primary Data Products expose data directly from source applications and are not based on other data products, representing the original data from a source system. The SAP variant is generated by Foundation Services from Line of Business applications. The non-SAP variant is federated through Dremio, which publishes external data in place without duplication.
Unified & 360 Data Products resolve master data across systems. Reltio consolidates source records into Golden Master records and cross-system key maps, and the Lakehouse joins these into unified views that span multiple sources.
Derived Data Products are curated by SAP from one or more existing data sets. They combine any of the preceding categories into use-case-specific outputs, delivering value-add built on other data products or APIs.
SAP Roadmaps - Business Capability -Data Product Management
Technical Aspects of Data Products
Data Products move through four stages on a single governed surface: production, storage, sharing and consumption.
Production
- Foundation Services (SAP): Produces Primary Data Products automatically from SAP Line of Business applications, written in UniForm and registered in Open Resource Discovery.
- Dremio (Non-SAP and Derived): Publishes non-SAP data in place as custom Primary Data Products without duplication, and derives gold-tier Derived Data Products through reflections.
- Reltio (Unified): Resolves source master data into Golden Master records and cross-system key maps, written back into the silver tier as Unified Data Products.
Storage
The storage layer writes a single physical copy of data once and serves it to many engines. Apache Iceberg is the primary format for cross-engine interoperability.
- Open Table Format: Data lands as Parquet and is compatible with Apache Iceberg tables. UniForm projects that single physical copy as Apache Iceberg, so Iceberg-compatible consumers read it natively without conversion or duplication.
- Medallion Architecture: The federated object store organizes data across Bronze, Silver and Gold tiers, with processing handled by the Spark framework.
Sharing and Governance
- Apache Iceberg with Apache Polaris: The primary sharing path in the SAP Lakehouse. Data Products are read zero-copy in place by Iceberg-compatible engines through the Enterprise Catalog Iceberg REST interface, with Apache Polaris authorizing catalog, namespace and table access.
- Open Resource Discovery (ORD): Enables decentralized discoverability of Data Products and their metadata across the landscape.
Consumption
Consumers reach Data Products through two entry points. Enterprise Discovery tells a consumer that a product exists and what it means. The Enterprise Catalog governs and serves the read.
- Enterprise Catalog (Apache Polaris): The Iceberg REST catalog authorizes catalog, namespace and table access and issues path-scoped credentials, so Iceberg-compatible engines read the governed Data Products zero-copy in place.
- Iceberg REST Engines: External engines such as Snowflake, BigQuery and Microsoft Fabric read Data Products zero-copy in place through the Enterprise Catalog Iceberg REST interface for AI, machine learning and analytics workloads.
- Dremio: Serves federated SQL over Arrow Flight SQL and ADBC for queries and joins across sources, and is the query path for BI tools such as Power BI, Tableau and Looker. Dremio reads the same governed Data Products through the Enterprise Catalog rather than sitting in front of it.
- SAP HANA Cloud: Consumes Data Products via SQL-on-Files and Federation over the Apache Iceberg representation, enabling workloads such as CDS views, calculation views and CAP applications to leverage governed BDC data without replication.
- SAP Analytics Cloud: Consumes Data Products for visualization, reporting and planning scenarios.
Implementation
The customer onboarding sequence provisions and activates Data Products within the landscape.
- Provisioning: Provision the service through SAP for Me.
- Source Connection: Connect the source applications.
- Activation: Activate the required Data Products.
- Content Installation: Install Intelligent Content.
Workflow - Data Product Discovery, Generation, Sharing and Consumption
Discovery and Generation: Consumers query ORD Aggregators to find Data Products and associated metadata. This diagram shows how Data Product definitions are discovered and generated. Source Lines of Business (LoB) provide Data Product definitions, which are read and listed by the SAP Business Data Cloud catalog. These definitions generate Apache Iceberg tables registered in the Enterprise Catalog, making Data Products available for consumption.
Data Sharing View: This diagram illustrates how Data Products are shared through the primary path. The Data Provider manages Iceberg tables in the BDC Federated Object Store, which the Enterprise Catalog exposes through the Iceberg REST interface. Apache Polaris authorizes catalog, namespace and table access and issues path-scoped credentials for zero-copy reads.
Consumption Workflow: This diagram illustrates how consumers discover, integrate, and access data products.
Data Product Extensibility
SAP Data Products also support customer extensions within any SAP Sources & locally defined extension fields to build Data Product Customer Extensions. This requires activation of data package directly to enable custom modeling directly. This is one of the simple extensibility option for adapting data products to enable custom modeling.
- SAP Data Products also support customer extensions within any SAP Sources & locally defined extension fields.
- Customer field extensions are supported on a “pass through basis” automatically added to the SAP Data Product and data processing pipeline
- Govern SAP data product + customer fields access in the data catalog and execute data product installation in SAP Datasphere.
- Create SAP Datasphere models & SAP Analytics Cloud stories for consumption.
Conclusion
Data Products in SAP Business Data Cloud streamline data sharing, integration, and analytics across domains. Supported by high-quality metadata and decentralized ownership principles, they are optimized for intensive reads and managed with a product mindset. Foundation Services ensure seamless discoverability and consumption, empowering businesses to make data-driven decisions efficiently.