Skip to main content

Integration with Databricks

Last updated on by s-krishnamoorthy

SAP Business Data Cloud facilitates seamless harmonization of business data from SAP and non-SAP data from Enterprise Dabricks for richer Analytics and AI use cases. BDC Connect for Databricks enables the bi-directional data sharing of curated data products with enterprise databricks leveraging the industry standard open delta share protocol, enabling efficient AI/ML workloads. SAP Business Data Cloud also allows direct JDBC connectivity with enterprise databricks delta lake at the SAP Datasphere layer, enabling a open data ecosystem integration.

In enterprise hybrid landscapes that span diverse computing platforms and cloud sources, Delta Share–based access to AI-ready data products delivers enhanced flexibility, optimized performance, and seamless interoperability.

Architecture​

image of solution diagram
Copy to clipboard
Solution Diagram Resources
You can download the Solution Diagram as a .drawio file for offline use. Alternatively, you may view and edit the Solution Diagram directly on draw.io.
Please note that any changes made online will need to be saved locally if you wish to keep them.
DownloadEdit

1. BDC Connect : Bi-directional delta share integration with enterprise databricks(NEW)​

  • With the release of BDC Connect for Enterprise Dabricks recently, SAP data products from SAP line of business applications can be shared directly with Enterprise Databricks over governed data access, and discoverable via Unity catalog.
  • Similarly, data from Enterprise Databricks can be shared as data products back to SAP Business Data Cloud catalog via BDC Python SDK.

    • Ref: brownfield integration

2. Integration with Databricks Delta Lake at SAP Datasphere Layer​

Delta Lake is an optimized storage layer that provides the foundation for tables in a lakehouse architecture on Databricks. It brings reliability to data lakes, ensuring ACID (Atomicity, Consistency, Isolation, Durability) transactions, scalable metadata handling, and unifying streaming and batch data processing.

Data from Databricks Delta Lake tables can be federated live into SAP Datasphere virtual remote models using SAP Datasphere's data federation architecture. This integration allows for the seamless augmentation of Databricks data with SAP business data in real-time. The federated data can be incorporated into unified semantic models, enabling efficient and real-time analytics through SAP Analytics Cloud dashboards.

The integration process involves:

  1. Connection Setup: Establishing a secure connection between SAP Datasphere and Databricks Delta Lake using supported connectors (JDBC connectivity using CamelJDBC Adapters) and authentication mechanisms.
  2. Data Federation: Configuring virtual tables in SAP Datasphere that reference the live data in Databricks Delta Lake without physically moving the data.
  3. Model Augmentation: Enhancing the federated data with SAP business data to create comprehensive and unified semantic models.
  4. Real-time Analytics: Utilizing SAP Analytics Cloud to build dashboards and reports that leverage the real-time, federated data for actionable insights.

This approach ensures that data remains consistent and up-to-date, providing a robust foundation for advanced analytics and decision-making processes.

Resources​