Monday 1 July 2024

SAP Datasphere: Replicating Custom CDS Views from S4 HANA to Azure Data Lake with CDC

Introduction


In this article I will share work experiences of using a custom CDS view on Joining BKPF and BSEG table in S4HANA and consuming it in SAP Datasphere for delta enabled replication (Change Data Capture) to Azure Data Lake Storage Gen2.

SAP Datasphere: Replicating Custom CDS Views from S4 HANA to Azure Data Lake with CDC

Create a Custom CDS View: ADT on Eclipse


Go to ABAP Repository Object and Click on the drop down for Core Data Services and then click on Data Definitions:

SAP Datasphere: Replicating Custom CDS Views from S4 HANA to Azure Data Lake with CDC

Provide the CDS view name and description. Note that we have not selected a referenced object, as we will be using two tables

SAP Datasphere: Replicating Custom CDS Views from S4 HANA to Azure Data Lake with CDC

Fill change request and click next, now select a Template view: Define View

SAP Datasphere: Replicating Custom CDS Views from S4 HANA to Azure Data Lake with CDC

Once you click on Finish, CDS view code is generated

SAP Datasphere: Replicating Custom CDS Views from S4 HANA to Azure Data Lake with CDC

Now modify the CDS view code and mention the tables BSEG and BKPF, add the join conditions and mention the columns in select clause.

Below is the code after modifying

SAP Datasphere: Replicating Custom CDS Views from S4 HANA to Azure Data Lake with CDC

Define CDS annotations 


Put in below annotations for "dataExtraction" and "changeDataCapture"

:{
    dataCategory: #CUBE,
    dataExtraction: {
        enabled: true,
        delta.changeDataCapture: {
            mapping:
            [ { table: 'BSEG', 
                role: #MAIN,
                viewElement: ['CompanyCode','Fiscalyear','Documentnumber','Lineitem'],
                tableElement: ['bukrs','gjahr','belnr','buzei']
              },
              { table: 'BKPF', 
                role: #LEFT_OUTER_TO_ONE_JOIN,
                viewElement: ['CompanyCode','Fiscalyear','Documentnumber'],
                tableElement: ['bukrs','gjahr','belnr']
              }
            ]
        }
    }
}
 
Now below is the completed CDS View Code with Data extraction and CDC enable

SAP Datasphere: Replicating Custom CDS Views from S4 HANA to Azure Data Lake with CDC

Save & Activate the View.

Replicate the CDS view in Datasphere (Initial and Delta Mode)


In SAP datasphere go to the Data Builder & Click on new Replication flow.

SAP Datasphere: Replicating Custom CDS Views from S4 HANA to Azure Data Lake with CDC

Ensure Source and Target connection are created and working in Datasphere, Here we have already created the connections.

SAP Datasphere: Replicating Custom CDS Views from S4 HANA to Azure Data Lake with CDC

Now create the replication flow by selecting Custom CDS view from S4 HANA System Connection.

SAP Datasphere: Replicating Custom CDS Views from S4 HANA to Azure Data Lake with CDC

Select CDS from source container and then add the source object i.e. Custom CDS View

SAP Datasphere: Replicating Custom CDS Views from S4 HANA to Azure Data Lake with CDC

SAP Datasphere: Replicating Custom CDS Views from S4 HANA to Azure Data Lake with CDC

Similarly select the Target connection and root folder in Azure Data Lake to place the target files.

Now select the Load Type as Initial and Delta and mention the Delta Load Interval as every 1 hr.

SAP Datasphere: Replicating Custom CDS Views from S4 HANA to Azure Data Lake with CDC

Click on Save and then Deploy replication flow.

Once deployment is complete, start the execution.

When we initiate the replication flow, the process begins with an initial load, followed by subsequent runs based on the delta load frequency to transfer the incremental changes

Monitor the Run:


Here we can see its showing the Initial load operations i.e. 1451 and followed by Delta load operations 110.

Please note replication flow was running for more than a day and automatic Delta records are captured and loaded into ADLS.

SAP Datasphere: Replicating Custom CDS Views from S4 HANA to Azure Data Lake with CDC

SAP Datasphere: Replicating Custom CDS Views from S4 HANA to Azure Data Lake with CDC

Validate the records in Azure Data Lake Storage

We can see in target data lake both Initial and Delta files are generate separately. We can see below initial load records with Operation_Type as L

SAP Datasphere: Replicating Custom CDS Views from S4 HANA to Azure Data Lake with CDC

Below are new delta records populated in separate file with Operation type as ‘A’.

SAP Datasphere: Replicating Custom CDS Views from S4 HANA to Azure Data Lake with CDC

SAP Datasphere: Replicating Custom CDS Views from S4 HANA to Azure Data Lake with CDC

Conclusion

This article is an attempt to showcase how some of the Custom S4 CDS views on multiple tables (extraction and CDC enabled) can be consumed in SAP Datasphere.

No comments:

Post a Comment