L flag alongside I and U when
determining which records are updated in the raw dataset. This is to account
for situations where the pipeline is somehow re-configured after data load,
but the CDC dataset can still be reused to avoid re-processing existing data.StockInHand views (ECC and S/4) for MATNR and
WERKS have been aligned with other views to be MaterialNumber_MATNR
and Plant_WERKS. Previous column names ArticleNumber_MATNR and
Site_WERKS still exist for compatibility reasons, but will be removed in
a future release. Customers are advised to change their upstream consumption assets accordingly.SalesOrders_V2 views (ECC and S/4) for ERDAT and
ERZET have been aligned to new names SalesDocumentCreationDate_ERDAT,
SalesOrderItemCreationDate_ERDAT, SalesOrderCreationTime_ERZET and
SalesDocumentItemCreationTime_ERZET to account for the correct
granularity of their source table (either VBAP or VBAK) . Previous
column names CreationDate_ERDAT and CreationTime_ERZET still exist for
compatibility reasons, but will eventually be removed in a future release.
Customers are advised to change their upstream consumption assets
accordingly. Also, Sales Order Item level calendar dimensions are now added
by default.Deliveries view for VGBEL, VGPOS, and XBLNR are
updated to InternalReferenceDocumentNumber_VGBEL,
InternalReferenceDocumentItem_VGPOS ExternalReferenceDocumentNumber_XBLNR
for more clarity. Previous column names SalesOrderNumber_VGBEL,
SalesOrderItem_VGPOS, and ReferenceDocumentNumber_XBLNR still exist for
compatibility reasons, but will eventually be removed in a future release.
Customers are advised to change their upstream consumption assets
accordingly. AccountingDocuments view as well as the downstream
AccountingDocumentsReceivables views, DoubtfulReceivables, and
DaysInArrear metrics are now positive instead of negative to align with
official SAP guidelines.SalesFulfillment and SalesFulfillment_PerOrder views are updated
to use SalesOrders_V2 instead of SalesOrders view as their upstream
source for both ECC and S/4. The view signatures are unchanged.Qty field data type in StockInHand views (ECC and S/4) has been
changed from STRING to NUMERIC.PurchaseDocuments_Flow
views (ECC and S/4) now align with the logic implemented in all other SAP
reporting views.JOIN condition.LEFT JOIN in the Unit of Measure Function and View
Utility code to avoid possible duplicate rows.ORDER BY clauses in some views
to improve performance.Opportunities).GLDocumentsHdr and RegionsMD are now removed as
they are no longer relevant.gcloud builds submit command for Cortex Framework deployments, the _BUILD_ACCOUNT parameter is now required. If you are updating an existing deployment, ensure you provide the appropriate build account in the following format: _BUILD_ACCOUNT='projects/SOURCE_PROJECT/serviceAccounts/SERVICE_ACCOUNT@SOURCE_PROJECT.iam.gserviceaccount.com'.VendorsMDInventoryByPlant.AccountingDocumentsReceivable.LeadsCaptureConversions Salesforce (SFDC) reporting table: LeadFirstResponeDatestamp corrected to LeadFirstResponseDatestamp.New Data Sources
/src/common/py_libs directory.PurchaseDocuments_Flow view.The following standalone accelerators and samples are removed from Cortex Framework:
Updated the Cortex libraries for Meta to use v21.0 of the Meta Marketing API.
]]>CampaignDailyAggByUserCountry view is now removed.CampaignDailyAgg view.hier_reader code has been fully deprecated. Relevant SAMPLE scripts have been updated to use the new hierarchy reader DAG output tables.currency_conversion, currency_decimal, and fiscal_date_dim) instead.GitHub Submodules
Submodules are officially removed. Workloads have been migrated as follows:
Note: As announced in release 5.4, workload specific repositories will continue to receive code updates for at least one more release.
New Data Sources
Marketing: YouTube (with DV360). Get comprehensive overview of your campaign spend, performance and audience targeting with actionable insights for paid media campaigns on YouTube purchased in DV360 platform. Access the Looker Block for YouTube (with DV360) with sample dashboards for faster data analysis and exploration.
Marketing: Google Analytics 4 (GA4). Understand website engagement and purchases using page views, time spent, call to action ratios, average purchases, lifetime value over time, traffic volumes, and success rates for defined conversion.
Oracle EBS: Measure and understand Order to Cash operational metrics around sales performance, order status, order fulfillment, invoicing, and receivables with new BigQuery integration templates and data models. Access the Looker Block for Oracle EBS with sample dashboards for further analytics.
New Cloud Build features support
Added support for the following Cloud Build features:
SAP
Currency_Conversion and Currency_Decimal functions will be deprecated in the next version. Join the views currency_conversion and currency_decimal instead.hier_reader local k9, which will be removed in the next version. For more information, see Cortex Framework: integration with SAP.SalesStatus_Items and S/4 Billing logic.Inventory: Moved the following views and functions from src/SAP/SAP_REPORTING/local_k9/inventory_snapshots to src/SAP/SAP_REPORTING/{SAP_FLAVOR}; and renamed them to make the structure more logical and succinct. Also added notes and instructions on this module in the Cortex Framework: integration with SAP . There isn't any content or logic change.
00_stock_characteristics_config to StockCharacteristicsConfig.00_slow_moving_threshold to SlowMovingThreshold.0_{weekly|monthly}_inventory_aggregation.sql to Aggregate{Weekly|Monthly}Inventory.stock_weekly_snapshots_inventory_aggregation_update to UpdateMonthlyInventoryAggregation.01_stock_{weekly|monthly}_snapshots to Update{Weekly|Monthly}StockSnapshots.Data Mesh
New documentation page
Misc Technical Notes
africa-south1 and europe-west10.airflow.operators.python_operator and airflow.contrib.operators.bigquery_operator operators.Known issues and limitations
This release changes the Cloud Build default service account behavior, and now it has one of the two types of service accounts by default:
project-number@cloudbuild.gserviceaccount.com).project-number-compute@developer.gserviceaccount.com). Consequently, the Quickstart demo deployment is updated to use either default service account. For more information, see Cloud Build Service Account Change.
]]>src/marketing/src/SOURCE/src/table_schema to src/marketing/src/SOURCE/config/table_schema matching other marketing sources.Google Ads
Keywords view to KeywordStats.CampaignStats, CampaignStatsByUserCountry, GeoTargetConstant), and new experimental views (CampaignDailyAgg, CampaignDailyAggByUserCountry).login_customer_id no longer needs to be explicitly specified in the Secret Manager configuration. Meta
src/marketing/src/Meta/src/raw/pipelines/config.ini.K9
INTEGER type from STRING based on source definition. Sustainability views are now GA.src/SFDC/src/raw_dag_generator/dependencies/config.ini. Default value is UTF-8.Repository READMEs
Plan migration
Marketing
SAP Finance
ProfitAndLoss) that shows company's revenues, expenses, and profits. Also demonstrating Financial Statement Version (FSV), cost-center, and profit-center hierarchy.financial_statements code, which significantly decreases execution time and resources required for FinancialStatement and BalanceSheet views.K9 - Sustainability
Other changes
DAG ApacheBeam[gcp] version requirements to 2.53.0. New test harness regions added: me-central1 and me-west1. Addressed a variety of vulnerabilities in Python library versions.C006 balances in the SAP Finance test harness data.SAP_REPORTING/common/materializer/generate_build_files.py).external_dags for SAP was converted into local K9.AccountsPayable for S4 to match invoice based on reference.Known issues and limitations
tmp) in the target bucket for DAGS. Remove it manually. There is no impact in the execution.InventoryByPlant and InventoryKeyMetrics for ECC and S4 to reduce memory processing requirements. Updated partitions and clusters for StockMonthlySnapshots and StockWeeklySnapshot.external_dag directory. These DAGs no longer search for SQL files from bq_data_replication directory. If you have modified the SQLs for them, follow these steps to migrate:
gs://<your_airflow_bucket>/data/bq_data_replication.gs://<your_deployment_bucket>/dags/sap/reporting/<dag_name>./dags directory from the deployment bucket into your Airflow bucket. Then, wait for Airflow to refresh, and unpause the DAGs in Airflow.gs://<your_airflow_bucket>/data/bq_data_replication.cortex-support@google.com.Mando Checker is now executed at the beginning of the validation process. The deployer attempts to create, write and delete temporary files and datasets in the targets to validate permissions.testData is set to True.turbo mode is set to False.CREATE statement. Check the new default configurations as this attempts to replace some views with tables. config.json. Legacy Dotenv files have been removed.SAP
Salesforce
SFDC/config/ingestion_settings.yaml. The configuration in cdc_placeholder_settings.yaml now takes care of the dependencies in the reporting views.Known issues and limitations
Experimental mode while we gather early feedback from customers on usage. SAP_CDC/external_dag/inventory_snapshots): Initializer and periodic update for weekly and monthly inventory snapshots and aggregations, slow moving threshold, and stock characteristic configuration. Review for CORTEX-CUSTOMER comments for potential updates needed for custom thresholds and material movements. These scripts are in PREVIEW and would be moved into another structure in deployment in the next major release. These scripts fill the base tables and need to be scheduled for the reporting views to function properly.AccountsPayableOverview, DaysPayableOutstanding, InventoryByPlant, InventoryKeyMetrics, MaterialLedger, MaterialMovementTypesMD
MaterialsBatchMD, POScheduleLine, PurchaseDocumentsHistory, SalesOrderStatus, SlowMovingThreshold, StockCharacteristicsConfig, StockInHand, StockMonthlySnapshots, and StockWeeklySnapshots.VendorLeadTimeOverview and VendorPerformanceOverview. These views show the reporting logic used in Looker in case you want to replicate them in another tool or a microservice. These views aren't deployed by default.config/config.json now produce the same result as many times as currencies are set as targets. Check for CORTEX-CUSTOMER comments for specific guidance if you deploy the data foundation with more than one currency.DEPLOY_SAP, DEPLOY_SFDC, DEPLOY_CDC, and TEST_DATA) are now read from the config file. These are still defaulted in cloudbuild.yaml substitutions. If you want to use the values in the file, the substitutions section in cloudbuild.yaml needs to be commented out. Substitutions from the command line would be phased out.currency_conversion DAG.Known Issues and Limitations
sap_config.json needs to be manually generated from config.env for the materializer to read as input. Dotenv files would be removed in the next major release, and this would be addressed for SAP submodules.UNION mode.sequential deployment (TURBO=false) for SFDC incorporated.IsArchived flag is removed from CDC processing for Salesforce.cloudbuild.sfdc.yaml not finding files to copy are now caught and surfaced gracefully.substitution defaults (for example, LOCATION) from cloudbuild.yaml file so all configurations are either passed from the command line or read from config/config.json. These parameters would be removed from substitution defaults in future releases.gsutil to avoid errors.
config/config.json) for deployment parameters. We maintain backward compatibility with the existing gcloud builds submit command parameters. Enhanced parameters like SAP UNION datasets and those for Salesforce, can be configured in config.json. This file format replaces the .env format for SAP_REPORTING in the next releases.AccountingDocumentsReceivable, AccountsPayable, AccountsPayableTurnover, CashDiscountUtilization, CurrencyConversion (exposing table processed by DAG), VendorPerformance, and VendorConfig (dependant on TVARVC values). SalesOrderPricing and PricingConditions. For more information, see the updated Looker reports to make use of the split models.Known issues and limitations
cdc_settings.yaml._GEN_EXT=true to allow for more flexibility in reporting. This table has been incorporated into views POSchedule, MaterialsValuation, PurchaseDocuments, Deliveries, Billing, SalesOrderScheduleLine, AccountingDocuments, InvoiceDocuments_Flow, POOrderHistory, and SalesOrders_V2. The structure of the table is generated without data if _GEN_EXT=false. If you don't want to execute the generation of other DAGs for external sources, remove them from the list in the generation script.AccountingDocuments, InvoiceDocuments_Flow,Billing, Deliveries, MaterialsValuation, PurchaseDocuments, POOrderHistory, and SalesOrders_V2 when using currency conversion generated from DAG materializing results (see CORTEX-CUSTOMER tags and differences when merging).OneTouchOrder, SalesOrderHeaderStatus, SalesOrderScheduleLine, SalesOrderPartnerFunction, and SalesOrderDetails_SAMPLE.ECC or S4 specific functions with non-specific (for example, fiscal_Case_ecc or fiscal_case_s4 now is fiscal_case). SQL flavor-specific functions would be commented out from generation in the next release.src/SAP/. UNION logic available if your SAP ECC and S/4 systems don't share the same MANDT. All views are now moved to /ecc or /s4 respectively. This alters the templates of all of the views and requires two sets of datasets (for example, one raw landing for ECC, one for S/4). These datasets can be passed to the deployment in the file sap_config.env.sap_config.env) instead of parameters from the gcloud builds submit --substitutions command. Parameters received from the substitutions override configurations in the file to maintain backwards compatibility. This affects all the cloudbuild.yaml files._SalesOrders_V2_) with less underlying joins for use cases that don't require them.experimental as we increase the functionality around and on top of them in collaboration with C3.ai. Experimental views are flagged with an ## EXPERIMENTAL comment.sap_config.env contains a variable called TURBO. If set to true, the cloudbuild.yaml file is generated dynamically from the dependencies file. This reduces the deployment time.## CORTEX-CUSTOMER.WeatherDaily, SalesOrders_V2, DeliveryBlockingReasonsMD, BillingBlockingReasonsMD, StorageLocationsMD, PurchaseDocumentTypesMD, MovementTypesMD, ReasonForMovementTypesMD, SpecialStocksMD, PurchasingOrganizationsMD, BatchesMD, MaterialPlantsMD, ProductionOrders, PurchasingGroupsMD, MaterialsValuation, MaterialsMovement, MaterialTypesMD, BillOfMaterialsMD, DivisionsMD, and ValuationAreasMD.MERGE operation now copies the test data from the raw landing dataset into the CDC dataset. Test data in CDC dataset isn't copied as it is from the raw landing. Test data is still meant to provide a base for specific demo scenarios and isn't guaranteed to be a reliable source with any quality or referential integrity.Known issues
9999-12-31..INCLUDE when generating DAGs and Runtime views.Features and improvements
setting.yaml file and test harness to deploy either ECC or S4:
MANDT = 050 for ECC and MANDT = 100 for S/4. These populates automatically based on the _SQL_FLAVOUR parameter in the build (default to ECC).MANDT = 800).FixedAssets, SDStatus_Items, GLDocumentsHdr, BSID and BSAD implementations, InvoiceDocuments_flow,DeliveriesStatus_PerSalesOrg, SalesFulfillment_perOrder, SalesFulfillment, and UoMUsage_SAMPLE.Billing, OrderToCas, and AccountingDocumentsReceivable._SQL_FLAVOUR parameter. Possible values are ECC or S4.Deliveries, PurchaseDocuments and SalesOrders.Bug Fixes
settings.yaml.Known issues
.INCLUDE in some tables in some versions of S/4 is known to be marked as key, which results in invalid SQLs in DAG generation. Recommendation as of now is to clear the keyflag field in the offending .INCLUDE entry in the replicated DD03L table in BigQuery when generating the DAG. New