Introducing Foolish Flamenco, the most ambitious release of the Sidra Data Platform!

As you know, Sidra Data Platform is our solution to provide organizations of any size not only with a Data Lake infrastructure and its associated services but also with a comprehensive business data strategy solution. Thanks to Sidra, our customers can intake/leverage the siloed data their business needs in a fast, effective, and governed way, and make it available through  specific use cases across the organization in a standardized way.

For years, we have been developing corporate Big Data solutions for multiple global companies, which has enriched our technical expertise, and taught us to preempt and solve the common needs of our customers. This has allowed us to develop a scalable and extensible solution that has been the origin of Sidra, and since then, we have been continuously working on the improvement of our platform.

For those (and many other reasons), we are proud to share with you the most ambitious version in the history of Sidra: Foolish Flamenco.

Sidra Data Platform: What’s new in Foolish  Flamenco

This release completes the exhaustive security review initiated with the release of 2019.R4 in October 2019, with the implementation of the new Extract API security model, as well as the adoption of Balea: an open-source authorization framework for ASP.NET Core whose development is being led by Plain Concepts.

We have added many functionalities and enhancements to the Data Catalog Web UI including, among others, metadata edition for Providers and Entities and Data Preview functionality with support for masking of sensitive information.

From a client application perspective, in this version we have updated our Data Labs application by making its structure more flexible and providing coverage for more analytical scenarios. We also released the first version of Sidra’s answer to Master Data management, which will help many customers in their business integration scenarios.

We also want to announce today our first public version of PySidra, the new Python package to operate with the Sidra Data Catalog, ML platform, etc. Although this package was already being used internally, we have decided to publish it, as we consider it a very useful tool for any data scientist currently working with Sidra.

As for DSUs and Sidra Core, we now support external views and tables in Databricks. In addition, we have made improvements to Sidra’s automated deployment model, taking advantage of the Azure DevOps new Multi-Stage pipelines feature.

Finally, we have implemented several improvements focused on reducing platform operating costs that will help reduce the cost of a default instance of Sidra.

Here is a list of the most relevant improvements made to the platform, as part of this release:

• Optimizations in the transfer query generator
• New features on the Data Catalog
• SQL Incremental Loads using the Change Tracking feature
• New Master Data module
• Data Labs module improvements
• Measures to reduce operational costs of the platform
• View support in Databricks
• Support for external tables in Databricks
• New security model in Extract activity
• New API to create Azure Linked Services
• Simplification of system pipelines
• New authorization model with Balea
• New Multi-Stage pipelines deployment model
• Improvements to the management web UI

We hope you enjoy this new version!