DATABRICKS™ INTEGRATION WITH MINSKY™

Introduction to Data-Warehouse:

A data warehouse is a type of data management system that often contains large amounts of historical data from single or multiple data sources such as APIs, Databases, Cloud Storage, etc., using the ETL (Extract Load Transform) process. It is designed to enable and support business intelligence (BI) activities, especially analytics to understand the relationships and trends across the data. Data warehouses are used to perform complex queries and analytics. This collection of business data is used for reporting and also to help an organisation to make better decisions.

Overview of Data Bricks™:

Data-bricks is an industry-leading data warehouse software, cloud-based data engineering tool used for processing and transforming massive quantities of data and exploring the data through machine learning models. Recently added to Azure, it’s the latest big data tool for the Microsoft cloud. Available to all organizations, it allows them to easily achieve the full potential of combining their data, ELT processes, and machine learning.

This Apache-Spark based platform runs a distributed system behind the scenes, (i.e.) the workload is automatically split across various processors and scales up and down on demand. Increased efficiency results in direct time and cost savings for massive tasks. Like with all Azure tools, resources (like the number of computing clusters) are easily managed and it takes just minutes to get started.

Goal of this project / Problem Statement:

Currently, Organizations are collecting vast amounts of data daily and developing advanced data warehouse systems to secure the data. Their intention is transforming it into vital information, or knowledge for developing better decision support systems (DSS). But still challenges exist with regard to the techniques of analysing and interpreting the exact meaning of enormous data especially to integrate artificial intelligence (AI) projects with data warehouses. It is much more likely that critical information is often overlooked or not tapped into from these vast amounts of data, while organizations invest huge amounts of financial resources on collecting, storing and securing the same data. After comprehensively evaluating the challenges in the current market, Ai labs developed a strategy of Integrating its proprietary Ai engine Minsky™ with Data warehouse (Databricks). The goal of this Project was to load data (structured/unstructured) to a Data-warehouse and only retrieve data as required (on demand) to a local Minsky™ DB for modelling/inferences.

Solution Overview:

Our model-driven architecture uses the Minsky™ Ai platform to provide an object model that makes it easy to integrate both internal and external data sources. The Ai Labs Minsky™ platform stores only real time data internally in SQL DB while maintain the big data in the Databricks cloud. The historical data files are loaded to Minsky using DBFS from Databricks for Modelling and the resultant model is saved in Minsky DB while the big data is still preserved in the Databricks cloud. Real time data is then retrieved from Databricks on-Demand for inferences/predictions.

Integration of Databricks™ with Minsky™

This consists of the following tasks:

• Use Databricks as a Data repository for historical Data
• Feed the historical data to Minsky for Modelling
• Save copy of this data in Databricks Data Repository
• Feed Real Time data to Minsky for Predictions and save a copy in Databricks Data Repository
• Save Minsky Output/Results in Databricks.
• Feed appropriate data to Business Intelligence (BI) tools for Dashboards, Reports.

Functional Flow:

• Data upload from local to data bricks (historical/ Big Data) in various formats
• Verify/view the uploaded historical data in data bricks and then build the Models that are stored in Minsky™
• Real time data is then uploaded to Databricks (from External data sources or IoT Devices) and later pulled to Minsky and tested against the Models for Inferences/Predictions.

Key Results:

• Seamless integration of Input Data to Databricks and Minsky™ platform.
• Increased data storage/movement efficiency.
• Enhanced Business Intelligence, Data Quality and Consistency.
• Streamlined Real-time Dataflow.
• Improved Performance and Decision-making process.
• Improved Data Security.
• User-Friendly cloud-based AI platform
• Scalable across various domains/data.
• Easy integration with other third-party solutions such as Tableau, Power Bi for data visualization.