Modern Data Platform

Due to an increase in service providers in the cloud computing domain, there is a lot of options available for the organizations to choose a platform for storing their data, processing it, presenting it, and using advanced technologies. Modern data platform is a set of different tools that are brought together to manage and use the data efficiently for organizations to take data driven decisions. 

Modern Data Platform should be made considering that it should be highly scalable, highly available and must be include all the tools required in it. The main components of MDP are as follows: 

1. Ingesting the data- This is the start of the MDP process where you must consider and select tools that will be used to fetch or access the data from different sources. These tools will pull the data and then give it to the MDP, and it works as a connection between the MDP and external data sources. The data coming from the sources may be structured, unstructured, streamlined or batch data. 

2. Data Storage- The data retrieved from various sources must be stored by the organization for later usage. For this, we can use Cloud Datalake, Cloud data store, data warehouse, etc. This must be selected based on cost, accessibility, and speed of access. 

3. Data processing- The data stored in the company’s storage is raw data and cannot be directly used for any purpose. So, the data has to undergo some processing in order to be used for making business decisions and getting any insights from it. For this, we can use Python/R, Bigquery, SQL, etc. This will convert raw data into usable data. 

4. Data Modelling and advanced technologies- This data can be used to feed into advanced technological models like machine learning, deep learning, etc. We can use Anaconda, AWS sagemaker, GCP ML engine, etc. These models will help in understanding patterns or getting insights from the data. 

5. Data presentation- This is the output stage of MDP. Here we can use tools which help in creating views, dashboards, etc. Tools like Tableau, PowerBI, Google analytics,etc can be used. This will be used for visualizing the data and works as a connection between MDP and humans to easily understand the data and take business decisions which would be data driven. 

6. Data Privacy, Access control- Data governance is one of the most important practices that should be implemented due to the threat of hacking and data being leaked out of organization. For this, we can use AWS identity and access management, security groups, etc. 

An MDP is a collection of tools that must be carefully selected, and the bundle of tools must be enough for everything that needs to maintain and use the data for the organization.