Databricks questions
Explain the process of moving a data located in ADLS to an on-prem SQL database
using, only, Data Bricks. - answer Set up transit virtual network with azure virtual
network gateway Need azure virtual network gateway in a transit vnet
Peer the azure databricks virtual network with the transit network
Create user defined routes and associate them with azure data bricks virtual network
subsets
Validate the set up Create cluster in dbrix workspace if failure Ping an on prem IP from
a notebook
How to trigger a pipeline in Databricks? - answerCreate datafactory
Create pipeline - in datafactory add pipeline
Create parameter to be used in pipeline - later will be passed to databricks notebook
activity
Launch data bricks workspace
Create new folder in workspace and new notebook
In datafactory add parameter to notebook activity
Validate
Publish
Add trigger in notebook space - trigger now
What's the difference between Interactive Clusters and Job Cluster? - answerInteractive
clusters are used to analyze data collaboratively with interactive notebooks. Job clusters
are used to run fast and robust automated workflows using the UI or API.
So, while in development phase, you will mostly use interactive cluster. On the other
hand once you move to production where things are more in a automated way, job
clusters comes in picture.
Scenario 1 - At any time, a business will drop multiple excel files on an Azure blob
location. How will we create a data bricks notebook to select files from the blob, parse,
process, and finally store them back in ADLS? And how can this process be
automated? - answerLaunch databricks workspace
Mount an Azure Blob Storage container
Go to libraries - maven - coordinates - spark install,
Go to workspace - users - new notebook - scala mount
blob storage so we can use it as a local drive
Read excel List sheets Take all sheets of xlsx and output csvs & store sheet names
Run on schedule
Explain the process of moving a data located in ADLS to an on-prem SQL database
using, only, Data Bricks. - answer Set up transit virtual network with azure virtual
network gateway Need azure virtual network gateway in a transit vnet
Peer the azure databricks virtual network with the transit network
Create user defined routes and associate them with azure data bricks virtual network
subsets
Validate the set up Create cluster in dbrix workspace if failure Ping an on prem IP from
a notebook
How to trigger a pipeline in Databricks? - answerCreate datafactory
Create pipeline - in datafactory add pipeline
Create parameter to be used in pipeline - later will be passed to databricks notebook
activity
Launch data bricks workspace
Create new folder in workspace and new notebook
In datafactory add parameter to notebook activity
Validate
Publish
Add trigger in notebook space - trigger now
What's the difference between Interactive Clusters and Job Cluster? - answerInteractive
clusters are used to analyze data collaboratively with interactive notebooks. Job clusters
are used to run fast and robust automated workflows using the UI or API.
So, while in development phase, you will mostly use interactive cluster. On the other
hand once you move to production where things are more in a automated way, job
clusters comes in picture.
Scenario 1 - At any time, a business will drop multiple excel files on an Azure blob
location. How will we create a data bricks notebook to select files from the blob, parse,
process, and finally store them back in ADLS? And how can this process be
automated? - answerLaunch databricks workspace
Mount an Azure Blob Storage container
Go to libraries - maven - coordinates - spark install,
Go to workspace - users - new notebook - scala mount
blob storage so we can use it as a local drive
Read excel List sheets Take all sheets of xlsx and output csvs & store sheet names
Run on schedule
