spark#

Databricks Connect

Databricks Connect allows you to connect your favorite IDE (PyCharm, VSCode, etc.) and other custom applications to Databricks compute and run Spark (or non-Spark) code.

This post is not a comprehensive guide on Databricks Connect; rather, it consists of side notes from the Azure Databricks docs. Most of the notes also apply to Databricks on AWS and GCP.

Convert json to parquet and send to Azure Blob Storage

Load a local json file into a PyArrow table, then write it to a parquet file in Azure Blob Storage without using pandas.

Databricks cluster access mode

Using Databricks Connect inside a container