Skip to content

debug#

Databricks Connect

Databricks Connect allows you to connect your favorite IDE (PyCharm, VSCode, etc.) and other custom applications to Databricks compute and run Spark (or non-Spark) code.

This post is not a comprehensive guide on Databricks Connect; rather, it consists of side notes from the Azure Databricks docs. Most of the notes also apply to Databricks on AWS and GCP.

Profiling Python code

NameScopeweb framework middlewareVSCode Extension
scalenecpu, gpu, memory, durationpartiallyyes
cProfile
(Python native, function level only and cli only)		durationnono
VizTracerdurationunknownyes
profyle
(based on Viztracer)		durationyesno
pyinstrumentdurationyesno
py-spydurationnono
yappi
(cli only)		durationunknownno
austindurationunknownyes

Interesting reading: