Process, port, and file usages in Linux
Use lsof, fuser, ss, pgrep, pstree, ps, htop, etc. to find process, port and file usages in Linux.
debug#
Databricks Connect
Databricks Connect allows you to connect your favorite IDE (PyCharm, VSCode, etc.) and other custom applications to Databricks compute and run Spark (or non-Spark) code.
This post is not a comprehensive guide on Databricks Connect; rather, it consists of side notes from the Azure Databricks docs. Most of the notes also apply to Databricks on AWS and GCP.
Profiling Python code
| Name | Scope | web framework middleware | VSCode Extension |
|---|---|---|---|
| scalene | cpu, gpu, memory, duration | partially | yes |
| cProfile (Python native, function level only and cli only) | duration | no | no |
| VizTracer | duration | unknown | yes |
| profyle (based on Viztracer) | duration | yes | no |
| pyinstrument | duration | yes | no |
| py-spy | duration | no | no |
| yappi (cli only) | duration | unknown | no |
| austin | duration | unknown | yes |
Interesting reading:
- [2023] Profiling in Python: How to Find Performance Bottlenecks
- [2023][Podcast] Measuring Multiple Facets of Python Performance With Scalene
- [2019][PDF] A Survey of Open-Source Python Profilers by Peter Norton
- [2020] Why you should try VizTracer to understand your python program
- [2023] How to Profile Asyncio Programs with cProfile