python#

Sqlalchemy eager loading

This posts describes the differences on selectinload, joinedload, subqueryload, these 3 popular eager loading techniques in Sqlalchemy (so as to SQLModel)

Getting all users from MS Graph API in few seconds

MS Graph API's endpoint for retrieving users, GET /users can return all users of the tenant. The default limit is 100 users per page, and the maximum limit is 999 users per page. If there are more than 999 users, the response will contain a @odata.nextLink field, which is a URL to the next page of users. For a big company having a large number of users (50,000, 100,000, or even more), and it can be time-consuming to retrieve all users.

While MS Graph API provides generous throttling limits, we should find a way to parallelize the queries. This post explores sharding as a strategy to retrieve all users in a matter of seconds. The idea is to get all users by dividing users based on the first character of the userPrincipalName field.For instance, shard 1 would encompass users whose userPrincipalName starts with a, shard 2 would handle users starting with b, and so forth.

Debugging in Python

Python packaging with Flit

This post provides a simple starter demonstration on how to use Flit for building and publishing Python package. However, for more complex builds, such as compiling C code, you still need the de facto standard setuptools.

bash-git-prompt tweaks

Some tweaks I made to bash-git-prompt. dynamic Python virtualenv path, new var GIT_MESSAGE, etc.

Python local version identifiers

Python local version identifiers are used to distinguish between different builds of the same version of a package. They are used to indicate that a package has been modified in some way from the original source code, but should still be considered the same version.

Github Actions: copdips/get-azure-keyvault-secrets-action

Recently, I began a new project that requires migrating some process from Azure Pipelines to Github Actions. One of the tasks involves retrieving secrets from Azure Key Vault.

In Azure Pipelines, we have an official task called AzureKeyVault@2 designed for this purpose. However, its official counterpart in Github Actions, Azure/get-keyvault-secrets@v1, has been deprecated. The recommended alternative is Azure CLI. While Azure CLI is a suitable option, it operates in a bash shell without multithreading. If numerous secrets need to be fetched, this can be time-consuming.

Databricks Python pip authentication

Before the Databricks Unit Catalog's release, we used init scripts stored in DBFS to generate the pip.conf file during cluster startup, allowing each cluster its unique auth token. But with init scripts no longer available in the Unit Catalog's shared mode, an alternative approach is required.

Github Actions: Python

Python Asyncio