Money is always important whether you are developing for yourself, your employer, or a client.
One way to save money in Azure is to shut down resources at the end of the day. As an Azure data engineer, it can be a tedious task that is easy to forget. There can be a lot of clicking through resources in the portal to get things shut down, especially if you have a lot of resources. Why not make our lives easier by scripting a solution?
Candidates for Shutdown
In this post, I’m going to talk about two resources that I usually shut down at the end of the workday while they are in active development: VMs and Synapse Analytic SQL Pools.
VMs
VMs are often used as development platforms during the day and are left idle throughout the night. While idle, the cost still accumulates. There are a few options available to shut down VMs. VMs can be shut down one by one manually via the portal.
There is also a great Start/Stop feature that can automatically shutdown your VMs at specific times. This is effective, but only if you work during fixed hours. The other option is shutting down via the Azure CLI which I will explain here.
Before going any further it is important to understand 2 VM states:
- Stopped – The VM is powered down. This is equivalent to (and can be achieved by) shutting down the VM from the OS. Costs are the same as if it is running.
- Stopped (Deallocated) – The compute resources are deallocated and you only pay for Azure Storage.
One caveat is that you will likely lose your dynamically assigned IP address when deallocating.
The portal Stop button moves the VM into the “Stopped (Deallocated)” state.
Azure CLI
Shutting down a VM via the Azure CLI gives the most control and flexibility. It is also an easy way to become familiar with the Azure CLI.
The commands covered today are
If you have a couple of static VMs you can just hard code these two commands and save them in a shell script and run them at the end of your workday. I recommend using the Azure Cloud Shell.
az vm stop --resource-group MyResourceGroup --name MyVm
az vm deallocate --name MyVm --no-wait --resource-group MyResourceGroup
If you have a few VMs in a resource group you can shut them all down with a commands like these:
az vm stop --ids $(az vm list -g MyResourceGroup --query "[].id" -o tsv)
az vm deallocate --ids $(az vm list -g MyResourceGroup --query "[].id" -o tsv)
Synapse SQL Pools
Synapse Analytics Workspaces (note: at the time of writing this is a preview feature and subject to change) have SQL and Spark Pools associated with them. If there are no workloads for your fixed performance level SQL pool you can safely scale it down or pause it overnight.
Just like VMs, this can be accomplished via the portal or via script.
The Azure CLI command to pause your SQL pool is:
az synapse sql pool pause --name sqlpoolcli1 --workspace-name testsynapseworkspace --resource-group rg
Dynamic Scripting
Up to now, you’ve seen that we can use Azure CLI to shut down VMs and Synapse SQL Pools via static scripts. This is very effective if you have one project and resources don’t change frequently. If you are working in multiple projects/resource groups bash scripting (via a UNIX terminal with Azure CLI installed) can be leveraged to use these commands dynamically.
The below script is one I developed to make it easier to shut down VMs and Synapse SQL Pools within a resource group. It is presented here in its current form but will be improved and expanded in my repository. Please feel free to use it and modify it to suit your needs. It will prompt you to select the resource group and list all possible SQL Pools and VMs to shut down. You can then select which resources to shutdown/pause. It will show the commands it will run and prompt you if you want to continue.
https://gist.github.com/ZMon3y/6fc6493e0e2c714e5adec7c5f524302a
Conclusion
I hope this helps you save time and money as you develop in Azure! Please reach out to me via the repository or on Twitter @MattSzafirPro.