by ezs | Feb 20, 2021 | evilzenscientist
Sigh.
Tag taxonomy cleanup.
Another great example of “hands off keyboards” and needing to deliver via automation. Avoid errors, enforce validation of metadata.

Azure Resource Graph explorer – find the scope and scale of the problem. I’ll add the usual gripe around tags being case sensitive in some places (API, PowerShell) and not in others (Azure Portal!).
Resources
| where tags.businesowner != ”
| project name, subscriptionId, resourceGroup, tags.businessowner, tags.businesowner
by ezs | Feb 7, 2021 | evilzenscientist
Back to “Azure Grim Reaper” – flagging and deleting unused storage in Azure.
There is no “nice” way to do this – the Azure Metrics help a lot.
Here’s some code that enumerates a list of subscriptions and pulls out storage accounts, transactions over a period of time, count of storage containers, tables and queues – and reads tags.
It’s generic enough for use in most places.
# setup
#import-module az
#import-module az.storage
# creds
#connect-azaccount
#
# set up array of subs
$subs= ‘<subscription ID>’, ‘<subscription ID>’
# today
$nowdate = Get-Date
#initialise output
$stgdata =@()
Write-Host “Enumerating” $subs.count “subscription(s)”
# loop through the subscription(s)
foreach ($subscription in $subs) {
# in subscription, next read the storage accounts
#set context
write-host “Switching to subscription” $subscription
set-azcontext -subscription $subscription |out-null
write-host “enumerating storage accounts”
$stgacclist = get-azstorageaccount
write-host “Total of” $stgacclist.count “storage accounts in subscription” $subscription
# loop through each storage account
foreach ($stgacc in $stgacclist) {
# in storage account, get all storage containers, tables and queues
# stgacc entity has StorageAccountNme and ResourceGroupName and tags
set-azcurrentstorageaccount -Name $stgacc.StorageAccountName -ResourceGroupName $stgacc.ResourceGroupName
write-host “enumerating storage account ” $stgacc.StorageAccountName ” in resource group” $stgacc.ResourceGroupName
write-host “storage containers”
$stgcontainers = get-azstoragecontainer
write-host “table service”
$tblservice = get-azstoragetable
write-host “queue service”
$qservice = get-azstoragequeue
#get transactions
$transactions = Get-AzMetric -ResourceId $stgacc.id -TimeGrain 0.1:00:00 -starttime ((get-date).AddDays(-60)) -endtime (get-date) -MetricNames “Transactions” -WarningAction SilentlyContinue
write-host “60 days of transactions” $(($transactions.Data | Measure-Object -Property total -Sum).sum)
#reset usage array
$lastused = @()
#loop through each blob service container
foreach ($container in $stgcontainers) {
# in blob storage container
write-host “collecting blob storage data”
$lastused += [PSCustomObject]@{
Subscription = $subscription
ResourceGroup = $stgacc.ResourceGroupName
StorageAccount = $stgacc.StorageAccountName
StorageContainer = $container.name
LastModified = $container.lastmodified.Date
Age = ($nowdate – $container.LastModified.Date).Days
}
#let’s find any unmanaged disks in the container
$allblobs = get-azstorageblob -container $container.name
$vhdblobs = $allblobs | Where-Object {$_.BlobType -eq ‘PageBlob’ -and $_.Name.EndsWith(‘.vhd’)}
}
write-host “Total of ” $stgcontainers.count “containers. Minimum age ” ($lastused.Age |Measure -Minimum).minimum “maximum age” ($lastused.Age |Measure -Maximum).maximum
#append to the array
$stgdata += [PSCustomObject]@{
Subscription = $subscription
ResourceGroup = $stgacc.ResourceGroupName
StorageAccount = $stgacc.StorageAccountName
Transactions = $(($transactions.Data | Measure-Object -Property total -Sum).sum)
ContainerCount = $stgcontainers.Count
TableCount = $tblservice.Count
QueueCount = $qservice.Count
AgingMin = ($lastused.Age |Measure -Minimum).minimum
AgingMax = ($lastused.Age |Measure -Maximum).maximum
vhd = $vhdblobs.count
blobs = $allblobs.count
#tag metadata
itowner = $stgacc.tags.itowner
businessowner = $stgacc.tags.businessowner
application =$stgacc.tags.application
costcenter = $stgacc.tags.costcenter
}
}
}
#output to csv
$stgdata | export-csv <somelocation>\storageacc.csv -force -NoTypeInformation
by ezs | Aug 29, 2020 | evilzenscientist
More fun this morning with grappling DAX to pull out the data I need.
We write multiple tags to Azure resources – from the resource group down to anything that takes them. Makes showback, accountability, support so much cleaner.
The tags are returned, in a non deterministic order, via the billing API – and look like this (data cleaned up), a list of tag/values. JSON-like ish kinda.
“application“: “Shared SQL“,”project“: “Shared Database Engineering“,”costcenter“: “1234“,”itowner“: “Buckley, Martin“,”supportgroup“: “Database Engineering“,”businessowner“: “Buckley, Martin“
I was looking at some DAX to pull this apart; PowerBI has a native “extract text between delimiters” function. It’s not going to build a tree of arbitary values; but I can now cleanly extract the tags I need to build this datamodel.
After I get this working, it might be time to pull all of the discrete DAX steps together to make it more efficient against a vast dataset.
Fun times on a Saturday.
Follow up to this. Extracting multiple pieces of insight from the tags – that drive KPIs for service improvement:
– resource groups with no costcenter tag (this is the cardinal sin – no costcenter and showback isn’t there, getting to health on this is easy)
– resource groups where itowner or businessowner don’t meet business rules (ICs can’t be owners)
– resource groups where the supportgroup is not valid (it doesn’t match a ServiceNow Assigment group)
Most of all – I can find everything tagged with “placeholder” or “set by policy” and sweep that broom through the tagging.
All in all – a good few hours of work.
by ezs | Apr 1, 2020 | evilzenscientist
Maintenance night – SLES 12 SP5 updates on Azure.
zypper patch-check
zypper migration
Nice and easy.
Side note – if products somehow get unregistered:
zypper pd
SUSEConnect -p sle-module-legacy/12/x86_64
by ezs | Jan 11, 2014 | evilzenscientist, Uncategorized
The never ending saga of running infrastructure has gone full circle. All of my self-hosted blogs and websites have moved from a Hyper-V server in my garage and up to Microsoft Windows Azure.
I spent a decade (2001-2011) self-hosting. Running both email and website on platforms as diverse as NetWare through RedHat, SUSE and finally Windows Server. In early 2011 I decided to get out of the hosting game and move the blogs to GoDaddy and the mail to Exchange Online (then BPOS, now Office 365).
GoDaddy really frustrated me. Poor performance, poor logging, strange “go slows” and unexpected disconnects of SSH. Understandably “I got what I paid for” – but frustrating none the less.
It has taken eighteen months – but slowly the various blogs and websites have been culled and archived, moved back to self-host and now up to Azure.
Windows Azure has really improved over the past two years – I started by evaluating the Platform as a Service capabilities back in November 2011. Functional – but missing some capabilities that I needed to run the photo blogs (all written for the LAMP stack).
More recently as the Infrastructure as a Service capabilities have moved from preview to production I tested this blog running on Azure. Performance and uptime have been great. The blog is running a SLES 11 SP3 platform on Azure – and it’s as functional as running on bare metal or any hypervisor.
I finally took the time to move the family blogs and business website to the platform over the last week or so. The migration of WordPress blogs and Gallery photo sites is pretty simple now.
Recent Comments