I was able to get past this using New-AzConnectedMachineRunCommand on Azure Arc connected machines. I just placed a script on the machines(could probably use a shared location if needed) that restored the rights before trying to connect to the hybrid worker. It can take up to 5 minutes to run on a machine, but worth it for me.
New-AzConnectedMachineRunCommand -ResourceGroupName "Resourcegroup" -MachineName "machinename" -Location "eastus" -RunCommandName "TestFile" -SourceScript "Invoke-Command -FilePath C:\Scripts\restorerights.ps1 -ComputerName machinename" -SubscriptionId "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"