FsLogix - Unclean logoff causing locked files until server reboot

Much R 101

Problem is described by M4deman under unclean-logoff-causing-locked-files-until-server-reboot

It seems to have something to do with the 2009 version.
The latest version of FSLogix is installed whats-new

Description

After a user logoff, the "System" Process (PID 4) locks the following folders:

C:\Users\local_username\AppData\Local\Microsoft\Credentials
C:\Users\local_username\AppData\Roaming\Microsoft\Credentials

The user is completely logged of, according to Task Manager.

In the FSLogix Profile Log file I can see the following:

[07:53:55.601][tid:00000c90.0000ce44][ERROR:00000020] Delete profile failed for sid S-1-5-21-3364776539-3721753400-1968955100-1179, Cleaning up manually. (Der Prozess kann nicht auf die Datei zugreifen, da sie von einem anderen Prozess verwendet wird.)
The last sentence means that the process cannot access the file, because another process already uses it.

Also the whole "local_username" folder cannot be deleted:

[08:23:15.479][tid:00000c90.0000bcc4][WARN: 00000005] Failed to delete C:\Users\local_usename (Access is denied)
Access Denied

Does someone have any info on this behaviour?

Anonymous

2021-01-14T03:03:58.32+00:00

Hello @Much R

Did you check affected user has full control and Owner over their FSLogix Folder and the VHDX files ?

Thanks
Karlie
Much R 101 Reputation points

2021-02-02T08:02:21.997+00:00

Hello,

Where is my answer from yesterday?
Why was this deleted?

How can I open a ticket at Microsoft now?

Thank you
Anonymous

2021-02-02T08:16:39.607+00:00

Hello @Much R

sorry I was going to delete my comment as it wasn't helpful, and your comment is below mine, so it seems they were deleted together. I want to restore but it doesn't has this option.

FSLogix is supported for use with versions of Windows and Office that are still included in the Microsoft Support Lifecycle.

How to open a FSLogix Support Request
https://social.msdn.microsoft.com/Forums/en-US/14ea5f6f-760b-4ad2-83cf-23c3cbc1bcc4/how-to-open-a-fslogix-support-request?forum=FSLogix

Best regards
Karlie
Germain 11 Reputation points

2021-02-19T11:44:10.117+00:00

We are currently experiencing the same issue. The user VHDX is locked by system and can only be unlocked by rebooting the server.
Has Microsoft acknowledged this issue?

Currently using: 2.9.7654.46150
Bracko Ewald 26 Reputation points

2021-02-19T12:10:09.137+00:00

Did you activate the registry value HKEY_LOCAL_MACHINE\SOFTWARE\FSLogix\Apps\CleanupInvalidSessions (REG_DWORD, 0x1)?
Do you use Cloud Cache or not?

Also, maybe you can check with the frx command line tool if there are still some redirects pointing to the use VHDX file. You can then use that tool to delete them manually and detach the VHDX files afterwards using diskpart.
Marius Neuhaus 1 Reputation point

2021-03-09T10:31:38.737+00:00

Hello,

we have the same problem.
We use on-premise terminalserver.:
Windows Server 2019
Citrix 1912 CU2 LTSR
FSLogix 2.9.7621.30127

I've tried the CleanupInvalidSessions RegKey, but it doesn't effect.
We don't use Cloud Cache.
FSLogix Configuration only local and not per gpo.
The frx command line tool dont show redirects pointing to the use VHDX file.
Permission are set correct. Users work correctly with the containers.
Luckily the problem occurs only randomly and not at every logoff.

In the FSLogix Configuration we include the group domain-users and exclude the domain-administrators.
Could it be a Problem?

Best Regards,
Marius
itsupport 1 Reputation point

2021-05-26T12:40:37.147+00:00

When this happens are you left with multiple <USERNAME>.original folders? I've seen it go as high as <USERNAME>.original11 before I just manually delete all the folders.
Guido Racamato 26 Reputation points

2021-09-28T17:39:34.337+00:00

Hello Everyone, For the Group Policy Access Denied, once you get an affected user, could you please modify the following registry? There's no reboot required.

HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProfileService\References\<SID affected user>
REG_BINARY: RefCount
Value: 0
Jon McLeod 1 Reputation point

2021-11-19T14:56:08.267+00:00

This is the answer. Thank you.
Farooq 5 Reputation points

2024-02-14T12:02:36.9533333+00:00

We are having the same issue. We have an RDS environment and time to time users get logged in with Temporary profile, when I check the FSlogix logs I am seeing the following error - Failed to open virtual disk: \server\Profiles\S-1-5-21-26711-2174_user\Profile_user.vhd (The process cannot access the file because it is being used by another process.)
What we done to try resolve this or at least try find the cause and reduce downtime is, we have removed AV from all RDS host, we have created a script which logs out all users at midnight but issue is still occurring. We have a case opened with Microsoft but unfortunately there has been no resolution. Another issue we had is where users could not sign into their Outlook, teams, Office profile on the host. Has anyone else experiencing this issue and has anyone managed to resolve this? Any help or suggestion would be much appreciated.

107 answers

Brandon Clark 1 Reputation point

2021-10-19T13:10:26.007+00:00

Hey all, just so you're aware. I troubleshot this error along with Microsoft Support for 48 hours straight. In that time I got to learn FsLogix pretty well. In OUR case, it was Symantec DLP, the Data Loss Protection software. The issue was our DLP software was OUT OF DATE. DLP (from Symantec/Broadcom) uses a FILTER DRIVER. This out of date filter driver was causing ONEDRIVE to hang, which in turn caused the Office Container to hang because of a stuck handle. Once we ugpraded our Symnatec DLP from 15.5 to 15.7, the issue went away. This was confirm with an open ticket to Broadcom as well.
Please sign in to rate this answer.
Андрей Михалевский 3,461 Reputation points

2021-10-19T13:17:11.19+00:00

We only use microsoft office 2019 pro and google chrome + adobe acrobat pdf.

We face this problem all the time. I even recorded it a couple of times when outlook hung up and I ended the session. After that I couldn't log in anymore.

In some cases, FSLogix can't properly terminate the process.

Is this a Microsoft Office, Windows Server 2019 or FSLogix problem ?

Brandon Clark 1 Reputation point

2021-10-19T13:31:42.78+00:00

Open CMD as Admin and run this: 'fltmc filters'

This will show you the filesystem filter drivers in place. It's likely to be one of those.

But yes, if a process is hung and cannot be stopped and that process is writing to an FsLogix container, then you need to find out WHY that process is getting hung. It's not FsLogix fault the process can't exit, likely.

Андрей Михалевский 3,461 Reputation points

2021-10-19T13:39:49.43+00:00

What should I do after running this command?

All applications fail in one way or another. I had no such problems with UPD.

Is there no way to force all processes to terminate on exit?

If it is so, it is not possible to fix this problem. In my 4 years of experience, terminal servers are always unstable. Including Microsoft applications.

Bracko Ewald 26 Reputation points

2021-10-19T13:42:03.99+00:00

You don't use any third party AV scanners, correct?
Then usually the Windows Defender is activated by default.
Have you set the recommended AV exclusions for the Windows Defender?
Also, keep in mind that you don't need to activate the Search Database related settings in FSLogix because Windows Server 2019 natively stores the user-related search database within the user profile now. In fact you should turn them off now to avoid issues there.

Brandon Clark 1 Reputation point

2021-10-19T13:42:07.203+00:00

Track down each one of the items listed in that command's output and ensure those filter drivers are up to date. These drivers are usually installed from software you've added.

I've been working in multi-session environments for 20 years. Trust me, it's always been hard.

Андрей Михалевский 3,461 Reputation points

2021-10-19T13:47:18.71+00:00

As I said, I only have 3 products on the terminal server. Office, Google Chrome and Adobe Acrobat. 64 architecture and the latest updates. Even with this configuration, 11 servers and 500 users, often create requests that they can't re-logon.

Андрей Михалевский 3,461 Reputation points

2021-10-19T13:48:14.107+00:00

Yes, I use these recommendations. It doesn't help.

Bracko Ewald 26 Reputation points

2021-10-19T14:46:52.333+00:00

But the FSLogix Apps Service is still running and doesn't suddenly stop, correct?
Can you see the FSLogix disks remaining in the Disk Management? Does it take quite long until the Disk Management opens?
We had recently a weird issue with one of our customers. The FSLogix disks frequently wouldn't disconnect and in that case often the FSLogix Apps Service crashed. This crash wasn't even properly reported in the System Log. Only the FSLogix service logs reported the shutdown of the FSLogix Apps Service. And you often also couldn't start this service anymore until the next reboot. A manual health check of the affected FSLogix disks would show that those disks were all without errors.
The RDS hosts are VMs running on VMware vSphere. It turned out that the physical servers the vSphere is running on had a NIC from the vendor installed that wasn't supported for that specific server model. After we replaced those NICs and updated the firmware on the physical servers everything runs smoothly now.
So sometimes it can be something deeper in the system that can cause those issues.

Андрей Михалевский 3,461 Reputation points

2021-10-19T14:55:11.45+00:00

As I understand it, we'll have to live with it. It is practically impossible to diagnose it on your own. And I do not want to pay to microsoft.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.
Martijn Kools 166 Reputation points

2021-10-19T14:13:13.96+00:00

I have two different Citrix environments. Both are pretty much identical. Same FSLogix version, same GPOs, same patch level. Almost same everything except for the language and some apps.

Both run on Citrix PVS, Windows Server 2019 with XenApp 1912 LTSR CU3, FSLogix 2105. Both vdisks are monthly rebuilt from scratch, fully scripted.

On one env I have the issue:

[01:20:09.192][tid:00001990.00001ff8][WARN: 00000005] Failed to delete C:\Users\local_x (Access is denied.)
[01:20:09.192][tid:00001990.00001ff8][WARN: 00000005] Failed to delete C:\Users\local_x (Access is denied.)
[01:20:09.192][tid:00001990.00001ff8][WARN: 00000005] Failed to delete C:\Users\local_x (Access is denied.)
[01:20:09.192][tid:00001990.00001ff8][WARN: 00000005] Failed to delete C:\Users\local_x (Access is denied.)
[01:20:09.192][tid:00001990.00001ff8][WARN: 00000005] Failed to delete C:\Users\local_x (Access is denied.)

On the other one I don't:

[15:17:25.851][tid:000017d4.000068e0][INFO] Successfully removed C:\Users\local_x

Both run Windows Defender with the EXACT same settings (they even share the same GPO).

I checked BrandonClark's command fltmc filters and they give the same output on both envs.

So why does it not work on env A and why does it work on env B?

Luckily, I don't have the issue where the users can't re-logon and the vdisk is locked. The vdisk actually is not locked any longer after logoff and people can just relogon. If they happen to hit the same server it will usually create a different folder like local_username_1 or something like that.
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.
Krüger Benjamin 1 Reputation point

2021-11-29T14:02:13.933+00:00

A preview update has been released. "KB5007266"

Addresses an issue that causessearchindexer.exe to keep handles to the per user search database in the path below after you sign out: “C:\Users\username\AppData\Roaming\Microsoft\Search\Data\Applications\<SID>\” As a result, searchindexer.exe stops working and duplicate profile names are created.

https://support.microsoft.com/en-us/topic/november-22-2021-kb5007266-os-build-17763-2330-preview-c9ba0c4c-c8b7-409a-8fac-a76ffae8f94f

https://twitter.com/\_POPPELGAARD/status/1463418452958453768
Please sign in to rate this answer.
Андрей Михалевский 3,461 Reputation points

2021-11-30T07:04:34.873+00:00

This update doesn't fix the problem !!!!
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.
MarkD 1 Reputation point

2021-11-24T21:57:57.35+00:00
Hi All, we battled variations of this and took us a while to pin ours down because we believed it to be centered on FSLogix. We experienced many of the symptoms laid out here and it was very intermittent and unpredictable.

Environment:

Citrix Virtual Apps (Published Desktops / Shared Session Hosts) Version 1912.

Windows Server 2019

Tried with two FSLogix versions: 2.9.7117.27413 and 2.9.7979.62170

Our Experience:

Users VHDX would get locked.

The Citrix Session would be logged off most of the time but not always.

Regardless of ^^ the session would show active from VHDX storage using "Net Session" sourcing from the VDA they were logged in from.

They would get temporary profiles due to profile lock.

Sometimes while in the middle of working, they would lock up.

Rebooting the non-persistent VDA seemed to be the only way to truly clear it along with manually closing the open profile on the VHDX Profile share.

This was random across 20+ VDA servers.

Our Troubleshooting:

We would watch the session timers on the shared storage using Net Session and keep an eye on ones that would have high counters.

We would review FSLogix logs on users who were getting temp profiles.

We would review event logs.

Lots of reading which brought me here.

Summary:

The real pointer came when we were able to remote to a user who was actively having a session lock while still working. Since the VDA's were being rebooted daily to help get through the issue, we would miss some log entries.

Ultimately, that user was creating a ton of Event ID 107 Citrix.Authentication.IdentityAssertion in the app log.

That led us to article: https://support.citrix.com/article/CTX255423

That was our issue, the Kerberos Renew Time was set to 10 hours. Many of our users due to various reasons have been working longer hours and we are requried to have very high session limit timers for idle but active then disconnected. If that user had never disconnected and reconnected during the day or even had their 20 minute idle screen come up, their ticket would expire and not renew causing all access to network resources to come to a halt. Primary indicator was loss of access to the VHDX. The session wouldn't tear down correctly due to this. We discovered this in our GPO's and modified that at the domain level and we have been error free since.

Another helper was a log aggregator that allowed us to see TGT requests and if the user only had the initial request and no subsequent requests from that machine, you could begin to predict the failure.

Check your Kerberos settings in your Default Domain Policy or whatever policy is applied at the domain level that manages Kerberos ticket times.

I am sure this won't solve it for all but based on what I read here, it might be contributing.

It was very frustrating to figure out. Looking back, since this has been this way since before I started working here, it explained some other strange access issues that got written off after reboots.

Hope this helps!
Please sign in to rate this answer.
bjkoekkoek 1 Reputation point

2021-11-25T14:35:33.12+00:00

To how many hours did you extend the maximum lifetime? Since 10 hours is the best practice by Microsoft.

MarkD 1 Reputation point

2021-11-25T15:52:05.77+00:00

@bjkoekkoek

We didn't adjust the practice for the ticket lifetime, that wasn't where the issue was. 10 hours is the best practice and where that is set for us.

It is the renewal time that was the problem. Microsoft suggests and sets the default to 7 days in a domain deployment but also mentions to adjust for balance between security and usability. We had employees working more than 10 hours that didn't get a renewal event in that time frame. Once the 10 hour renewal period passed, that ticket couldn't be renewed which caused access to the profile disk to be denied. So, basically, it has to be longer than the maximum time a user would work consistently without potentially renewing the ticket.

https://free.blessedness.top/en-us/windows/security/threat-protection/security-policy-settings/maximum-lifetime-for-user-ticket-renewal
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.
Jan Koppelin 1 Reputation point

2021-11-27T09:12:10.74+00:00

Hello everyone,
after an update from 1912 CU2 to 1912 CU4, we have the exact the same problems.

After a long search we restored the master image from the backup and the problems were gone...

We examined the difference between the master images and found that the working worker has not "Citrix User Profile Manager" and a "Citrix User Profile Manager WMI Plug-In" installed. Is there a connection? Do you have these VDA plugins installed?
Please sign in to rate this answer.
Jan Koppelin 1 Reputation point

2021-11-30T08:14:21.977+00:00

I can reproduce the problem. If I install the VDA with CPM and CPM-WMI plugin, it comes to the errors with FS-Logix. If I remove the plugins, there are no problems with FS-Logix. Can anyone confirm that?
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

FsLogix - Unclean logoff causing locked files until server reboot

107 answers

Your answer