FsLogix - Unclean logoff causing locked files until server reboot

Much R 101

Problem is described by M4deman under unclean-logoff-causing-locked-files-until-server-reboot

It seems to have something to do with the 2009 version.
The latest version of FSLogix is installed whats-new

Description

After a user logoff, the "System" Process (PID 4) locks the following folders:

C:\Users\local_username\AppData\Local\Microsoft\Credentials
C:\Users\local_username\AppData\Roaming\Microsoft\Credentials

The user is completely logged of, according to Task Manager.

In the FSLogix Profile Log file I can see the following:

[07:53:55.601][tid:00000c90.0000ce44][ERROR:00000020] Delete profile failed for sid S-1-5-21-3364776539-3721753400-1968955100-1179, Cleaning up manually. (Der Prozess kann nicht auf die Datei zugreifen, da sie von einem anderen Prozess verwendet wird.)
The last sentence means that the process cannot access the file, because another process already uses it.

Also the whole "local_username" folder cannot be deleted:

[08:23:15.479][tid:00000c90.0000bcc4][WARN: 00000005] Failed to delete C:\Users\local_usename (Access is denied)
Access Denied

Does someone have any info on this behaviour?

Anonymous

2021-01-14T03:03:58.32+00:00

Hello @Much R

Did you check affected user has full control and Owner over their FSLogix Folder and the VHDX files ?

Thanks
Karlie
Much R 101 Reputation points

2021-02-02T08:02:21.997+00:00

Hello,

Where is my answer from yesterday?
Why was this deleted?

How can I open a ticket at Microsoft now?

Thank you
Anonymous

2021-02-02T08:16:39.607+00:00

Hello @Much R

sorry I was going to delete my comment as it wasn't helpful, and your comment is below mine, so it seems they were deleted together. I want to restore but it doesn't has this option.

FSLogix is supported for use with versions of Windows and Office that are still included in the Microsoft Support Lifecycle.

How to open a FSLogix Support Request
https://social.msdn.microsoft.com/Forums/en-US/14ea5f6f-760b-4ad2-83cf-23c3cbc1bcc4/how-to-open-a-fslogix-support-request?forum=FSLogix

Best regards
Karlie
Germain 11 Reputation points

2021-02-19T11:44:10.117+00:00

We are currently experiencing the same issue. The user VHDX is locked by system and can only be unlocked by rebooting the server.
Has Microsoft acknowledged this issue?

Currently using: 2.9.7654.46150
Bracko Ewald 26 Reputation points

2021-02-19T12:10:09.137+00:00

Did you activate the registry value HKEY_LOCAL_MACHINE\SOFTWARE\FSLogix\Apps\CleanupInvalidSessions (REG_DWORD, 0x1)?
Do you use Cloud Cache or not?

Also, maybe you can check with the frx command line tool if there are still some redirects pointing to the use VHDX file. You can then use that tool to delete them manually and detach the VHDX files afterwards using diskpart.
Marius Neuhaus 1 Reputation point

2021-03-09T10:31:38.737+00:00

Hello,

we have the same problem.
We use on-premise terminalserver.:
Windows Server 2019
Citrix 1912 CU2 LTSR
FSLogix 2.9.7621.30127

I've tried the CleanupInvalidSessions RegKey, but it doesn't effect.
We don't use Cloud Cache.
FSLogix Configuration only local and not per gpo.
The frx command line tool dont show redirects pointing to the use VHDX file.
Permission are set correct. Users work correctly with the containers.
Luckily the problem occurs only randomly and not at every logoff.

In the FSLogix Configuration we include the group domain-users and exclude the domain-administrators.
Could it be a Problem?

Best Regards,
Marius
itsupport 1 Reputation point

2021-05-26T12:40:37.147+00:00

When this happens are you left with multiple <USERNAME>.original folders? I've seen it go as high as <USERNAME>.original11 before I just manually delete all the folders.
Guido Racamato 26 Reputation points

2021-09-28T17:39:34.337+00:00

Hello Everyone, For the Group Policy Access Denied, once you get an affected user, could you please modify the following registry? There's no reboot required.

HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProfileService\References\<SID affected user>
REG_BINARY: RefCount
Value: 0
Jon McLeod 1 Reputation point

2021-11-19T14:56:08.267+00:00

This is the answer. Thank you.
Farooq 5 Reputation points

2024-02-14T12:02:36.9533333+00:00

We are having the same issue. We have an RDS environment and time to time users get logged in with Temporary profile, when I check the FSlogix logs I am seeing the following error - Failed to open virtual disk: \server\Profiles\S-1-5-21-26711-2174_user\Profile_user.vhd (The process cannot access the file because it is being used by another process.)
What we done to try resolve this or at least try find the cause and reduce downtime is, we have removed AV from all RDS host, we have created a script which logs out all users at midnight but issue is still occurring. We have a case opened with Microsoft but unfortunately there has been no resolution. Another issue we had is where users could not sign into their Outlook, teams, Office profile on the host. Has anyone else experiencing this issue and has anyone managed to resolve this? Any help or suggestion would be much appreciated.

107 answers

Marius Neuhaus 1 Reputation point

2021-03-11T07:41:48.95+00:00

Hello,

I have created a ticket.
I hope microsoft will get back to me quickly.
I'll post updates and soultions from microsoft.

I'm curious, for the answer.

Best Regards,
Marius
Please sign in to rate this answer.
Stefanos Evangelou 116 Reputation points

2021-03-11T11:03:34.787+00:00

Hello,

A lot of Microsoft customers have been affected by this issue and a lot of tickets have been opened on the same. My ticket has been open since Jan 29th as a critical ticket. We have performed very thorough testing, took verbose traces and carried out all actions requested by MS FsLogix support team, to no avail. For some reason they are very slow in responding. The support team is having discussions with the FsLogix product group team but they have yet to provide us with a clear and conclusive update on how the issue will be fixed. All evidence so far points to a potential software bug but we have not yet received official confirmation on this from the technical support side, despite our numerous emails to the escalation team.

Hopefully there will be action from the Microsoft side soon, since this is affecting a growing number of customers and production environments worldwide.

Marius Neuhaus 1 Reputation point

2021-03-11T11:36:56.937+00:00

a little bit more pressure for MS is okay ;D

I agree with you and hope we get some answers/fixes from MS

Dlo 96 Reputation points

2021-03-11T12:00:10.67+00:00

I just keep getting they are going threw logs every time they give me an update.

Sountharanayagam, Jan 1 Reputation point

2021-03-15T09:41:22.79+00:00

Hello everyone, we are facing the same Issue. MS Case already opened for 1 Months. Still waiting for response...
Current Workaround: Reboot the Servers every night.

Stefanos Evangelou 116 Reputation points

2021-03-16T14:04:08.75+00:00

Hello,

Unfortunately I am get the same type of response with this ticket now open for almost 1.5 month. They are requesting for same type of traces/logs which we sent long time ago and for which we never got a clear analysis of or the logs were not helpful to determine the root cause. I am starting to think that they may have issue identifying the problem from their end and to be honest I am not optimistic about a resolution any time soon.

Stefanos Evangelou 116 Reputation points

2021-03-22T15:19:49.457+00:00

Hello,
It seems we are getting closer to this being identified as a bug for resolution in an upcoming FsLogix release: https://stefanos.cloud/blog/kb/how-to-resolve-error-group-policy-client-service-failed-the-logon-access-denied-in-citrix-and-fslogix-environments/ (see bottom or article for my latest updates from FsLogix technical support and product group).

Stefanos Evangelou 116 Reputation points

2021-04-15T08:41:52.457+00:00

Issue continues to occur since the root cause has not been isolated yet from the MS side. Newest update is that the technical support team in collaboration with the product group requested a full memory crash dump from the affected Citrix session machine right after the issue occurs. Analysis of the full memory dump is now in progress: https://stefanos.cloud/blog/kb/how-to-resolve-error-group-policy-client-service-failed-the-logon-access-denied-in-citrix-and-fslogix-environments/. Hopefully this will help sort the issue out.

Stefanos Evangelou 116 Reputation points

2021-05-20T15:13:22.087+00:00

After 4 months of my Microsoft Technical Support critical case having been open, I received a reply today (May 20) stating that the case will be closed and that the product group is still investigating the issue without a known ETA for providing a fix (the issue has clearly been identified as a bug but never officially admitted as such).
See https://stefanos.cloud/blog/kb/how-to-resolve-error-group-policy-client-service-failed-the-logon-access-denied-in-citrix-and-fslogix-environments/ (bottom of article) for details on the response from the technical support team.

Stefanos Evangelou 116 Reputation points

2021-05-20T15:20:52.417+00:00

After 4 months of this case having been open with Microsoft Technical Support today (20th May) I received a response by email (see https://stefanos.cloud/blog/kb/how-to-resolve-error-group-policy-client-service-failed-the-logon-access-denied-in-citrix-and-fslogix-environments/). In essence, the case was closed and an internal case is still open in the FsLogix product group without an ETA for providing a fix. Clearly the support team and product group have not realized the significant impact this has on multiple customers and environments.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.
bjkoekkoek 1 Reputation point

2021-03-16T07:32:08.713+00:00

We are encountering the same issues with the locked credentials folder in the local_ user folder on windows server 2019 and the latest version of FSLogix.
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.
Mihail Gusev 1 Reputation point

2021-03-16T08:25:34.39+00:00

Hi,
I had the same problem with locked vhdx, but now it seems to be working fine. My RDS farm has been working for 1,5 month without the issue. I have several On-Prem RDSH servers besed on Server 2019 Std with latest FSlogix agent 2.9.7654.46150.

I faced the issue with locked vhdx after I updated FSLogix agent from 2.9.7349.30108 to 2.9.7621.30127. To reslove the issue I turned on CleanupInvalidSessions reg key and updated FSLogix agent to 2.9.7654.46150. After that I haven't seen the issue with locked vhdx.

I've opened a case in MS support, but they didn't help me, becasue the issue gone away.

I want to add, that I also have a problem with undeleted local_username folders. I've been seeing it from the first installation of RDS farm with fslogix 2.9.7349.30108. I don't know if it makes bad influence on user experience or not.
Please sign in to rate this answer.
Stefanos Evangelou 116 Reputation points

2021-03-16T14:05:54.717+00:00

Hello,

With regards to the local_username profiles, yes it does have a blocking impact on the end user experience. If you read throughout this thread, you will see that many users are facing the same gpclient error at Citrix or RDS logon and this blocks the user from signing into the desktop.

Mihail Gusev 1 Reputation point

2021-03-16T14:26:37.983+00:00

Luckily, I haven't seen this problem in my infratructure

Mihail Gusev 1 Reputation point

2021-04-06T06:55:27.657+00:00

My happiness was short-lived, the problem appeared again: locked vhdx profile disk file and opened Credentials folders with completely logged off user. User received error "Group Policy Client service failed the logon. Access denied." I had to reboot the server to fix it.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.
Dlo 96 Reputation points

2021-03-17T11:01:14.847+00:00

So we finally got something small from Microsoft I know we all dont have the same issue and mines a little different. But this is what I got back so far.

Hope you are doing good!
I got an update on this:
Even though the feature is not enabled, logging is still on:
Please Turn off the logging for ODFC containers and monitor the behavior.

I asked what does that have to do with the issue and then she replied with...

Hello David,
Thanks for your response.
This is accordance with the Outlook Logs and Windows Search that were checked that day in the meeting.
Also, if you enable ODFC along with Profile logs, it collects same data twice and stores twice which may hamper the Performance.
In any way, you have not enabled ODFC, however, Logging for ODFC is enabled.
It is recommended to disable logging for the ODFC and monitor the behavior.
Please sign in to rate this answer.
Dlo 96 Reputation points

2021-03-17T11:02:32.843+00:00

This may be more for the local folders then the VHD issue per say but maybe it can help a few people out. I have not tried this yet as I have to do change control and stuff and not sure if disabling it needs a reboot or not. If it needs reboots then I have to wait a few days till we do some reboots.

Dlo 96 Reputation points

2021-03-17T11:30:28.043+00:00

Also we have Office 365 Containers completely disabled in a GPO and its still logging. I am looking for the key to disable logging now.

bjkoekkoek 1 Reputation point

2021-03-17T12:17:33.49+00:00

I assume this can only be a fix if you are not using the ODFC containers... So no fix for when you are using this for Onedrive...

Dlo 96 Reputation points

2021-03-17T12:20:20.667+00:00

I would agree we only use the profile container which does the whole user profile.
We don't use the Office 365 container

But this is also only the logging its not disabling the whole feature per say we just don't use the feature.

bjkoekkoek 1 Reputation point

2021-03-17T12:22:56.447+00:00

Maybe we could try disabling that logging. Will have a look on that tomorrow if that makes a difference.

bjkoekkoek 1 Reputation point

2021-03-18T14:57:02.547+00:00

I disabled the ODFC logging on our development machines. Very curious what the behavior will be.

Dlo 96 Reputation points

2021-03-18T15:00:10.253+00:00

We are seeing odd behaviors to once I disabled that log I was digging more into things.

So we have those containers disabled for one so logging shouldnt of been happening. But after disabling logging we were still seeing stuff in the logs like the outlook stuff trying to attach but then saying oh I am disabled. Not sure why it was trying to do stuff even tho it was disabled. So we disabled more stuff and are testing now as well.

bjkoekkoek 1 Reputation point

2021-03-19T13:56:46.353+00:00

I used the following:
"LoggingEnabled"=dword:00000001
"ODFC"=dword:00000000
"ConfigTool"=dword:00000001
"Profile"=dword:00000001
"Search"=dword:00000001

I noticed that when only LoggingEnabled was set the default values of the specific logging items were not used. After specifically enabling/disabling other parts of logging using these registry keys it looks fine in the frxtray.

So far I haven't seen left over local_ folders in c:\Users, but it wasn't a very busy day today.

Adam Gemberling 1 Reputation point

2021-03-24T09:32:49.647+00:00

Any update on the results for these changes? I have a small dev environment through which I have been testing but unfortunately have been failing to replicate the problem outside of production on any version of changes.

Curious to know if either of you have seen an improvement.

bjkoekkoek 1 Reputation point

2021-03-24T10:18:47.75+00:00

Nope no improvements on my end so far...
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.
CitrixMember 6 Reputation points

2021-03-26T10:42:49.977+00:00

It seems that Microsoft working on the problem as a bug, and we have to wait and see what they find out.
For those of you experiencing this in Citrix, have you tried this out?
May this be the performance hamper and might cause fslogix svc timeout?

https://support.citrix.com/article/CTX279517

We're testing this out now.
From earlier perspective we have excluded chrome and other things from Ctxhooks, which has improved the performance for the specific application.
Please sign in to rate this answer.
bjkoekkoek 1 Reputation point

2021-03-26T12:10:40.067+00:00

Looking forward to your test results!

CitrixMember 6 Reputation points

2021-04-06T13:45:00.41+00:00

Still problem unfornately.
It also seems Microsoft working with the bug,without finding the bottleneck yet.
We wait and see what MS Engineering find out.

It would be highly appreciated finding a permanent solution for this now.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

FsLogix - Unclean logoff causing locked files until server reboot

107 answers

Your answer