High Queued Items and RedisTimeoutException in .NET 9 Application - Potential Thread Starvation?

Thanks for reaching out. The situation is a classic sign of .NET ThreadPool starvation of severe ThreadPool throttling, it's specifically impacting the ability of the StackExchange.Redis continuation to run promptly , even though the total number of free worker threads (32742) it's high.

Here is the breakdown of your question and the recommended steps

1.      Given the large number of queued items, could this be an indication of Thread Pool starvation, despite the availability of IOCP and worker threads?

Yes , this is a strong indication of thread pool throttling leading to “thread starvation” for the queued work.

·         The crucial detail here is the comparison of busy threads versus min threads delete that:

o   IOCP: Busy =0, Min=1 (No issue here, the IOCP pool is for low-level I/O operations).

o   Worker: Busy=25 , Min=16, since Busy (25) is greater than Min (16), the .NET Thread Pool is in a throttled state where it has to create new threads slowly (typically one thread per 500 ms).

·         StackExchange.Redis Heavily relies on the dot net thread pool two execute continuation of its asynchronous operation (the code that runs after the I/O for the Redis command completes).

·         When the thread pool is throttled, the massive queue of QueuedItems (2996460) the represent millions of asynchronous continuations (tasks) Waiting for an available thread to run. Since new threads are created slowly and existing threads might be blower busy these tasks piles up, causing the RedisTimeoutException as the 6000ms timeout is hit while the command’s continuation is still waiting in the queue.

2.      Is there a potential issue with Task queuing in the application or thread pool exhaustion that we are missing?

The issue is not thread pool exhaustion (you have many free threads,32742) but rather thread pool throttling and task queuing due to low Min setting.

The main reason for this vast queue are likely:

·         Sync-over-Async Abuse: The single most common cause is the use of synchronous blocking calls (e.g., Result, .Wait()) on asynchronous code paths. This blocks a ThreadPool worker thread indefinitely until the Redis operation completes, forcing the ThreadPool to spawn a new thread (slowly, due to throttling) to handle the next request. Check your code for StackExchange.Redis calls that end in .Result or .Wait().

·         Long Running CPU- bound Work: Worker threads are being held up by a long running synchronous code (CPU or I/O bound) that should either be optimized or explicitly offloaded using Task.Run() (though this can increase pressure).

·         Low Min Worker Threads: The low default Min =16 on a potentially multi-core machine is too small for a high-throughput ASP.NET or background service, causing the throttling to kick in frequently.

3.      How can we determine whether the Redis client (StackExchange.Redis) or our code is the culprit for this queuing issue?

The StackExchange.Redis library itself is usually non-blocking and efficient. The problem is almost certainly your application code interacting With the thread pool, specifically by blocking worker threads.

Two definitively determine the culprit:

·         code review(immediate action): Scrutinize  all Code paths that call StackExchange.Redis methods. Ensure you are using the  Async  methods(e.g., db.HashGetAsync()) And are awaiting them all the way up the call  stack to the entry point (e.g., controller method, BackgroundService ExecuteAsync). Avoid .Result or .Wait() at all costs.

·         Profiling( Deeper Analysis): Use a profiler like Visual Studio Profiler or dotnet trace To capture a trace  during the timeout event. Analyze the call stacks of the 25 busy worker threads. This will reveal exactly which lines of your code are holding those threads, which will likely point to the synchronous blocking calls or long running work.

Mitigation and configuration updates

4.      Are there any specific configuration updates or diagnostics we should perform to mitigate this issue? The application is background service. So does it help setting ThreadPool.SetMinThreads?

Yes, setting ThreaPool.SetMinThreads Is the most critical and immediate step to mitigate specific throttling issue

 configuration update:

ThreadPool.SetMinThreads

Set the minimum number of worker and IOCP threads at the very start of your application (e.g., in your background service’s host configuration or main method):

             //recommended starting point for a high throughput application/service

             //adjust based on your server’s core count and load testing.

            //a common starting point is 50x to 100x the number of cores, or based on the observed “busy” count.

           Int minWorkerThreads=200; //Increase this significantly from the default 16

           Int minIOThreads=200; //often kept in sync with worker threads

           System.Threading.ThreadPool.SetMinThreads (minWorkerThreads, minIOThreads);




  

Why this helps:

By increasing Min (e.g., from 16 to 200) the thread pool will immediately allocate up to 200 worker threads where needed, without hitting the throttling delay( 500ms per thread) as soon as busy count exceeds 16. This allow StackExchange.Redis’s continuations to find a thread and execute much faster, resolving the backlog of QueuedItems before the 6000ms time out is reached.

Let me know if you need any further help with this. We'll be happy to assist.

If you find this helpful, please mark this as answered.