Custom vision training been stuck for 2+ weeks?!

Matt 0 Reputation points
2025-09-18T06:59:30.1766667+00:00

Only used 4 hours of training budget and that was 2 weeks ago. Still training. No option to pause/reset etc.

Really need this fixed. When is a fix rolling out?

Azure AI Custom Vision
Azure AI Custom Vision
An Azure artificial intelligence service and end-to-end platform for applying computer vision to specific domains.
0 comments No comments
{count} votes

2 answers

Sort by: Most helpful
  1. Alex Burlachenko 18,310 Reputation points Volunteer Moderator
    2025-09-18T08:24:03.6566667+00:00

    this is a known service-side issue that sometimes happens with custom vision. the training process gets stuck in a loop or hangs on a backend error. since you cannot cancel or reset it, the only real fix is to contact azure support directly.

    you need to open a support ticket for your custom vision project. be sure to give them your project id, project name, and the region where your custom vision resource is located. they have tools on their end to force-stop the training and get your project back to a usable state.

    while you wait for them, its worth trying to create a brand new project and resource in a different azure region. sometimes a fresh start in a new location can bypass whatever glitch is happening. just use a small subset of your data to test if the new project trains correctly.

    also, check the service health status in your azure portal. there might be an ongoing incident affecting custom vision that is causing these delays.

    really hope they can get this resolved for you quickly. waiting that long is completely unacceptable.

    Best regards,

    Alex

    and "yes" if you would follow me at Q&A - personaly thx.
    P.S. If my answer help to you, please Accept my answer
    

    https://ctrlaltdel.blog/

    0 comments No comments

  2. Anshika Varshney 1,910 Reputation points Microsoft External Staff Moderator
    2025-09-23T10:20:19.3933333+00:00

    Hello Matt,

    Thank you for reaching out on the Microsoft Q&A.

    You are asking about Custom Vision training job has been stuck in “Training…” for more than two weeks, even though you’ve only used about 4 hours of your training budget. There’s also no option to pause or reset, and you’d like to know when this will be fixed.

    From recent updates and community discussions, this isn’t an isolated case. Other users have also reported jobs getting stuck for days or weeks. The most common reasons seem to be high demand on training resources or limited GPU availability, especially during peak times. There’s no confirmed timeline for a fix, but some users were able to work around the issue by deleting the stuck iteration through the REST API and starting a new training run.

    Custom Vision is still in preview, and Microsoft has shared that improvements are on the roadmap, including object recognition and edge deployment. In the meantime, if your job remains stuck, trying the REST API to remove the current iteration and then retraining is a practical option. It may also help to check Azure Service Health for any known issues that could be affecting availability.

    Some documents for your References:

    If you feel that your quires have been resolved, please accept the answer by clicking the "Upvote" and "Accept Answer" on the post.

    Thank you!


Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.