is it possible to use Azure video indexer to explain the video content and quickly locate the video clip?

Question

is it possible to use Azure video indexer to explain the video content and quickly locate the video clip?

Yu Cai 100

Hello, may I ask is it possible to use Azure video indexer to explain the video content? For example, I want to use it to identify the car accident, monitor the grocery store surveillance video et al. How can I specifically train Azure video indexer to do this job? For instance, in below video, Azure video indexer mistake the dog as the bicycle although I mentioned dog in the tag list,. However, can I fine tuning video indexer so that it will not mistake the dog as a bicycle?

User's image

In the https://free.blessedness.top/en-us/azure/azure-video-indexer/customize-brands-model-overview, I do not find the details how to do it? If Microsoft provide a video instruction how to use these tools, it would be easier for us.

User's image

Yu Cai 100 Reputation points

2023-11-27T21:39:24.53+00:00

For the video data I processed, there is no accompanying audio data, only pure video data. To use Azure video indexer for tasks to monitor the grocery story such as shop-lifting, how can I train my data for this task using Azure video indexer? Could I add tags or text description for a specific range of video? I saw Azure video indexer allow to add tags or text descriptions for video. But how to train them?
VasaviLankipalle-MSFT 18,706 Reputation points Moderator

2023-11-27T22:27:38.94+00:00

Hello @Yu Cai , Thanks for using Microsoft Q&A Platform.

Regarding your question related to Customize content Model please visit this documentation to know more about these features: https://free.blessedness.top/en-us/azure/azure-video-indexer/customize-person-model-with-website

Also please view this document to learn about the View and update transcriptions and to view and edit insights.
Yu Cai 100 Reputation points

2023-11-27T22:40:05.4166667+00:00

Thank you so much. Based on my search, I do not find the tutorial how to train the Azure video indexer customer model to detect the tasks such as finding shop-lifting. I think this task need to train the Azure video indexer. How can I train it, I can add tags and text description to the video when I upload it. But how to train it? It looks that I do not find the answer in the manual.
Yu Cai 100 Reputation points

2023-11-28T00:31:16.97+00:00

Hello, when you mentioned View and update Transcriptions, I think I can try to add the description text to label my video. However, I cannot edit it although I click "Edit" button. It is not what the tutorial mention. For example, in 00:09 time point, a person show up, but the AI generated time line does not detect the person. I want to label this time point as "person show up". However, it does not allow me to change.

I also want to edit "bicycle", which is wrong. I need to label it as "dog", it does not allow me to do it.

Do you know why? If I can label them correctly, how can I train this data using Microsoft Azure indexer? Too many questions that need the help.
Yu Cai 100 Reputation points

2023-11-29T05:53:32.4066667+00:00

Are there any comments?
Yu Cai 100 Reputation points

2023-11-29T18:19:30.5133333+00:00

I click the Labels, now I can see label, but it is only allowed me to delete it, do allow me to add the label.

1 answer

Your answer

Yu Cai 100 Reputation points

2023-11-27T21:39:24.53+00:00

For the video data I processed, there is no accompanying audio data, only pure video data. To use Azure video indexer for tasks to monitor the grocery story such as shop-lifting, how can I train my data for this task using Azure video indexer? Could I add tags or text description for a specific range of video? I saw Azure video indexer allow to add tags or text descriptions for video. But how to train them?
VasaviLankipalle-MSFT 18,706 Reputation points Moderator

2023-11-27T22:27:38.94+00:00

Hello @Yu Cai , Thanks for using Microsoft Q&A Platform.

Regarding your question related to Customize content Model please visit this documentation to know more about these features: https://free.blessedness.top/en-us/azure/azure-video-indexer/customize-person-model-with-website

Also please view this document to learn about the View and update transcriptions and to view and edit insights.
Yu Cai 100 Reputation points

2023-11-27T22:40:05.4166667+00:00

Thank you so much. Based on my search, I do not find the tutorial how to train the Azure video indexer customer model to detect the tasks such as finding shop-lifting. I think this task need to train the Azure video indexer. How can I train it, I can add tags and text description to the video when I upload it. But how to train it? It looks that I do not find the answer in the manual.
Yu Cai 100 Reputation points

2023-11-28T00:31:16.97+00:00

Hello, when you mentioned View and update Transcriptions, I think I can try to add the description text to label my video. However, I cannot edit it although I click "Edit" button. It is not what the tutorial mention. For example, in 00:09 time point, a person show up, but the AI generated time line does not detect the person. I want to label this time point as "person show up". However, it does not allow me to change.

I also want to edit "bicycle", which is wrong. I need to label it as "dog", it does not allow me to do it.

Do you know why? If I can label them correctly, how can I train this data using Microsoft Azure indexer? Too many questions that need the help.
Yu Cai 100 Reputation points

2023-11-29T05:53:32.4066667+00:00

Are there any comments?
Yu Cai 100 Reputation points

2023-11-29T18:19:30.5133333+00:00

I click the Labels, now I can see label, but it is only allowed me to delete it, do allow me to add the label.

Answer 1

Hello Yu Cai,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

I understand that you are looking for possible ways to use Azure video indexer to explain the video content and quickly locate the video clip.

Azure Video Indexer is not designed for direct training or fine-tuning, but you can extend its capabilities using custom models, Logic Apps, and OpenAI. For tasks like shoplifting detection, you must build a custom pipeline using AVI for indexing and your own model for classification. AVI is not a trainable model in the traditional ML sense. It uses pre-trained models for object detection, speech recognition, and sentiment analysis. You cannot fine-tune these models directly. However, you can overcome AVI’s limitations by integrate your own custom model using Azure Logic Apps and Azure OpenAI or Azure AI Computer Vision using the following steps:

Index the video using AVI.
Extract frames or object metadata using AVI API.
Send frames to your custom model (hosted on Azure AI).
Classify objects/actions using your model (e.g., detect shoplifting).
Patch AVI insights with corrected labels via API.

The links here will give you more details on the above steps: https://github.com/Azure-Samples/azure-video-indexer-samples/blob/master/BringYourOwn-Samples/README.MD and https://www.youtube.com/watch?v=yMqJufR9Rfs to extend its capabilities using custom models, Logic Apps, and OpenAI.

I hope this is helpful! Do not hesitate to let me know if you have any other questions or clarifications.

Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful.

Share via

is it possible to use Azure video indexer to explain the video content and quickly locate the video clip?

1 answer

Your answer