How to use AI Search for Structured & Unstructured data combined
Hello,
I'm developing an AI chatbot that answers user queries based on information in pre-uploaded documents. The user has the option to upload files which get stored in Azure Storage Account.
Here are the steps I have followed and this worked.
- Created a Container in Storage Account.
- Created a Search Index, a Data source, and an Indexer.
- When a file is uploaded from the front-end, the app will store it in the Container and run the Indexer.
All the uploaded docs are unstructured data, i.e., of type .pdf, .docx, .txt, and that's why this approach worked.
Now, I need to allow search on structured data files like .csv & .xlsx.
I know that for structured data, I need to create custom fields in the index. But the challenge here is, as the end user uploads those files, the schema is unknown. Also, the schema in each file may vary.
The entire uploaded documents set may be related (example: Sales_Data_May.xlsx, Sales_Report_May.docx, Sales_Targets_May.pdf), or they may all be completely irrelevant to each other.
The end user will have only one chat page. For the query asked, the system should search only the relevant documents for the answer.
Could anyone please help me with an overview & high level steps on the approach to follow?
Should I create one Search Index or multiple? Is there any other better suited service like Document Intelligence?
Thank you