Share via


AIFoundryModel.Microsoft Class

Definition

Models published by Microsoft.

public static class AIFoundryModel.Microsoft
type AIFoundryModel.Microsoft = class
Public Class AIFoundryModel.Microsoft
Inheritance
AIFoundryModel.Microsoft

Fields

AzureAiContentSafety

Azure AI Content Safety ## Introduction Azure AI Content Safety is a safety system for monitoring content generated by both foundation models and humans. Detect and block potential risks, threats, and quality problems. You can build an advanced safety system for foundation models to detect and mitigate harmful content and risks in user prompts and AI-generated outputs. Use Prompt Shields to detect and block prompt injection attacks, groundedness detection to pinpoint ungrounded or hallucinated materials, and protected material detection to identify copyrighted or owned content. ## Core Features - Block harmful input and output - Description: Detect and block violence, hate, sexual, and self-harm content for both text, images and multimodal. Configure severity thresholds for your specific use case and adhere to your responsible AI policies. - Key Features: Violence, hate, sexual, and self-harm content detection. Custom blocklist. - Policy customization with custom categories - Description: Create unique content filters tailored to your requirements using custom categories. Quickly train a new custom category by providing examples of content you need to block. - Key Features: Custom categories - Identify the security risks - Description: Safeguard your AI applications against prompt injection attacks and jailbreak attempts. Identify and mitigate both direct and indirect threats with prompt shields. - Key Features: Direct jailbreak attack, indirect prompt injection from docs. - Detect and correct Gen AI hallucinations - Description: Identify and correct generative AI hallucinations and ensure outputs are reliable, accurate, and grounded in data with groundedness detection. - Key Features: Groundedness detection, reasoning, and correction. - Identify protected material - Description: Pinpoint copyrighted content and provide sources for preexisting text and code with protected material detection. - Key Features: Protected material for code, protected material for text ## Use Cases - Generative AI services screen user-submitted prompts and generated outputs to ensure safe and appropriate content. - Online marketplaces monitor and filter product listings and other user-generated content to prevent harmful or inappropriate material. - Gaming platforms manage and moderate user-created game content and in-game communication to maintain a safe environment. - Social media platforms review and regulate user-uploaded images and posts to enforce community standards and prevent harmful content. - Enterprise media companies implement centralized content moderation systems to ensure the safety and appropriateness of their published materials. - K-12 educational technology providers filter out potentially harmful or inappropriate content to create a safe learning environment for students and educators. ## Benefits - No ML experience required: Incorporate content safety features into your projects with no machine learning experience required. - Effortlessly customize your RAI policies: Customizing your content safety classifiers can be done with one line of description, a few samples using Custom Categories. - State of the art models: ready for use APIs, SOTA models, and flexible deployment options reduce the need for ongoing manual training or extensive customization. Microsoft has a science team and policy experts working on the frontier of Gen AI to constantly improve the safety and security models to ensure our customers can develop and deploy generative AI safely and responsibly. - Global Reach: Support more than 100 languages, enabling businesses to communicate effectively with customers, partners, and employees worldwide. - Scalable and Reliable: Built on Azure’s cloud infrastructure, the Azure AI Content Safety service scales automatically to meet demand, from small business applications to global enterprise workloads. - Security and Compliance: Azure AI Content Safety runs on Azure’s secure cloud infrastructure, ensuring data privacy and compliance with global standards. User data is not stored after the translation process. - Flexible deployment: Azure AI Content Safety can be deployed on cloud, on premises and on devices. ## Technical Details - Deployment - Container for on-premise deployment: Content safety containers overview - Azure AI Content Safety - Azure AI services | Microsoft Learn - Embedded Content Safety: Embedded Content Safety - Azure AI Content Safety - Azure AI services | Microsoft Learn - Cloud: Azure AI Content Safety documentation - Quickstarts, Tutorials, API Reference - Azure AI services | Microsoft Learn - Requirements: Requirements vary feature by feature, for more details, refer to the Azure AI Content Safety documentation: Azure AI Content Safety documentation - Quickstarts, Tutorials, API Reference - Azure AI services | Microsoft Learn. - Support: Azure AI Content Safety is part of Azure AI Services. Support options for AI Services can be found here: Azure AI services support and help options - Azure AI services | Microsoft Learn. ## Pricing Explore pricing options here: Azure AI Content Safety - Pricing | Microsoft Azure.

AzureAiContentUnderstanding

Azure AI Content Understanding ## Introduction Azure AI Content Understanding empowers you to transform unstructured multimodal data—such as text, images, audio, and video—into structured, actionable insights. By streamlining content processing with advanced AI techniques like schema extraction and grounding, it delivers accurate structured data for downstream applications. Offering prebuilt templates for common use cases and customizable models, it helps you unify diverse data types into a single, efficient pipeline, optimizing workflows and accelerating time to value. ## Core Features - Multimodal data ingestion Ingest a range of modalities such as documents, images, audio, or video. Use a variety of AI models to convert the input data into a structured format that can be easily processed and analyzed by downstream services or applications. - Customizable output schemas Customize the schemas of extracted results to meet your specific needs. Tailor the format and structure of summaries, insights, or features to include only the most relevant details—such as key points or timestamps—from video or audio files. - Confidence scores Leverage confidence scores to minimize human intervention and continuously improve accuracy through user feedback. - Output ready for downstream applications Automate business processes by building enterprise AI apps or agentic workflows. Use outputs that downstream applications can consume for reasoning with retrieval-augmented generation (RAG). - Grounding Ensure the information extracted, inferred, or abstracted is represented in the underlying content. - Automatic labeling Save time and effort on manual annotation and create models quicker by using large language models (LLMs) to extract fields from various document types. ## Use Cases - Post-call analytics for call centers: Generate insights from call recordings, track key performance indicators (KPIs), and answer customer questions more accurately and efficiently. - Tax process automation: Streamline the tax return process by extracting data from tax forms to create a consolidated view of information across various documents. - Media asset management: Extract features from images and videos to provide richer tools for targeted content and enhance media asset management solutions. - Chart understanding: Enhance chart understanding by automating the analysis and interpretation of various types of charts and diagrams using Content Understanding. ## Benefits - Streamline workflows: Azure AI Content Understanding standardizes the extraction of content, structure, and insights from various content types into a unified process. - Simplify field extraction: Field extraction in Content Understanding makes it easier to generate structured output from unstructured content. Define a schema to extract, classify, or generate field values with no complex prompt engineering. - Enhance accuracy: Content Understanding employs multiple AI models to analyze and cross-validate information simultaneously, resulting in more accurate and reliable results. - Confidence scores & grounding: Content Understanding ensures the accuracy of extracted values while minimizing the cost of human review. ## Technical Details - Deployment: Deployment options may vary by service, reference the following docs for more information: Create an Azure AI Services multi-service resource. - Requirements: Requirements may vary depending on the input data you are analyzing, reference the following docs for more information: Service quotas and limits. - Support: Support options for AI Services can be found here: Azure AI services support and help options. ## Pricing View up-to-date pay-as-you-go pricing details here: Azure AI Content Understanding pricing.

AzureAiDocumentIntelligence

Azure AI Document Intelligence Document Intelligence is a cloud-based service that enables you to build intelligent document processing solutions. Massive amounts of data, spanning a wide variety of data types, are stored in forms and documents. Document Intelligence enables you to effectively manage the velocity at which data is collected and processed and is key to improved operations, informed data-driven decisions, and enlightened innovation. ## Core Features - General extraction models - Description: General extraction models enable text extraction from forms and documents and return structured business-ready content ready for your organization's action, use, or development. - Key Features - Read model allows you to extract written or printed text liens, words, locations, and detected languages. - Layout model, on top of text extraction, extracts structural information like tables, selection marks, paragraphs, titles, headings, and subheadings. Layout model can also output the extraction results in a Markdown format, enabling you to define your semantic chunking strategy based on provided building blocks, allowing for easier RAG (Retrieval Augmented Generation). - Prebuilt models - Description: Prebuilt models enable you to add intelligent document processing to your apps and flows without having to train and build your own models. Prebuilt models extract a pre-defined set of fields depending on the document type. - Key Features - Financial Services and Legal Documents: Credit Cards, Bank Statement, Pay Slip, Check, Invoices, Receipts, Contracts. - US Tax Documents: Unified Tax, W-2, 1099 Combo, 1040 (multiple variations), 1098 (multiple variations), 1099 (multiple variations). - US Mortgage Documents: 1003, 1004, 1005, 1008, Closing Disclosure. - Personal Identification Documents: Identity Documents, Health Insurance Cards, Marriage Certificates. - Custom models - Description: Custom models are trained using your labeled datasets to extract distinct data from forms and documents, specific to your use cases. Standalone custom models can be combined to create composed models. - Key Features - Document field extraction models - Custom generative: Build a custom extraction model using generative AI for documents with unstructured format and varying templates. - Custom neural: Extract data from mixed-type documents. - Custom template: Extract data from static layouts. - Custom composed: Extract data using a collection of models. Explicitly choose the classifier and enable confidence-based routing based on the threshold you set. - Custom classification models - Custom classifier: Identify designated document types (classes) before invoking an extraction model. - Add-on capabilities - Description: Use the add-on features to extend the results to include more features extracted from your documents. Some add-on features incur an extra cost. These optional features can be enabled and disabled depending on the scenario of the document extraction. - Key Features - High resolution extraction - Formula extraction - Font extraction - Barcode extraction - Language detection - Searchable PDF output ## Use Cases - Accounts payable: A company can increase the efficiency of its accounts payable clerks by using the prebuilt invoice model and custom forms to speed up invoice data entry with a human in the loop. The prebuilt invoice model can extract key fields, such as Invoice Total and Shipping Address. - Insurance form processing: A customer can train a model by using custom forms to extract a key-value pair in insurance forms and then feeds the data to their business flow to improve the accuracy and efficiency of their process. For their unique forms, customers can build their own model that extracts key values by using custom forms. These extracted values then become actionable data for various workflows within their business. - Bank form processing: A bank can use the prebuilt ID model and custom forms to speed up the data entry for "know your customer" documentation, or to speed up data entry for a mortgage packet. If a bank requires their customers to submit personal identification as part of a process, the prebuilt ID model can extract key values, such as Name and Document Number, speeding up the overall time for data entry. - Robotic process automation (RPA): Using the custom extraction model, customers can extract specific data needed from distinct types of documents. The key-value pair extracted can then be entered into various systems such as databases, or CRM systems, through RPA, replacing manual data entry. Customers can also use custom classification model to categorize documents based on their content and file them in proper location. As such, an organized set of data extracted from the custom model can be an essential first step to document RPA scenarios for businesses that manage large volumes of documents regularly. ## Benefits - No experience required: Incorporate Document Intelligence features into your projects with no machine learning experience required. - Effortlessly customize your models: Training your own custom extraction and classification model can be done with as little as one document labeled, making it easy to train your own models. - State of the art models: ready for use APIs, constantly enhanced models, and flexible deployment options reduce the need for ongoing manual training or extensive customization. ## Technical Details: - Deployment: Deployment options may vary by service, reference the following docs for more information: Use Document Intelligence models and Install and run containers. - Requirements: Requirements may vary slightly depending on the model you are using to analyze the documents. Reference the following docs for more information: Service quotas and limits. - Support: Support options for AI Services can be found here: Azure AI services support and help options - Azure AI services | Microsoft Learn. ## Pricing View up-to-date pricing information for the pay-as-you-go pricing model here: Azure AI Document Intelligence pricing.

AzureAiLanguage
AzureAiSpeech

Azure AI Speech ## Introduction The Speech service provides speech to text and text to speech capabilities with a Speech resource. You can transcribe speech to text with high accuracy, produce natural-sounding text to speech voices, translate spoken audio, and use speaker recognition during conversations. Create custom voices, add specific words to your base vocabulary, or build your own models. Run Speech anywhere, in the cloud or at the edge in containers. It's easy to speech enable your applications, tools, and devices with the Speech CLI, Speech SDK, and REST APIs. ## Core Features - Speech To Text - Description: Use speech to text to transcribe audio into text, either in real-time or asynchronously with batch transcription. Convert audio to text from a range of sources, including microphones, audio files, and blob storage. Use speaker diarization to determine who said what and when. Get readable transcripts with automatic formatting and punctuation. The base model might not be sufficient if the audio contains ambient noise or includes numerous industry and domain-specific jargon. In these cases, you can create and train custom speech models with acoustic, language, and pronunciation data. Custom speech models are private and can offer a competitive advantage. - Key Features - Real Time Speech To Text - Transcriptions, captions, or subtitles for live meetings - Diarization - Pronunciation assessment - Contact center agents assist - Dictation - Voice agents - Fast Transcription - Quick audio or video transcription, subtitles, and edit - Video translation - Batch Transcription - Transcriptions, captions, or subtitles for prerecorded audio - Contact center post-call analytics - Diarization - Custom Speech - Models with enhanced accuracy for specific domains and conditions. - Text To Speech - Description: With text to speech, you can convert input text into human like synthesized speech. Use human-like prebuilt neural voices out of the box in more than 140 locales and 500 voices or create a custom neural voice that's unique to your product or brand. You can also enhance the voice experience by using together with Text to speech Avatar to convert text to life-like and high-quality synthetic talking avatar videos. - Prebuilt neural voice: Highly natural out-of-the-box voices. Check the prebuilt neural voice samples the Voice Gallery and determine the right voice for your business needs. - Custom neural voice: Besides the prebuilt neural voices that come out of the box, you can also create a custom neural voice that is recognizable and unique to your brand or product. Custom neural voices are private and can offer a competitive advantage. Check the custom neural voice samples here. - Text to speech Avatar: You can convert text into a digital video of a photorealistic human (either a prebuilt avatar or a custom text to speech avatar) speaking with a natural-sounding voice. It works best with the Azure neural voices. - Key Features - Prebuilt neural voice - Neural voice (incl. OpenAI-based voices) - Neural HD voice (incl. OpenAI-based voices) - Custom neural voice - Professional voice - Personal voice - TTS Avatar - Prebuilt avatar - Custom avatar - Speech Translation - Description: Speech Translation enables real-time, multi-language translation of speech, allowing you to add end-to-end, real-time, multi-language translation capabilities to your applications, tools, and devices. - Key Features - Realtime Speech Translation: This Speech service supports real-time, multi-language speech to speech and speech to text translation of audio streams. - Support both audio and text output - Automatic language detection - Integrated customization built-in - Video Translation: This end-to-end solution performs video translation covering global locales. - End-to-end solution with both no-code and API support - GPT built-in to optimize the translation content, augmented by content editing - Personal voice (limited access) to keep the original timbre, emotions, intonation & style intact ## Use cases Speech To Text | Use case | Scenario | Solution | | :---: | :--- | :--- | | Live meeting transcriptions and captions | A virtual event platform needs to provide real-time captions for webinars. | Integrate real-time speech to text using the Speech SDK to transcribe spoken content into captions displayed live during the event. | | Customer service enhancement | A call center wants to assist agents by providing real-time transcriptions of customer calls. | Use real-time speech to text via the Speech CLI to transcribe calls, enabling agents to better understand and respond to customer queries. | | Video subtitling | A video-hosting platform wants to quickly generate a set of subtitles for a video. | Use fast transcription to quickly get a set of subtitles for the entire video. | | Educational tools | An e-learning platform aims to provide transcriptions for video lectures. | Apply batch transcription through the speech to text REST API to process prerecorded lecture videos, generating text transcripts for students. | | Healthcare documentation | A healthcare provider needs to document patient consultations. | Use real-time speech to text for dictation, allowing healthcare professionals to speak their notes and have them transcribed instantly. Use a custom model to enhance recognition of specific medical terms. | | Media and entertainment | A media company wants to create subtitles for a large archive of videos. | Use batch transcription to process the video files in bulk, generating accurate subtitles for each video. | | Market research | A market research firm needs to analyze customer feedback from audio recordings. | Employ batch transcription to convert audio feedback into text, enabling easier analysis and insights extraction. | Text To Speech | Use case | Scenario | | :---: | :--- | | Educational or interactive learning | To create a fictional brand or character voice for reading or speaking educational materials, online learning, interactive lesson plans, simulation learning, or guided museum tours. | | Media Entertainment | To create a fictional brand or character voice for reading or speaking entertainment content for video games, movies, TV, recorded music, podcasts, audio books, or augmented or virtual reality. | | Media Marketing | To create a fictional brand or character voice for reading or speaking marketing and product or service media, product introductions, business promotion, or advertisements. | | Self-authored content | To create a voice for reading content authored by the voice talent. | | Accessibility Features | For use in audio description systems and narration, including any fictional brand or character voice, or to facilitate communication by people with speech impairments. | | Interactive Voice Response (IVR) Systems | To create voices, including any fictional brand or character voice, for call center operations, telephony systems, or responses for phone interactions. | | Public Service and Informational Announcements | To create a fictional brand or character voice for communicating public service information, including announcements for public venues, or for informational broadcasts such as traffic, weather, event information, and schedules. This use case is not intended for journalistic or news content. | | Translation and Localization | For use in translation applications for translating conversations in different languages or translating audio media. | | Virtual Assistant or Chatbot | To create a fictional brand or character voice for smart assistants in or for virtual web assistants, appliances, cars, home appliances, toys, control of IoT devices, navigation systems, reading out personal messages, virtual companions, or customer service scenarios. | Speech Translation | Use case | Scenario | | :---: | :--- | | Realtime translated caption/subtitle | Realtime translated captions /subtitles for meetings or audio/video content | | Realtime audio/video translation (speech-to-speech) | Translate audio/video into target language audio. The input can be short-form videos, live broadcasts, online or in-person conversations (e.g., Live Interpreter), etc. | | Batch Video Translation | Automated dubbing of spoken content in videos from one language to another | ## Benefits Text To Speech - Global Reach with Extensive Locale and Voice Coverage: Azure TTS supports more than 140 languages and dialects, along with 400+ unique neural voices. Its widespread data center coverage across 60+ Azure regions makes it highly accessible globally, ensuring low-latency voice services in key markets across North America, Europe, Asia Pacific, and emerging markets in Africa, South America, and the Middle East. - Customization Capabilities: Azure’s Custom Neural Voice enables businesses to create unique, branded voices in a low/no code self-serving portal that can speak in specific accents or styles, reflecting a company’s identity. This customization extends to creating regional variants and accents, making Azure ideal for multinational corporations seeking to tailor voices to specific local audiences. - Flexible Deployment Options: Azure TTS can be deployed in the cloud, on-premises, or at the edge. - Security and Compliance: Azure offers end-to-end encryption, comprehensive compliance certifications (like GDPR, HIPAA, ISO 27001, SOC), and a strong focus on privacy. - TTS Avatar as a Differentiator: Azure’s TTS avatars, combined with Custom Neural Voice, create immersive, interactive virtual characters. This innovation allows businesses to integrate human-like avatars in customer service, e-learning, and entertainment, providing visually engaging interactions that go beyond simple audio output. Speech Translation - Multiple language detection: Model will detect multiple languages among the supported languages in the same audio stream. - Automatic language detection: No need to specify input languages – model will detect them automatically. - Integrated custom translation: Adapt model to your domain-specific vocabulary. - Simple & Quick: End-to-end solution that performs video translation covering global locales - High quality: GPT built-in to optimize the translation content, augmented by content editing - Personalized (Limited Access): Keep the original timbre, emotions, intonation & style intact ## Pricing Speech is available for many languages, regions, and price points.

AzureAiTranslator
AzureAiVision
MaiDsR1

MAI-DS-R1 is a DeepSeek-R1 reasoning model that has been post-trained by the Microsoft AI team to fill in information gaps in the previous version of the model and improve its harm protections while maintaining R1 reasoning capabilities.

ModelRouter

Model router is a deployable AI model that is trained to select the most suitable large language model (LLM) for a given prompt.

Phi4

Phi-4 14B, a highly capable model for low latency scenarios.

Phi4MiniInstruct

3.8B parameters Small Language Model outperforming larger models in reasoning, math, coding, and function-calling

Phi4MiniReasoning

Lightweight math reasoning model optimized for multi-step problem solving

Phi4Reasoning

State-of-the-art open-weight reasoning model.

Applies to