Azure OpenAI Integration for LLM Cost Tracker

Overview

Binadox LLM Cost Tracker provides comprehensive monitoring and cost analysis for your Azure OpenAI resources. This integration allows you to track usage patterns, monitor costs, analyze performance metrics, and optimize your AI spending across all Azure OpenAI deployments.

The LLM Cost Tracker provides detailed insights into:

Token usage and API call volumes
Cost breakdown by model and deployment
Usage trends and patterns
Resource utilization optimization recommendations

Prerequisites

Before setting up the integration, ensure you have:

Azure subscription with Azure OpenAI resources deployed
Azure CLI installed or access to Azure Portal
Administrative permissions in Azure to create service principals and assign roles
Binadox account

Authentication Method

This guide uses Service Principal authentication, which is the recommended approach for production environments. Service Principal provides:

Secure, token-based authentication
Fine-grained permission control
Full access to Azure Monitor and Cost Management APIs
Automated resource discovery across resource groups
Scalability for multiple Azure OpenAI resources

Step 1: Create Azure Service Principal

Method 1 (Recommended): Using Azure CLI

1. Login to Azure and set your subscription:

# Login to Azure
az login

# List available subscriptions
az account list --output table

# Set your target subscription
az account set --subscription "Your Subscription Name"

2. Get your subscription ID

az account show --query id --output tsv

Save this subscription ID for later configuration.

3. Create the service principal

az ad sp create-for-rbac \
    --name "Binadox-LLM-Cost-Tracker" \
    --role "Reader" \
    --scopes "/subscriptions/YOUR_SUBSCRIPTION_ID"

4. Save the output securely: The command will return credentials in this format

{
  "appId": "12345678-1234-1234-1234-123456789012",        # client_id
  "displayName": "Binadox-LLM-Cost-Tracker",
  "password": "87654321-4321-4321-4321-210987654321",     # client_secret
  "tenant": "11111111-1111-1111-1111-111111111111"        # tenant_id
}

Important: Save these values immediately. The password (client_secret) cannot be retrieved later.

Method 1: Using Azure Portal

Navigate to Azure Active Directory → App registrations
Click New registration
Enter name: “Binadox-LLM-Cost-Tracker”
Select Accounts in this organizational directory only
Click Register
Note the Application (client) ID and Directory (tenant) ID
Go to Certificates & secrets → New client secret
Add description “Binadox LLM Cost Tracker” and select expiry period
Click Add and copy the secret value immediately

Step 2: Assign Required Permissions

The service principal needs specific Azure RBAC roles to access Azure OpenAI monitoring data:

1. Reader Role (for resource discovery)

az role assignment create \
    --assignee "YOUR_SERVICE_PRINCIPAL_APP_ID" \
    --role "Reader" \
    --scope "/subscriptions/YOUR_SUBSCRIPTION_ID"

2. Monitoring Reader Role (for metrics and logs)

az role assignment create \
    --assignee "YOUR_SERVICE_PRINCIPAL_APP_ID" \
    --role "Monitoring Reader" \
    --scope "/subscriptions/YOUR_SUBSCRIPTION_ID"

3. Cost Management Reader Role (for cost data)

az role assignment create \
    --assignee "YOUR_SERVICE_PRINCIPAL_APP_ID" \
    --role "Cost Management Reader" \
    --scope "/subscriptions/YOUR_SUBSCRIPTION_ID"

4. Log Analytics Reader Role (optional, for detailed logs)

az role assignment create \
    --assignee "YOUR_SERVICE_PRINCIPAL_APP_ID" \
    --role "Log Analytics Reader" \
    --scope "/subscriptions/YOUR_SUBSCRIPTION_ID"

Note: Role assignments may take 5-10 minutes to propagate across Azure services.

Step 3: Verify Service Principal Setup

Test your service principal configuration:

# Login with service principal
az login --service-principal \
    --username "YOUR_CLIENT_ID" \
    --password "YOUR_CLIENT_SECRET" \
    --tenant "YOUR_TENANT_ID"

# Test access to Cognitive Services
az cognitiveservices account list

# Test access to metrics (replace with your actual resource details)
az monitor metrics list \
    --resource "/subscriptions/YOUR_SUBSCRIPTION_ID/resourceGroups/YOUR_RG/providers/Microsoft.CognitiveServices/accounts/YOUR_OPENAI_RESOURCE" \
    --metric-names "Calls"

Step 4: Configure LLM Cost Tracker in Binadox

1. Access LLM Cost Tracker:

Log into your Binadox dashboard
Navigate to LLM Cost Tracker section
Click Add New Integration

2. Select Azure OpenAI from the available providers

3. Fill in the following information from your service principal

Field	Value	Description
Connection Name	Azure OpenAI Production	Descriptive name for this integration
Tenant ID	Your tenant ID	Directory (tenant) ID from service principal
Client ID	Your client ID	Application (client) ID from service principal
Client Secret	Your client secret	Secret value from service principal
Subscription ID	Your subscription ID	Azure subscription containing OpenAI resources

Troubleshooting

Common Issues and Solutions

Issue	Symptoms	Solutions
401 Unauthorized	Connection test fails with authentication error	Verify client_id, client_secret, and tenant_id are correct; Check if the client secret has expired in Azure Portal; Ensure you’re using the correct Azure tenant.
403 Forbidden	Authentication succeeds but data collection fails	Verify all required role assignments are in place; Wait 10-15 minutes for role assignments to propagate; Check that roles are assigned at the subscription level, not resource level.
No Azure OpenAI Resources Found	Integration succeeds but no resources are discovered	Verify Azure OpenAI resources exist in the specified subscription; Check resource group filters if configured; Ensure the service principal has Reader role on the subscription.
Missing Cost Data	Usage metrics appear but cost data is missing	Verify “Cost Management Reader” role is assigned; Cost data may take 24-48 hours to appear for new resources; Check that Azure OpenAI resources have generated billing data.
Incomplete Metrics Data	Some metrics are missing or inconsistent	Verify “Monitoring Reader” role is assigned; Ensure Azure OpenAI resources have active deployments; Check that diagnostic settings are enabled for detailed metrics.

Validating Permissions

Check current role assignments for your service principal:

# List all role assignments
az role assignment list \
    --assignee "YOUR_SERVICE_PRINCIPAL_APP_ID" \
    --output table

# Verify specific resource access
az cognitiveservices account show \
    --name "YOUR_OPENAI_RESOURCE" \
    --resource-group "YOUR_RG"

Testing API Access

Verify service principal can access required APIs:

# Test Azure Monitor access
az monitor metrics list \
    --resource "/subscriptions/SUB_ID/resourceGroups/RG_NAME/providers/Microsoft.CognitiveServices/accounts/RESOURCE_NAME" \
    --metric-names "Calls,TokenTransaction,DataIn,DataOut"

# Test Cost Management access
az costmanagement query \
    --type "Usage" \
    --scope "/subscriptions/YOUR_SUBSCRIPTION_ID" \
    --time-period from="2024-01-01" to="2024-01-31"

Support and Resources

Binadox Support: Contact support through your Binadox dashboard
Azure OpenAI Documentation: Azure OpenAI Service Documentation
Azure Monitor Documentation: Azure Monitor Documentation
Cost Management Documentation: Azure Cost Management

For additional assistance or feature requests related to LLM Cost Tracker, please reach out to the Binadox support team.