Azure OpenAI Integration for LLM Cost Tracker

Overview

Binadox LLM Cost Tracker provides comprehensive monitoring and cost analysis for your Azure OpenAI resources. This integration allows you to track usage patterns, monitor costs, analyze performance metrics, and optimize your AI spending across all Azure OpenAI deployments.

The LLM Cost Tracker provides detailed insights into:

• Token usage and API call volumes
• Cost breakdown by model and deployment
• Usage trends and patterns
• Resource utilization optimization recommendations

Prerequisites

Before setting up the integration, ensure you have:

• Azure subscription with Azure OpenAI resources deployed
• Azure CLI installed or access to Azure Portal
• Administrative permissions in Azure to create service principals and assign roles
• Binadox account 

Authentication Method

This guide uses Service Principal authentication, which is the recommended approach for production environments. Service Principal provides:

• Secure, token-based authentication
• Fine-grained permission control
• Full access to Azure Monitor and Cost Management APIs
• Automated resource discovery across resource groups
• Scalability for multiple Azure OpenAI resources

Step 1: Create Azure Service Principal

Method 1 (Recommended): Using Azure CLI

1. Login to Azure and set your subscription:

# Login to Azure
az login

# List available subscriptions
az account list --output table

# Set your target subscription
az account set --subscription "Your Subscription Name"

2. Get your subscription ID

az account show --query id --output tsv 

Save this subscription ID for later configuration.

3. Create the service principal

az ad sp create-for-rbac \
    --name "Binadox-LLM-Cost-Tracker" \
    --role "Reader" \
    --scopes "/subscriptions/YOUR_SUBSCRIPTION_ID"

4. Save the output securely: The command will return credentials in this format

{
  "appId": "12345678-1234-1234-1234-123456789012",        # client_id
  "displayName": "Binadox-LLM-Cost-Tracker",
  "password": "87654321-4321-4321-4321-210987654321",     # client_secret
  "tenant": "11111111-1111-1111-1111-111111111111"        # tenant_id
}

Important: Save these values immediately. The password (client_secret) cannot be retrieved later.

Method 1: Using Azure Portal

1. Navigate to Azure Active Directory → App registrations
2. Click New registration
3. Enter name: “Binadox-LLM-Cost-Tracker”
4. Select Accounts in this organizational directory only
5. Click Register
6. Note the Application (client) ID and Directory (tenant) ID
7. Go to Certificates & secrets → New client secret
8. Add description “Binadox LLM Cost Tracker” and select expiry period
9. Click Add and copy the secret value immediately

Step 2: Assign Required Permissions

The service principal needs specific Azure RBAC roles to access Azure OpenAI monitoring data:

1. Reader Role (for resource discovery)

az role assignment create \
    --assignee "YOUR_SERVICE_PRINCIPAL_APP_ID" \
    --role "Reader" \
    --scope "/subscriptions/YOUR_SUBSCRIPTION_ID"

2. Monitoring Reader Role (for metrics and logs)

az role assignment create \
    --assignee "YOUR_SERVICE_PRINCIPAL_APP_ID" \
    --role "Monitoring Reader" \
    --scope "/subscriptions/YOUR_SUBSCRIPTION_ID"

3. Cost Management Reader Role (for cost data)

az role assignment create \
    --assignee "YOUR_SERVICE_PRINCIPAL_APP_ID" \
    --role "Cost Management Reader" \
    --scope "/subscriptions/YOUR_SUBSCRIPTION_ID"

4. Log Analytics Reader Role (optional, for detailed logs)

az role assignment create \
    --assignee "YOUR_SERVICE_PRINCIPAL_APP_ID" \
    --role "Log Analytics Reader" \
    --scope "/subscriptions/YOUR_SUBSCRIPTION_ID"

Note: Role assignments may take 5-10 minutes to propagate across Azure services.

Step 3: Verify Service Principal Setup

Test your service principal configuration:

# Login with service principal
az login --service-principal \
    --username "YOUR_CLIENT_ID" \
    --password "YOUR_CLIENT_SECRET" \
    --tenant "YOUR_TENANT_ID"

# Test access to Cognitive Services
az cognitiveservices account list

# Test access to metrics (replace with your actual resource details)
az monitor metrics list \
    --resource "/subscriptions/YOUR_SUBSCRIPTION_ID/resourceGroups/YOUR_RG/providers/Microsoft.CognitiveServices/accounts/YOUR_OPENAI_RESOURCE" \
    --metric-names "Calls"

Step 4: Configure LLM Cost Tracker in Binadox

1. Access LLM Cost Tracker:

• Log into your Binadox dashboard
• Navigate to LLM Cost Tracker section
• Click Add New Integration

2. Select Azure OpenAI from the available providers

3. Fill in the following information from your service principal

Troubleshooting

Common Issues and Solutions

Validating Permissions

Check current role assignments for your service principal:

# List all role assignments
az role assignment list \
    --assignee "YOUR_SERVICE_PRINCIPAL_APP_ID" \
    --output table

# Verify specific resource access
az cognitiveservices account show \
    --name "YOUR_OPENAI_RESOURCE" \
    --resource-group "YOUR_RG"

Testing API Access

Verify service principal can access required APIs:

# Test Azure Monitor access
az monitor metrics list \
    --resource "/subscriptions/SUB_ID/resourceGroups/RG_NAME/providers/Microsoft.CognitiveServices/accounts/RESOURCE_NAME" \
    --metric-names "Calls,TokenTransaction,DataIn,DataOut"

# Test Cost Management access
az costmanagement query \
    --type "Usage" \
    --scope "/subscriptions/YOUR_SUBSCRIPTION_ID" \
    --time-period from="2024-01-01" to="2024-01-31"

Support and Resources

• Binadox Support: Contact support through your Binadox dashboard
• Azure OpenAI Documentation: Azure OpenAI Service Documentation
• Azure Monitor Documentation: Azure Monitor Documentation
• Cost Management Documentation: Azure Cost Management

For additional assistance or feature requests related to LLM Cost Tracker, please reach out to the Binadox support team.