AWS Bedrock Integration for LLM Cost Tracker

Overview

Binadox LLM Cost Tracker provides comprehensive monitoring and cost analysis for your AWS Bedrock resources. This integration allows you to track usage patterns, monitor costs, analyze performance metrics, and optimize your AI spending across all AWS Bedrock models and deployments.

The LLM Cost Tracker provides detailed insights into:

• Model invocations and API call volumes
• Cost breakdown by model and region
• Usage trends and patterns
• Token consumption analysis
• Resource utilization optimization recommendations

Prerequisites

Before setting up the integration, ensure you have:

• AWS account with AWS Bedrock enabled
• AWS CLI installed or access to AWS Console
• Administrative permissions in AWS to create IAM users and assign policies
• Binadox account

Authentication Method

This guide uses IAM User authentication with programmatic access keys, which is the recommended approach for external integrations. IAM User authentication provides:

• Secure, key-based authentication
• Fine-grained permission control
• Full access to CloudWatch, Cost Explorer, and Bedrock APIs
• Cross-region monitoring capabilities
• Scalability for multiple AWS accounts

Step 1: Create AWS IAM User

Method 1 (Recommended): Using AWS Console

1. Log into AWS Console and navigate to IAM → Users → Create user.
2. Enter the user name as binadox-llm-cost-tracker and select Programmatic access (Access key – Programmatic access). Do not select AWS Console access since this user will only be used for API access.
3. Click Next: Permissions and select Attach existing policies directly. We’ll create and attach a custom policy in the next step, so you can proceed without selecting any policies at this stage.
4. Optionally, click Next: Tags to add descriptive tags such as Key=Purpose, Value=Binadox LLM Monitoring. Then click Next: Review to review your configuration.
5. Click Create user to complete the setup.

Important: Immediately copy and save both the Access Key ID and Secret Access Key as these credentials cannot be retrieved later. Store them securely as you’ll need them for the Binadox configuration.

Method 1: Using AWS CLI

# Create the IAM user
aws iam create-user --user-name binadox-llm-cost-tracker

# Create access key for programmatic access
aws iam create-access-key --user-name binadox-llm-cost-tracker

The output will show:

{
    "AccessKey": {
        "UserName": "binadox-llm-cost-tracker",
        "AccessKeyId": "AKIAIOSFODNN7EXAMPLE",          # client_id
        "Status": "Active",
        "SecretAccessKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"  # client_secret
    }
}

Important: Save these values securely. The secret access key cannot be retrieved later.

Step 2: Create and Attach Required Permissions

The IAM user needs specific permissions to access AWS Bedrock monitoring data. Create a comprehensive IAM policy with the following permissions:

Create Custom IAM Policy

In the AWS Console, go to IAM → Policies → Create policy and click the JSON tab to enter the policy directly.

Paste the Policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "BedrockAccess",
            "Effect": "Allow",
            "Action": [
                "bedrock:ListFoundationModels",
                "bedrock:GetFoundationModel",
                "bedrock:ListModelCustomizationJobs",
                "bedrock:GetModelCustomizationJob",
                "bedrock:GetModelInvocationLoggingConfiguration",
                "bedrock:ListProvisionedModelThroughputs",
                "bedrock:GetProvisionedModelThroughput"
            ],
            "Resource": "*"
        },
        {
            "Sid": "CloudWatchMetricsAccess",
            "Effect": "Allow",
            "Action": [
                "cloudwatch:GetMetricStatistics",
                "cloudwatch:GetMetricData",
                "cloudwatch:ListMetrics",
                "cloudwatch:GetMetricStream"
            ],
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "cloudwatch:namespace": [
                        "AWS/Bedrock",
                        "AWS/Bedrock/Runtime",
                        "AWS/Bedrock/Agents",
                        "AWS/Bedrock/Guardrails",
                        "AWS/Bedrock/KnowledgeBases"
                    ]
                }
            }
        },
        {
            "Sid": "CostExplorerAccess",
            "Effect": "Allow",
            "Action": [
                "ce:GetCostAndUsage",
                "ce:GetCostAndUsageWithResources",
                "ce:GetCostForecast",
                "ce:GetDimensionValues",
                "ce:GetTags",
                "ce:GetUsageForecast",
                "ce:DescribeCostCategoryDefinition",
                "ce:ListCostCategoryDefinitions"
            ],
            "Resource": "*"
        },
        {
            "Sid": "CloudWatchLogsAccess",
            "Effect": "Allow",
            "Action": [
                "logs:DescribeLogGroups",
                "logs:DescribeLogStreams",
                "logs:GetLogEvents",
                "logs:FilterLogEvents",
                "logs:StartQuery",
                "logs:StopQuery",
                "logs:GetQueryResults"
            ],
            "Resource": [
                "arn:aws:logs:*:*:log-group:/aws/bedrock/*",
                "arn:aws:logs:*:*:log-group::/aws/bedrock/*:*"
            ]
        },
        {
            "Sid": "TaggingAccess",
            "Effect": "Allow",
            "Action": [
                "tag:GetResources",
                "tag:GetTagKeys",
                "tag:GetTagValues"
            ],
            "Resource": "*"
        },
        {
            "Sid": "EC2RegionsAccess",
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeRegions"
            ],
            "Resource": "*"
        }
    ]
}

Click Next: Tags (optional). Click Next: Review

• Policy name: BinadoxBedrockMonitoringPolicy
• Description: Policy for Binadox LLM Cost Tracker to monitor AWS Bedrock resources
• Click Create policy

Attach Policy to User

Go to IAM → Users → binadox-llm-cost-tracker. Attach Policy:

• Click Add permissions → Attach existing policies directly
• Search for BinadoxBedrockMonitoringPolicy
• Select the policy and click Next: Review
• Click Add permissions

Alternative: Using AWS CLI

# Save the policy to a file
cat > binadox-bedrock-monitoring-policy.json << 'EOF'
{
    "Version": "2012-10-17",
    "Statement": [
        # ... (paste the policy JSON above) ...
    ]
}
EOF

# Create the policy
aws iam create-policy \
    --policy-name BinadoxBedrockMonitoringPolicy \
    --policy-document file://binadox-bedrock-monitoring-policy.json

# Attach policy to user
aws iam attach-user-policy \
    --user-name binadox-llm-cost-tracker \
    --policy-arn arn:aws:iam::YOUR_ACCOUNT_ID:policy/BinadoxBedrockMonitoringPolicy

Step 3: Enable Bedrock Model Invocation Logging (Optional)

For detailed usage analytics, enable model invocation logging:

Create CloudWatch Log Group

# Create log group for Bedrock invocations
aws logs create-log-group --log-group-name /aws/bedrock/modelinvocations

Enable Model Invocation Logging

# Enable logging for Bedrock (replace YOUR_ACCOUNT_ID)
aws bedrock put-model-invocation-logging-configuration \
    --logging-config '{
        "cloudWatchConfig": {
            "logGroupName": "/aws/bedrock/modelinvocations",
            "roleArn": "arn:aws:iam::YOUR_ACCOUNT_ID:role/BedrockLoggingRole"
        },
        "textDataDeliveryEnabled": true,
        "imageDataDeliveryEnabled": true,
        "embeddingDataDeliveryEnabled": true
    }'

Step 4: Verify IAM User Setup

Test your IAM user configuration:

# Configure AWS CLI with your new credentials
aws configure --profile binadox-bedrock
# AWS Access Key ID: YOUR_ACCESS_KEY_ID
# AWS Secret Access Key: YOUR_SECRET_ACCESS_KEY
# Default region name: us-east-1
# Default output format: json

# Test Bedrock access
aws bedrock list-foundation-models --profile binadox-bedrock

# Test CloudWatch access
aws cloudwatch list-metrics --namespace AWS/Bedrock --profile binadox-bedrock

# Test Cost Explorer access
aws ce get-cost-and-usage \
    --time-period Start=2024-01-01,End=2024-01-31 \
    --granularity DAILY \
    --metrics "UnblendedCost" \
    --filter '{
        "Dimensions": {
            "Key": "SERVICE",
            "Values": ["Amazon Bedrock"]
        }
    }' \
    --profile binadox-bedrock

Step 5: Configure LLM Cost Tracker in Binadox

1. Access LLM Cost Tracker:

• Log into your Binadox dashboard
• Navigate to LLM Cost Tracker section
• Click Add New Integration

2. Select AWS Bedrock from the available providers

3. Enter Connection Details

Troubleshooting

Common Issues and Solutions

Validating Permissions

Check current permissions for your IAM user:

# List attached policies
aws iam list-attached-user-policies --user-name binadox-llm-cost-tracker

# Test specific permissions
aws iam simulate-principal-policy \
    --policy-source-arn arn:aws:iam::YOUR_ACCOUNT_ID:user/binadox-llm-cost-tracker \
    --action-names bedrock:ListFoundationModels cloudwatch:GetMetricStatistics \
    --resource-arns "*"

# Check available Bedrock models
aws bedrock list-foundation-models --region us-east-1

# Verify Cost Explorer access
aws ce get-cost-and-usage \
    --time-period Start=2024-01-01,End=2024-01-02 \
    --granularity DAILY \
    --metrics "UnblendedCost"

Testing API Access

Verify IAM user can access required APIs:

# Test CloudWatch metrics
aws cloudwatch list-metrics \
    --namespace AWS/Bedrock \
    --dimensions Name=ModelId,Value=anthropic.claude-v2

# Test logging configuration
aws bedrock get-model-invocation-logging-configuration

# Check regions where Bedrock is available
aws ec2 describe-regions --query "Regions[?RegionName=='us-east-1' || RegionName=='us-west-2' || RegionName=='eu-central-1'].RegionName"

Support and Resources

• Binadox Support: Contact support through your Binadox dashboard
• AWS Bedrock Documentation: AWS Bedrock User Guide
• CloudWatch Metrics: Monitoring Bedrock with CloudWatch
• IAM Best Practices: AWS IAM Best Practices

For additional assistance or feature requests related to LLM Cost Tracker, please reach out to the Binadox support team.