AWS Generative AI Best Practices Framework v2: A Guide to Auditing Amazon Bedrock and SageMaker AI

AWS Audit Manager has upgraded its generative AI framework to version 2 on June 11, 2024. The framework helps organizations monitor and demonstrate how their generative AI workloads on Amazon Bedrock and Amazon SageMaker AI follow recommended best practices. While v2 adds Amazon SageMaker AI coverage alongside Amazon Bedrock, v1 is no longer supported. Existing v1 assessments continue to function, but new assessments should be created from v2. Note that AWS Audit Manager is no longer open to new customers, though existing customers can continue using the service as usual.

Generative AI focuses on enabling machines to create content resembling their training examples, supporting use cases such as conversational AI, creative content generation, data analysis, and process automation. Because this technology introduces new governance and responsibility challenges, AWS provides tooling and guidance to help build and manage applications responsibly. The framework was developed with input from AI experts, compliance practitioners, AWS security assurance specialists, and Deloitte. Each automated control maps to an AWS data source from which Audit Manager collects evidence.

The framework evaluates implementations against eight principles: Responsibility (establishing and following ethical guidelines for deploying and using models), Safety (setting clear parameters to prevent harmful outputs), Fairness (considering how AI affects different user subgroups), Sustainability (improving efficiency and pursuing more sustainable power sources), Resilience (maintaining integrity and availability mechanisms), Privacy (protecting sensitive data from theft or leakage), Accuracy (building reliable and robust AI systems), and Security (preventing unauthorized access to generative AI systems).

Consider an application that uses a third-party foundation model available through Amazon Bedrock. The framework can collect evidence showing that usage aligns with best practices. For instance, controls can help demonstrate that organizations document data sources and quality, periodically evaluate models against accuracy and safety benchmarks, use automated monitoring to detect biased outputs in real time, track model usage and permissions, establish breach notification procedures for inadvertent PII exposure, implement real-time monitoring with alerts for anomalies, conduct error analysis to identify root causes of inaccuracies, and apply end-to-end encryption to model inputs and outputs.

Amazon Bedrock and SageMaker AI customers can use the framework directly in AWS Audit Manager. Run assessments in the AWS accounts and Regions where generative AI models and applications operate. If CloudWatch logs for Bedrock or SageMaker AI are encrypted with a customer-managed KMS key, ensure Audit Manager can access that key by selecting it in the Audit Manager data encryption settings. The custom model usage evidence relies on the Bedrock ListCustomModels operation, which is currently supported only in US East (N. Virginia) and US West (Oregon), so custom model evidence may not appear for Asia Pacific (Tokyo), Asia Pacific (Singapore), or Europe (Frankfurt). Additionally, enable the required AWS Config rules, since the framework depends on them to collect intended evidence.

The framework contains prebuilt controls with descriptions and testing procedures, grouped into eight control sets aligned with generative AI best practices. To begin, create an assessment in Audit Manager, which starts evaluating AWS resources against the framework's controls. When audit time arrives, reviewers can browse the assessment's evidence folders or select evidence to include in an assessment report. With Evidence Finder enabled, specific evidence can be searched, exported as CSV, or used to generate a report. The framework can be customized by creating an editable copy of an existing framework to meet specific internal audit requirements. Note that the framework does not validate that systems comply with generative AI best practices, nor does it guarantee a passing audit result, and manual evidence collection controls are not checked automatically.

Some prompts may need to be evaluated against a specific model. Organizations can create a predefined prompt list covering their use cases, then use the Bedrock InvokeModel operation to test each prompt and collect responses as manual evidence for an Audit Manager assessment. InvokeModel can be used through HTTP requests (for example, with Postman), the AWS CLI invoke-model command, or automated validation with CloudWatch Synthetics canaries that periodically test prompts and monitor model behavior. Sample test prompts cover inappropriate content, PII disclosure (for example, verifying that a phone number is masked as {PHONE}), and profanity filtering.

The framework's controls are designed to support evidence collection and do not verify that a system is compliant with generative AI best practices. Using the framework does not guarantee passing an audit. Procedural controls requiring manual evidence are not checked automatically by AWS Audit Manager. Organizations should treat the framework as a starting point and supplement it with their own governance processes, reviews, and legal or compliance guidance as appropriate.

To learn more, review the list of standard controls included in the framework in AWS Audit Manager, create an assessment based on the framework, or customize an existing framework to support specific requirements. Further reading includes the Amazon Bedrock User Guide, the Amazon SageMaker AI User Guide, the AWS guidance on turning responsible AI from theory into practice, and the guide to responsible use of machine learning.