SDK A/B Testing Guide

This guide covers advanced A/B testing implementation using the PromptCompose SDK, including rollout strategies, conversion tracking, and optimization patterns.

A/B Testing Fundamentals

How SDK A/B Testing Works

When you resolve a prompt through the SDK:

1. Test Check: SDK checks for active A/B tests on the prompt
2. Variant Assignment: User gets assigned to a variant based on strategy
3. Content Resolution: Assigned variant content is returned
4. Tracking: Conversion events can be reported back to measure success

Key Components

Session Consistency: Users see the same variant throughout their session Rollout Strategies: Control how users are distributed across variants Conversion Tracking: Measure which variants drive better outcomes Statistical Analysis: Platform provides significance testing and results

Basic A/B Testing Implementation

Simple A/B Test Resolution

import { PromptCompose } from '@promptcompose/sdk';

const promptCompose = new PromptCompose(apiKey, projectId);

async function getPromptWithTesting(userId: string, promptId: string, variables: any) {
  const result = await promptCompose.resolvePrompt(promptId, {
    config: {
      abTesting: {
        sessionId: `user-${userId}` // Ensures consistent variant per user
      }
    }
  }, variables);

  // Check if this was an A/B test
  if (result.variant && result.abTest) {
    console.log(`User ${userId} saw variant "${result.variant.name}" from test "${result.abTest.name}"`);
    
    // Store for later conversion tracking
    return {
      content: result.content,
      testId: result.abTest.publicId,
      variantId: result.variant.publicId,
      isTest: true
    };
  }

  return {
    content: result.content,
    isTest: false
  };
}

Conversion Tracking

When users complete the desired action, report it:

async function reportConversion(testId: string, variantId: string, userId: string, success: boolean) {
  try {
    await promptCompose.reportABResult(testId, {
      variantId: variantId,
      status: success ? 'success' : 'fail',
      sessionId: `user-${userId}`
    });
    
    console.log(`Conversion reported: ${success ? 'success' : 'failure'} for variant ${variantId}`);
  } catch (error) {
    console.error('Failed to report conversion:', error.message);
  }
}

// Usage example
const promptResult = await getPromptWithTesting('123', 'welcome-prompt', { name: 'John' });

// Display the content to user
displayMessage(promptResult.content);

// Later, when user performs desired action
if (userCompletedSignup && promptResult.isTest) {
  await reportConversion(
    promptResult.testId,
    promptResult.variantId,
    '123',
    true
  );
}

Rollout Strategies

Weighted Distribution

Randomly assigns users based on configured percentages:

async function weightedABTest(userId: string, promptId: string) {
  // SDK automatically handles weighted distribution
  const result = await promptCompose.resolvePrompt(promptId, {
    config: {
      abTesting: {
        sessionId: `user-${userId}`,
        // No specific variant - let the weighted algorithm decide
      }
    }
  });

  return result;
}

Best for:

• Standard A/B testing scenarios
• Statistical significance testing
• Even traffic distribution needs

Sequential Testing

Tests variants in rotating cycles:

async function sequentialABTest(userId: string, promptId: string) {
  // Sequential strategy is configured on the platform
  const result = await promptCompose.resolvePrompt(promptId, {
    config: {
      abTesting: {
        sessionId: `user-${userId}`,
        // Sequential rotation handled by platform
      }
    }
  });

  return result;
}

Best for:

• Time-based comparisons
• Seasonal effect considerations
• Gradual rollout scenarios

Manual Variant Selection

Explicitly control which users see which variants:

async function manualABTest(userId: string, promptId: string, userSegment: string) {
  let variantId;

  // Business logic for variant assignment
  if (userSegment === 'premium') {
    variantId = 'variant-premium-focus';
  } else if (userSegment === 'new-user') {
    variantId = 'variant-onboarding';
  } else {
    variantId = 'variant-control';
  }

  const result = await promptCompose.resolvePrompt(promptId, {
    config: {
      abTesting: {
        variantId: variantId,
        sessionId: `user-${userId}`
      }
    }
  });

  return result;
}

Best for:

• User segmentation testing
• Feature flags
• Targeted rollouts

Advanced Implementation Patterns

Comprehensive A/B Testing Service

interface TestResult {
  content: string;
  testId?: string;
  variantId?: string;
  variantName?: string;
  testName?: string;
}

interface ConversionEvent {
  testId: string;
  variantId: string;
  userId: string;
  status: 'success' | 'fail' | 'timeout' | 'error';
  metadata?: Record<string, any>;
}

class ABTestingService {
  private promptCompose: PromptCompose;
  private activeTests = new Map<string, any>(); // userId -> test info
  
  constructor(apiKey: string, projectId: string) {
    this.promptCompose = new PromptCompose(apiKey, projectId);
  }

  async init() {
    await this.promptCompose.init();
  }

  async servePrompt(
    promptId: string, 
    userId: string, 
    variables?: Record<string, any>,
    context?: Record<string, any>
  ): Promise<TestResult> {
    try {
      const sessionId = this.generateSessionId(userId, context);
      
      const result = await this.promptCompose.resolvePrompt(promptId, {
        config: {
          abTesting: { sessionId }
        }
      }, variables);

      // Store test information for conversion tracking
      if (result.variant && result.abTest) {
        this.activeTests.set(userId, {
          testId: result.abTest.publicId,
          variantId: result.variant.publicId,
          variantName: result.variant.name,
          testName: result.abTest.name,
          promptId: promptId,
          servedAt: new Date(),
          context: context
        });

        return {
          content: result.content,
          testId: result.abTest.publicId,
          variantId: result.variant.publicId,
          variantName: result.variant.name,
          testName: result.abTest.name
        };
      }

      return { content: result.content };
    } catch (error) {
      console.error('Failed to serve prompt:', error);
      // Return fallback content
      return { content: 'Default message' };
    }
  }

  async reportConversion(
    userId: string, 
    status: 'success' | 'fail' | 'timeout' | 'error',
    metadata?: Record<string, any>
  ): Promise<boolean> {
    const testInfo = this.activeTests.get(userId);
    if (!testInfo) {
      console.warn(`No active test found for user ${userId}`);
      return false;
    }

    try {
      await this.promptCompose.reportABResult(testInfo.testId, {
        variantId: testInfo.variantId,
        status: status,
        sessionId: this.generateSessionId(userId, testInfo.context)
      });

      console.log(`Conversion reported for user ${userId}: ${status}`);
      
      // Clean up
      this.activeTests.delete(userId);
      
      return true;
    } catch (error) {
      console.error('Failed to report conversion:', error);
      return false;
    }
  }

  async batchReportConversions(events: ConversionEvent[]): Promise<void> {
    const promises = events.map(event => 
      this.promptCompose.reportABResult(event.testId, {
        variantId: event.variantId,
        status: event.status,
        sessionId: this.generateSessionId(event.userId)
      })
    );

    try {
      await Promise.allSettled(promises);
      console.log(`Batch reported ${events.length} conversion events`);
    } catch (error) {
      console.error('Batch conversion reporting failed:', error);
    }
  }

  getActiveTest(userId: string) {
    return this.activeTests.get(userId);
  }

  private generateSessionId(userId: string, context?: Record<string, any>): string {
    const contextStr = context ? JSON.stringify(context) : '';
    return `${userId}-${contextStr}`.replace(/[^a-zA-Z0-9-_]/g, '');
  }
}

Usage Example

const abTestingService = new ABTestingService(
  process.env.PROMPT_COMPOSE_API_KEY!,
  process.env.PROMPT_COMPOSE_PROJECT_ID!
);

await abTestingService.init();

// Serve a prompt with A/B testing
const result = await abTestingService.servePrompt(
  'welcome-email-prompt',
  'user-123',
  { userName: 'John', companyName: 'Acme' },
  { deviceType: 'mobile', location: 'US' }
);

console.log('Email content:', result.content);

// Display the email to user...

// Later, when user performs desired action
if (userOpenedEmail) {
  await abTestingService.reportConversion('user-123', 'success');
}

User Segmentation and Context

Context-Aware Testing

interface UserContext {
  deviceType: 'mobile' | 'desktop' | 'tablet';
  location: string;
  userType: 'new' | 'returning' | 'premium';
  timeOfDay: 'morning' | 'afternoon' | 'evening';
}

class ContextAwareABTesting {
  private promptCompose: PromptCompose;

  constructor(apiKey: string, projectId: string) {
    this.promptCompose = new PromptCompose(apiKey, projectId);
  }

  async resolveWithContext(
    promptId: string, 
    userId: string, 
    variables: any, 
    context: UserContext
  ) {
    // Create context-aware session ID
    const sessionId = this.createContextSessionId(userId, context);

    // For manual strategy, you could choose variants based on context
    const config = await this.getConfigForContext(promptId, context, sessionId);

    const result = await this.promptCompose.resolvePrompt(promptId, config, variables);

    return {
      ...result,
      context: context,
      sessionId: sessionId
    };
  }

  private createContextSessionId(userId: string, context: UserContext): string {
    return `${userId}-${context.deviceType}-${context.userType}`;
  }

  private async getConfigForContext(
    promptId: string, 
    context: UserContext, 
    sessionId: string
  ) {
    // Get available tests to check strategy
    const tests = await this.promptCompose.listABTests();
    const relevantTest = tests.find(test => test.prompt.publicId === promptId);

    if (!relevantTest || relevantTest.rolloutStrategy !== 'manual') {
      // Use standard A/B testing
      return {
        config: {
          abTesting: { sessionId }
        }
      };
    }

    // For manual strategy, choose variant based on context
    const variantId = this.selectVariantByContext(relevantTest.variants, context);

    return {
      config: {
        abTesting: {
          sessionId,
          variantId
        }
      }
    };
  }

  private selectVariantByContext(variants: any[], context: UserContext): string {
    // Example logic - customize based on your needs
    if (context.deviceType === 'mobile') {
      const mobileVariant = variants.find(v => v.name.includes('mobile'));
      if (mobileVariant) return mobileVariant.publicId;
    }

    if (context.userType === 'premium') {
      const premiumVariant = variants.find(v => v.name.includes('premium'));
      if (premiumVariant) return premiumVariant.publicId;
    }

    // Default to first variant (often control)
    return variants[0]?.publicId;
  }
}

Feature Flag Integration

class FeatureFlagABTesting {
  private promptCompose: PromptCompose;
  private featureFlags: any; // Your feature flag service

  constructor(apiKey: string, projectId: string, featureFlags: any) {
    this.promptCompose = new PromptCompose(apiKey, projectId);
    this.featureFlags = featureFlags;
  }

  async resolveWithFlags(
    promptId: string, 
    userId: string, 
    variables: any
  ) {
    const flags = await this.featureFlags.getFlags(userId);

    // Check if A/B testing is disabled by feature flag
    if (flags.disableAbTesting) {
      return await this.promptCompose.resolvePrompt(promptId, {
        config: {
          versionId: flags.forceVersion || 'latest',
          abTesting: { enabled: false }
        }
      }, variables);
    }

    // Check for forced variant
    if (flags.forceVariant) {
      return await this.promptCompose.resolvePrompt(promptId, {
        config: {
          abTesting: {
            variantId: flags.forceVariant,
            sessionId: `user-${userId}`
          }
        }
      }, variables);
    }

    // Standard A/B testing
    return await this.promptCompose.resolvePrompt(promptId, {
      config: {
        abTesting: { sessionId: `user-${userId}` }
      }
    }, variables);
  }
}

Performance Optimization

Caching with A/B Testing

interface CacheEntry {
  content: string;
  variant?: string;
  test?: string;
  timestamp: number;
}

class CachedABTesting {
  private promptCompose: PromptCompose;
  private cache = new Map<string, CacheEntry>();
  private cacheTimeout = 5 * 60 * 1000; // 5 minutes

  constructor(apiKey: string, projectId: string) {
    this.promptCompose = new PromptCompose(apiKey, projectId);
  }

  async resolveWithCache(
    promptId: string,
    userId: string,
    variables: any
  ): Promise<any> {
    // Create cache key that includes user for consistent A/B assignment
    const cacheKey = this.getCacheKey(promptId, userId, variables);
    const cached = this.cache.get(cacheKey);

    if (cached && Date.now() - cached.timestamp < this.cacheTimeout) {
      return {
        content: cached.content,
        fromCache: true,
        variant: cached.variant,
        test: cached.test
      };
    }

    const result = await this.promptCompose.resolvePrompt(promptId, {
      config: {
        abTesting: { sessionId: `user-${userId}` }
      }
    }, variables);

    // Cache the result
    this.cache.set(cacheKey, {
      content: result.content,
      variant: result.variant?.name,
      test: result.abTest?.name,
      timestamp: Date.now()
    });

    return result;
  }

  private getCacheKey(promptId: string, userId: string, variables: any): string {
    return `${promptId}-${userId}-${JSON.stringify(variables)}`;
  }

  clearCache() {
    this.cache.clear();
  }

  // Clean expired entries
  cleanup() {
    const now = Date.now();
    for (const [key, entry] of this.cache.entries()) {
      if (now - entry.timestamp > this.cacheTimeout) {
        this.cache.delete(key);
      }
    }
  }
}

Batch Operations

class BatchABTesting {
  private promptCompose: PromptCompose;

  constructor(apiKey: string, projectId: string) {
    this.promptCompose = new PromptCompose(apiKey, projectId);
  }

  async resolveMultiplePrompts(
    requests: Array<{
      promptId: string;
      userId: string;
      variables?: any;
    }>
  ) {
    const promises = requests.map(req => 
      this.promptCompose.resolvePrompt(req.promptId, {
        config: {
          abTesting: { sessionId: `user-${req.userId}` }
        }
      }, req.variables).catch(error => ({ error: error.message }))
    );

    const results = await Promise.all(promises);
    
    return results.map((result, index) => ({
      request: requests[index],
      result: result,
      success: !('error' in result)
    }));
  }

  async batchReportConversions(
    conversions: Array<{
      testId: string;
      variantId: string;
      userId: string;
      status: 'success' | 'fail';
    }>
  ) {
    const promises = conversions.map(conv => 
      this.promptCompose.reportABResult(conv.testId, {
        variantId: conv.variantId,
        status: conv.status,
        sessionId: `user-${conv.userId}`
      }).catch(error => ({ error: error.message }))
    );

    const results = await Promise.allSettled(promises);
    
    const successful = results.filter(r => r.status === 'fulfilled').length;
    const failed = results.length - successful;
    
    console.log(`Batch conversion reporting: ${successful} successful, ${failed} failed`);
    
    return { successful, failed, results };
  }
}

Analytics and Monitoring

Custom Analytics Integration

class AnalyticsABTesting {
  private promptCompose: PromptCompose;
  private analytics: any; // Your analytics service

  constructor(apiKey: string, projectId: string, analytics: any) {
    this.promptCompose = new PromptCompose(apiKey, projectId);
    this.analytics = analytics;
  }

  async serveWithAnalytics(
    promptId: string,
    userId: string,
    variables: any,
    analyticsContext: any = {}
  ) {
    const startTime = Date.now();

    try {
      const result = await this.promptCompose.resolvePrompt(promptId, {
        config: {
          abTesting: { sessionId: `user-${userId}` }
        }
      }, variables);

      const duration = Date.now() - startTime;

      // Track prompt serving
      this.analytics.track('prompt_served', {
        userId,
        promptId,
        variant: result.variant?.name,
        test: result.abTest?.name,
        duration,
        ...analyticsContext
      });

      // Track A/B test exposure if applicable
      if (result.variant && result.abTest) {
        this.analytics.track('ab_test_exposure', {
          userId,
          testId: result.abTest.publicId,
          testName: result.abTest.name,
          variantId: result.variant.publicId,
          variantName: result.variant.name,
          promptId,
          ...analyticsContext
        });
      }

      return result;
    } catch (error) {
      this.analytics.track('prompt_error', {
        userId,
        promptId,
        error: error.message,
        duration: Date.now() - startTime,
        ...analyticsContext
      });
      
      throw error;
    }
  }

  async reportConversionWithAnalytics(
    userId: string,
    testId: string,
    variantId: string,
    status: 'success' | 'fail',
    value?: number,
    metadata?: any
  ) {
    try {
      // Report to PromptCompose
      await this.promptCompose.reportABResult(testId, {
        variantId,
        status,
        sessionId: `user-${userId}`
      });

      // Track in analytics
      this.analytics.track('ab_test_conversion', {
        userId,
        testId,
        variantId,
        status,
        value,
        ...metadata
      });

      return true;
    } catch (error) {
      this.analytics.track('conversion_error', {
        userId,
        testId,
        variantId,
        error: error.message
      });
      
      throw error;
    }
  }
}

Testing and Quality Assurance

A/B Test Validation

class ABTestValidator {
  private promptCompose: PromptCompose;

  constructor(apiKey: string, projectId: string) {
    this.promptCompose = new PromptCompose(apiKey, projectId);
  }

  async validateTest(promptId: string): Promise<{
    isValid: boolean;
    issues: string[];
    variants: any[];
  }> {
    const issues: string[] = [];
    let variants: any[] = [];

    try {
      // Check if prompt exists
      const prompt = await this.promptCompose.getPrompt(promptId);
      
      // Get active tests
      const tests = await this.promptCompose.listABTests();
      const activeTest = tests.find(t => 
        t.prompt.publicId === promptId && t.status === 'active'
      );

      if (!activeTest) {
        issues.push('No active A/B test found for this prompt');
        return { isValid: false, issues, variants };
      }

      variants = activeTest.variants;

      // Validate variants
      if (variants.length < 2) {
        issues.push('Test must have at least 2 variants');
      }

      // Check variant weights (for weighted strategy)
      if (activeTest.rolloutStrategy === 'weighted') {
        const totalWeight = variants.reduce((sum, v) => sum + v.weight, 0);
        if (Math.abs(totalWeight - 100) > 0.01) {
          issues.push(`Variant weights sum to ${totalWeight}%, not 100%`);
        }
      }

      // Test each variant
      for (const variant of variants) {
        try {
          await this.promptCompose.resolvePrompt(promptId, {
            config: {
              abTesting: {
                variantId: variant.publicId,
                sessionId: 'test-session'
              }
            }
          }, {});
        } catch (error) {
          issues.push(`Variant "${variant.name}" failed to resolve: ${error.message}`);
        }
      }

    } catch (error) {
      issues.push(`Failed to validate test: ${error.message}`);
    }

    return {
      isValid: issues.length === 0,
      issues,
      variants
    };
  }

  async testAllVariants(promptId: string, testVariables?: any) {
    const tests = await this.promptCompose.listABTests();
    const activeTest = tests.find(t => 
      t.prompt.publicId === promptId && t.status === 'active'
    );

    if (!activeTest) {
      throw new Error('No active test found for this prompt');
    }

    const results = [];

    for (const variant of activeTest.variants) {
      try {
        const result = await this.promptCompose.resolvePrompt(promptId, {
          config: {
            abTesting: {
              variantId: variant.publicId,
              sessionId: `test-${variant.publicId}`
            }
          }
        }, testVariables || {});

        results.push({
          variant: variant.name,
          variantId: variant.publicId,
          success: true,
          content: result.content.substring(0, 100) + '...', // Truncated
          length: result.content.length
        });
      } catch (error) {
        results.push({
          variant: variant.name,
          variantId: variant.publicId,
          success: false,
          error: error.message
        });
      }
    }

    return results;
  }
}

Best Practices

Session Management

class SessionManager {
  private sessions = new Map<string, string>();

  getSessionId(userId: string, context?: any): string {
    const key = this.createSessionKey(userId, context);
    
    if (!this.sessions.has(key)) {
      const sessionId = this.generateSessionId(userId, context);
      this.sessions.set(key, sessionId);
    }
    
    return this.sessions.get(key)!;
  }

  private createSessionKey(userId: string, context?: any): string {
    const contextStr = context ? JSON.stringify(context) : '';
    return `${userId}-${contextStr}`;
  }

  private generateSessionId(userId: string, context?: any): string {
    const timestamp = Date.now();
    const contextStr = context?.deviceType || 'unknown';
    return `${userId}-${contextStr}-${timestamp}`;
  }

  clearSessions() {
    this.sessions.clear();
  }
}

Error Handling and Fallbacks

async function robustABTestResolve(
  promptCompose: PromptCompose,
  promptId: string,
  userId: string,
  variables: any,
  fallbackContent: string
): Promise<string> {
  try {
    const result = await promptCompose.resolvePrompt(promptId, {
      config: {
        abTesting: { sessionId: `user-${userId}` }
      }
    }, variables);

    return result.content;
  } catch (error) {
    console.error('A/B test resolution failed:', error.message);
    
    // Try without A/B testing
    try {
      const fallbackResult = await promptCompose.resolvePrompt(promptId, {
        config: { abTesting: { enabled: false } }
      }, variables);
      
      return fallbackResult.content;
    } catch (fallbackError) {
      console.error('Fallback resolution failed:', fallbackError.message);
      return fallbackContent;
    }
  }
}

Integration Examples

Express.js Middleware

function abTestingMiddleware(promptCompose: PromptCompose) {
  return async (req: any, res: any, next: any) => {
    req.abTesting = {
      resolve: async (promptId: string, variables?: any) => {
        const userId = req.user?.id || req.sessionID;
        return await promptCompose.resolvePrompt(promptId, {
          config: { abTesting: { sessionId: `user-${userId}` } }
        }, variables);
      },
      
      reportConversion: async (testId: string, variantId: string, success: boolean) => {
        const userId = req.user?.id || req.sessionID;
        return await promptCompose.reportABResult(testId, {
          variantId,
          status: success ? 'success' : 'fail',
          sessionId: `user-${userId}`
        });
      }
    };
    
    next();
  };
}

// Usage
app.use(abTestingMiddleware(promptCompose));

app.get('/welcome', async (req, res) => {
  const result = await req.abTesting.resolve('welcome-message', {
    userName: req.user.name
  });
  
  res.json({ message: result.content });
});

For more advanced patterns and examples, see:

• API Reference
• Best Practices Guide
• Implementation Examples