Skip to content

BackfillRevenuePixelHandler

Publisher Platform Revenue Backfill Processing Microservice

Service Type: Kubernetes Microservice
Technology: Node.js (JavaScript)
Runtime: Node.js Container
Last Updated: 2025-07-01

📋 Overview

The BackfillRevenuePixelHandler microservice processes revenue backfill events for the Publisher platform. It handles revenue pixel tracking for backfill clicks, integrating with AWS Kinesis for data streaming and DynamoDB for click data retrieval. The service is specifically designed to track and process revenue events from MediaAlpha backfill campaigns.

Key Features

  • Revenue Pixel Processing: Handles backfill revenue pixel tracking
  • AWS Kinesis Integration: Streams backfill click data to AWS Kinesis Firehose
  • DynamoDB Integration: Retrieves click keys and impression data
  • Revenue Event Generation: Creates standardized revenue events for downstream processing
  • Validation & Error Handling: Comprehensive request validation and retry mechanisms
  • MediaAlpha Integration: Specialized handling for MediaAlpha backfill campaigns

🛠 Technology Stack

Component Technology
Language JavaScript (Node.js)
Runtime Node.js Container
Framework HTTP Server
Cloud Services AWS Kinesis Firehose, DynamoDB
Data Streaming AWS Kinesis
Build Tool Webpack
Container Docker
Orchestration Kubernetes

🏗 Architecture

Service Architecture

  • Deployment Pattern: Kubernetes Deployment with HTTP Service
  • Input: HTTP GET requests with query parameters
  • Output Streams:
  • Revenue events to Event Hub
  • Backfill click data to AWS Kinesis Firehose
  • Data Sources: DynamoDB for click key retrieval
  • Configuration: ConfigMap and environment variables

System Flow

graph TD
    A[Revenue Pixel Request] --> B[BackfillRevenuePixelHandler]
    B --> C[Request Validation]
    C --> D[Revenue Object Creation]
    D --> E[DynamoDB Click Lookup]
    E --> F[Backfill Click Processing]
    F --> G[AWS Kinesis Firehose]
    G --> H[Data Lake/Analytics]
    B --> I[Revenue Event Hub]
    I --> J[Revenue Processing Pipeline]

🔄 Processing Pipeline

Revenue Processing Flow

graph LR
    A[HTTP Request] --> B[Parameter Validation]
    B --> C[Revenue Object Creation]
    C --> D[Click Key Retrieval]
    D --> E[Kinesis Stream Processing]
    E --> F[Event Hub Publishing]

Data Flow Architecture

sequenceDiagram
    participant Client
    participant Handler as BackfillHandler
    participant DDB as DynamoDB
    participant Kinesis as AWS Kinesis
    participant EventHub as Event Hub

    Client->>Handler: Revenue Pixel Request
    Handler->>Handler: Validate Parameters
    Handler->>DDB: Get Click Key Data
    DDB-->>Handler: Click Information
    Handler->>Kinesis: Send Backfill Click
    Handler->>EventHub: Send Revenue Event
    Handler-->>Client: Success Response

📡 API Specification

Revenue Pixel Endpoint

  • Method: GET
  • Authentication: Internal service authentication
  • Content-Type: application/json

Query Parameters

Parameter Type Required Description
wizsid string Yes Wizard session identifier
money number Yes Revenue amount (currency)
offername string No Placement/offer name
carrier string No Insurance carrier name

Request Example

GET /backfill-revenue?wizsid=abc123&money=25.50&offername=auto-insurance&carrier=progressive

Response Format

Success (200): 

{
  "status": "success"
}

Error (400): 

{
  "error": "Missing parameter: wizsid must be provided"
}

🔧 Business Logic

Revenue Event Structure

The service creates standardized revenue events with the following structure: 

{
  "MonetizationEvent": "BACKFILLCLICK",
  "Price": 25.50,
  "Wizsid": "session-id",
  "TimestampUTC": "2025-01-01T12:00:00.000Z",
  "TrusteeID": "MediaAlpha",
  "LineOfBusiness": "QuoteWizard",
  "Source": "QuoteWizard",
  "ProductCategory": "Insurance",
  "ExternalTransactionIdentifier": "click-key-or-random",
  "data": {
    "placement": "offer-name",
    "carrier": "carrier-name",
    "url": "original-request-url"
  }
}

Click Key Processing

  1. DynamoDB Lookup: Retrieves click data using wizsid and carrier
  2. Key Extraction: Extracts AdRequestKey, ClickKey, and BackfillImpressionKey
  3. Backfill Object Creation: Creates structured backfill click object
  4. Kinesis Streaming: Sends data to AWS Kinesis Firehose

Backfill Click Object Structure

{
  "adrequestkey": "ad-request-identifier",
  "backfillclickkey": "click-identifier",
  "backfillimpressionkey": "impression-identifier", 
  "click_timestamp": "timestamp-number",
  "clicktype": "click",
  "valid": true
}

Validation Rules

  • wizsid: Must be present and non-empty
  • money: Must be present and valid currency amount (>= 0)
  • offername: Optional placement identifier
  • carrier: Optional carrier identifier

Error Handling

  • Missing Parameters: Returns specific error messages for missing required fields
  • Invalid Money: Validates currency format and positive values
  • Kinesis Failures: Implements retry mechanism with configurable attempts
  • DynamoDB Failures: Graceful handling with fallback to random click keys

⚙️ Configuration

Environment Variables

  • AWS Configuration: Region, access keys, secret keys
  • Kinesis Settings: Delivery stream name, retry configuration
  • DynamoDB Settings: Table names and connection parameters
  • Logging: Log levels and Application Insights settings

Key Configuration Files

  • config/config.js: Main configuration
  • config/config.int.js: Integration environment
  • config/config.prod.js: Production environment

AWS Configuration Structure

{
  "DeltyAWS": {
    "region": "us-east-1",
    "accessKeyId": "aws-access-key",
    "secretAccessKey": "aws-secret-key"
  },
  "retries": 3
}

🚀 Deployment

Kubernetes Deployment

# Apply Kubernetes manifests
kubectl apply -f k8s/

# Check deployment status
kubectl get pods -l app=backfillrevenuepixelhandler

# View logs
kubectl logs -l app=backfillrevenuepixelhandler -f

Local Development

# Install dependencies
npm install

# Run tests
npm test

# Start service
npm start

📊 Monitoring & Health Checks

Health Endpoints

  • Liveness: /live - Basic health check
  • Readiness: /ready - Service readiness check

Key Metrics to Monitor

  • Revenue pixel processing rate
  • AWS Kinesis delivery success rate
  • DynamoDB lookup performance
  • Revenue amount validation errors
  • Click key retrieval success rate

Business Metrics

  • Total revenue processed
  • Average revenue per click
  • Carrier-specific performance
  • Placement effectiveness
  • Error rates by parameter type

🔧 Troubleshooting

Common Issues

Missing Revenue Events

  1. Check query parameter validation
  2. Verify wizsid and money parameters are provided
  3. Validate currency amount format

AWS Kinesis Failures

# Check AWS credentials and permissions
# Verify Kinesis Firehose stream exists
# Review retry configuration

DynamoDB Connection Issues

  1. Verify AWS credentials and region settings
  2. Check DynamoDB table accessibility
  3. Validate click key data structure

Revenue Validation Errors

  • Invalid Money: Ensure money parameter is valid number >= 0
  • Missing Wizsid: Verify session identifier is provided
  • URL Encoding: Check for proper URL encoding of parameters

🧪 Testing

Unit Tests

npm test

Manual Testing

# Test valid revenue pixel
curl "http://localhost:8080?wizsid=test123&money=25.50&offername=auto&carrier=progressive"

# Test missing parameters
curl "http://localhost:8080?money=25.50"

# Test invalid money amount
curl "http://localhost:8080?wizsid=test123&money=-5.00"

🤝 Dependencies

External Dependencies

  • AWS Kinesis Firehose: Data streaming service
  • AWS DynamoDB: Click key data storage
  • AWS SDK: AWS service integration

Internal Dependencies

  • Shared Node Modules: Logger utilities
  • Configuration Service: AWS and application settings
  • Event Hub: Revenue event publishing

NPM Dependencies

  • aws-sdk: AWS service integration
  • randomstring: Random identifier generation

This documentation was created through manual code analysis of the BackfillRevenuePixelHandler microservice codebase.