RevenueEnrichment Service

Overview

The RevenueEnrichment is a sophisticated Azure Function microservice that processes revenue events through a comprehensive 16-stage pipeline to enrich, calculate, and route revenue data. This service acts as the central revenue processing engine for the Publisher platform, handling complex business logic for revenue attribution, cost calculations, and modifier applications across multiple revenue streams.

Business Purpose

This service serves as the core revenue processing engine that: - Processes revenue events from multiple sources (conversions, clicks, sessions, leads, calls) - Applies complex business rules for revenue calculation and attribution - Handles multiple revenue models (CPA, CPC, CPM, Revenue Share, SEM) - Calculates costs, margins, and modifier applications - Routes processed revenue to appropriate downstream systems (Snowflake, Exchange, Unaccounted) - Maintains revenue audit trails and compliance data - Supports real-time revenue reporting and analytics

Architecture

Service Type

Platform: Azure Functions (Containerized Kubernetes Microservice)
Runtime: Node.js
Trigger: HTTP Trigger (Anonymous authentication)
Pattern: Pipeline-Based Event Processing with Multi-Stage Enrichment

Key Components

graph TD
    A[Revenue Events] --> B[RevenueEnrichment]
    B --> C[Handler.js]
    C --> D[Pipeline Orchestrator]

    D --> E[Stage 1: Create Record]
    E --> F[Stage 2: Get Router Session]
    F --> G[Stage 3: Get Document Cache]
    G --> H[Stage 4: Extract Required Info]
    H --> I[Stage 5: Get Event Details]
    I --> J[Stage 6: Get Completed Session]
    J --> K[Stage 7: Get Cost Rules]
    K --> L[Stage 8: Decide Revenue Action]
    L --> M[Stage 9-12: Revenue Enrichment]
    M --> N[Stage 13-14: Get Modifiers]
    N --> O[Stage 15: Apply Modifiers]
    O --> P[Stage 16: Add Trustee Records]

    P --> Q{Revenue Type?}
    Q -->|Publisher| R[Event Hub: publisherrevenue]
    Q -->|Unaccounted| S[Event Hub: unaccountedrevenue]
    Q -->|Exchange| T[Event Hub: exchangerevenue]

    U[Router Session Service] --> F
    V[Document Cache Service] --> G
    W[Session Cache Service] --> J
    X[Cosmos DB] --> K

Pipeline Architecture

16-Stage Processing Pipeline

Stage 1: Create Record (1.0.CreateRec)

Initializes revenue record with session ID and event type
Validates required fields (wizsid, eventType)
Sets processing timestamp

Stage 2: Get Router Session (2.0.GetRouterSession)

Retrieves router session data for context
Provides campaign and routing information

Stage 3: Get Document Cache (3.0.GetDocumentCache)

Fetches cached campaign and vendor configurations
Retrieves business rules and settings

Stage 4: Extract Required Info (4.0.ExtractRequiredInfo)

Extracts essential data from event and session
Prepares data for revenue calculations

Stage 5: Get Event Specific Details (5.0.GetEventSpecificDetails)

Processes event-type specific information
Handles conversion, click, session, lead, and call events

Stage 6: Get Completed Session (6.0.GetCompletedSession)

Retrieves completed session data from cache
Provides session completion context

Stage 7: Get Cost/Revenue Rules (7.0-7.1)

Retrieves cost rules and revenue share rules
Applies business-specific pricing models

Stage 8: Decide Revenue Action (8.0.DecideRevenueAction)

Determines appropriate revenue processing path
Routes to specific enrichment modules

Stages 9-12: Revenue Enrichment

9.0: Private Exchange enrichment
9.1: PPC (Pay-Per-Click) enrichment
10.0: CPA (Cost-Per-Action) enrichment
11.0: CPA Click enrichment
12.0: Advertiser margin override
12.1: Revenue share and SEM enrichment
12.2: Secondary revenue share

Stages 13-15: Modifier Application

13.0: Schedule-based modifiers
14.0: State-based modifiers
15.0: Apply all modifiers to revenue

Stage 16: Add Trustee Records (16.0.AddTrusteeIdRecord)

Adds compliance and audit records
Maintains revenue attribution trails

Supported Event Types

Primary Event Types

SESSION: User session completion events
CONVERSION: Conversion tracking events
CLICK: Click-through events
BACKFILLCLICK: Backfill click events
LEAD: Lead generation events
CALL: Call tracking events

Revenue Models

CPA (Cost Per Action): Fixed cost per conversion
CPC (Cost Per Click): Fixed cost per click
CPM (Cost Per Mille): Cost per thousand impressions
Revenue Share: Percentage-based revenue sharing
SEM (Search Engine Marketing): Search-specific revenue models

Data Flow

Event Processing Flow

Event Reception: Receives revenue events from multiple Event Hubs
Pipeline Initialization: Sets up processing context and services
Sequential Processing: Executes 16-stage pipeline for each event
Revenue Calculation: Applies business rules and calculations
Modifier Application: Applies schedule, state, and custom modifiers
Revenue Routing: Routes to appropriate destination Event Hub
Audit Trail: Maintains comprehensive processing logs

Event Structure

Input Event Format

{
    "wizsid": "session-identifier",
    "eventType": "CONVERSION|CLICK|SESSION|LEAD|CALL",
    "campaignId": "campaign-identifier",
    "vendorId": "vendor-identifier",
    "amount": 25.50,
    "timestamp": "2023-01-01T00:00:00Z",
    "metadata": {
        "source": "platform-source",
        "referrer": "referrer-url"
    }
}

Output Event Format

{
    "recordid": "unique-record-id",
    "wizsid": "session-identifier",
    "eventType": "processed-event-type",
    "revenueAmount": 25.50,
    "costAmount": 15.30,
    "margin": 10.20,
    "modifiers": {
        "schedule": 1.1,
        "state": 0.95,
        "custom": 1.05
    },
    "reTimestamp": "2023-01-01T00:00:00.000",
    "destination": "publisherrevenue|unaccountedrevenue|exchangerevenue"
}

Core Functionality

Revenue Calculation Engine

Base Revenue: Calculates base revenue from event data
Cost Application: Applies cost rules and vendor agreements
Margin Calculation: Computes profit margins and overrides
Modifier Application: Applies time, location, and custom modifiers
Final Attribution: Determines final revenue attribution

Business Rules Engine

Campaign Rules: Campaign-specific revenue and cost rules
Vendor Agreements: Vendor-specific pricing and terms
Modifier Rules: Schedule, geographic, and performance modifiers
Compliance Rules: Audit and regulatory compliance requirements

Key Features

16-Stage Pipeline: Comprehensive processing pipeline for revenue events
Multiple Revenue Models: Support for CPA, CPC, CPM, Revenue Share, SEM
Dynamic Modifiers: Real-time application of business modifiers
Audit Compliance: Complete audit trail for revenue processing
Multi-Destination Routing: Intelligent routing to appropriate systems
Error Recovery: Comprehensive error handling and retry mechanisms
Performance Optimization: Efficient processing for high-volume events

Performance Characteristics

Processing Metrics

Throughput: ~1000 events per minute
Latency: 200-500ms per event (depending on pipeline complexity)
Success Rate: 99.8% processing success rate
Pipeline Efficiency: Average 16 stages completed in <300ms

Scalability Features

Horizontal Scaling: Stateless pipeline design for scaling
Concurrent Processing: Parallel event processing capabilities
Cache Optimization: Efficient use of session and document caches
Database Optimization: Optimized Cosmos DB queries and operations

Dependencies

External Services

Cosmos DB: Campaign, vendor, and rule storage
Event Hubs: Input and output event streaming
Router Session Service: Session context and routing data
Document Cache Service: Cached configuration data
Session Cache Service: Session completion data

Key NPM Packages

@azure/cosmos: Cosmos DB integration
async: Asynchronous pipeline processing
moment: Date/time handling and formatting
idgen: Unique identifier generation

Configuration

Environment-Specific Settings

Development: Development databases and reduced processing
Integration: Integration testing with staging data
Production: Production databases with full processing pipeline

Key Configuration Elements

Cosmos DB connection strings
Event Hub configurations for input/output
Pipeline stage configurations
Business rule parameters
Modifier calculation settings

Error Handling

Error Scenarios

Missing Session Data: Unable to retrieve router or session information
Invalid Event Type: Unsupported or malformed event types
Rule Application Failures: Cost or revenue rule processing errors
Database Connectivity: Cosmos DB connection or query failures
Pipeline Stage Failures: Individual stage processing errors

Recovery Mechanisms

Failed Event Collection: Automatic collection of failed events for retry
Pipeline Continuation: Graceful handling of stage failures
Data Validation: Comprehensive validation at each pipeline stage
Audit Logging: Detailed logging for debugging and compliance

Monitoring and Observability

Application Insights Integration

Transaction Monitoring: Real-time transaction tracking
Performance Metrics: Pipeline stage performance monitoring
Error Tracking: Comprehensive error logging and alerting
Custom Telemetry: Business-specific metrics and KPIs

Key Metrics

Revenue processing volumes and rates
Pipeline stage performance and bottlenecks
Error rates by stage and event type
Business metrics (revenue, costs, margins)

Support and Operations

Support Contacts

Level 1: Development Team On-Call (delty-funnels-cms@quotewizard.opsgenie.net)
Level 2: Mike Smith (303-905-4465)
Level 3: Kannaiyan Natesan (650-605-3290)
Level 4: Kyle Johnson (206-280-1209)

Operational Monitoring

Application Insights dashboard monitoring
Cosmos DB checkpoint monitoring
Event Hub throughput monitoring
Pipeline performance tracking

This service integrates with the broader Publisher ecosystem: - PostbackHandler: Provides revenue events for processing - RouterV2: Provides session and routing context - DocumentCacheHandler: Provides cached configuration data - Snowflake: Consumes processed revenue data - Reporting Services: Use enriched revenue data for analytics

Troubleshooting

Common Issues

Pipeline Bottlenecks: Monitor stage performance and optimize slow stages
Data Quality Issues: Validate input event formats and required fields
Rule Application Errors: Verify cost and revenue rule configurations
Cache Performance: Monitor cache hit rates and refresh strategies
Database Performance: Optimize Cosmos DB queries and indexing

Debug Steps

Check Application Insights for pipeline execution traces
Verify Event Hub message flow and processing rates
Review Cosmos DB query performance and connection health
Validate business rule configurations and data quality
Monitor cache performance and data freshness

Development

Local Development Setup

Clone repository
Install dependencies: npm install
Configure Cosmos DB connections
Set up Event Hub connection strings
Configure cache service connections
Run tests: npm test
Test pipeline: npm run test-pipeline

Code Structure

src/Handler.js: Main pipeline orchestrator
src/Pipeline/: 16 pipeline stage implementations
src/Services/: External service integrations
config/: Environment-specific configurations
runbooks/: Operational documentation and procedures