Skip to content

SourcesHandler Microservice

Overview

The SourcesHandler microservice is responsible for processing and managing source-related data within the Publisher ecosystem. It handles campaign sources, product sources, publisher names, and experience types by processing messages from various collections and maintaining synchronized data across multiple storage systems.

Business Purpose

This service acts as a central hub for managing source data integrity and synchronization across the Publisher platform. It processes events related to campaigns, vendors, and configuration changes to maintain up-to-date source information in both Cosmos DB and S3 storage systems.

Architecture

Service Type

  • Deployment: Kubernetes containerized microservice
  • Trigger: HTTP-triggered Azure Function
  • Runtime: Node.js

Key Components

  1. ProductSourcesHandler: Manages product-related source data
  2. CampaignSourcesHandler: Handles campaign source information
  3. PublisherNamesHandler: Processes publisher name data with override capabilities
  4. ExperienceTypesHandler: Manages experience type configurations

Data Flow

graph TD
    A[HTTP Request] --> B[Main Handler]
    B --> C[Message Processing]
    C --> D[ProductSourcesHandler]
    C --> E[CampaignSourcesHandler]
    C --> F[PublisherNamesHandler]
    C --> G[ExperienceTypesHandler]

    D --> H[Cosmos DB - Vendors]
    D --> I[Cosmos DB - Campaigns]
    D --> J[S3 Storage]

    E --> H
    E --> I
    E --> J

    F --> H
    F --> I
    F --> J

    G --> J

    H --> K[System Events]
    I --> K
    J --> K

Dependencies

External Services

  • Azure Cosmos DB: Publisher database for Vendors and Campaigns collections
  • AWS S3: Storage for source data files
  • Azure Event Hub: System events publishing

NPM Dependencies

  • @azure/cosmos: Cosmos DB client
  • aws-sdk: AWS services integration
  • async: Asynchronous flow control
  • lodash: Utility functions
  • titleize: String formatting
  • idgen: ID generation
  • fast-json-patch: JSON patching operations

Configuration

Environment-Specific Configs

  • config.js: Development configuration
  • config.int.js: Integration environment
  • config.prod.js: Production environment

Key Configuration Parameters

  • LogLevel: Logging verbosity (default: 5)
  • S3 Bucket: dataetlhandler with key prefix appcache/
  • maxRetries: Maximum retry attempts (default: 5)
  • Event Hub: Consumer group sourceshandler for documentevent hub

API Endpoints

POST /

Processes source-related messages in batch format.

Request Body: Array of message objects 

[
  {
    "collection": "Campaigns|Vendors|configuration",
    "action": "CREATE|UPDATE|DELETE",
    "record": { /* record data */ },
    "oldRecord": { /* previous record state */ }
  }
]

Response: - Status: 200 OK - Body: Array of failed events (if any)

Message Processing Logic

Supported Collections

  1. Campaigns: Campaign-related source updates
  2. Vendors: Publisher/vendor source information
  3. configuration: Experience type configurations (id: 'etype')

Processing Rules

  • Ignores messages without valid collection or record data
  • Skips DELETE actions for most handlers (except ProductSourcesHandler)
  • Filters out campaigns with status "review"
  • Handles vendor ID overrides using sourceGroupVendorId

Special Features

  • Source Group Override Names: Predefined mappings for specific source groups
  • Retry Logic: Configurable retry attempts with detailed logging
  • Parallel Processing: Multiple handlers process messages concurrently
  • Error Handling: Failed events are collected and returned for retry

Data Storage

Cosmos DB Collections

  • Vendors: Publisher/vendor information
  • Campaigns: Campaign data and source associations

S3 Storage Structure

  • Bucket: dataetlhandler
  • Prefix: appcache/
  • Files: Various source data files including experience types

Monitoring and Logging

Application Insights

  • Connection string configured for telemetry
  • Custom logging levels for different environments
  • Error tracking and performance monitoring

Event Publishing

  • System events published to publishersystemevents Event Hub
  • Failed event tracking and alerting

Error Handling

Retry Mechanism

  • Maximum retry attempts: 5 (configurable)
  • Retry attempt tracking via HTTP headers
  • Progressive logging levels based on retry count

Failure Scenarios

  • Database connection failures
  • S3 storage access issues
  • Invalid message formats
  • Processing timeouts

Development

Local Setup

  1. Install dependencies: npm install
  2. Configure environment variables
  3. Set up Azure Cosmos DB and AWS S3 credentials
  4. Run locally using Azure Functions Core Tools

Testing

  • Test file: test.js
  • Manual testing capabilities for individual components

Build Process

  • Webpack configuration for bundling
  • Terser plugin for code minification
  • Copy plugin for static assets

Deployment

Kubernetes Configuration

  • Containerized deployment
  • Health check endpoints: /live and /ready
  • Environment-specific function.json configurations

Environment Variables

  • Azure Cosmos DB connection strings
  • AWS S3 credentials
  • Application Insights instrumentation key
  • Event Hub connection strings

This service integrates with: - DocumentEventHandler: Source of document change events - CacheSync: Downstream cache synchronization - ReportGenerator: Source data for reporting - Various Handler Services: Consumers of source data

Troubleshooting

Common Issues

  1. High Retry Rates: Check Cosmos DB and S3 connectivity
  2. Missing Source Data: Verify message format and collection names
  3. Performance Issues: Monitor batch sizes and processing times
  4. Configuration Errors: Validate environment-specific settings

Monitoring Points

  • Failed event counts
  • Processing latency
  • Retry attempt frequencies
  • Storage operation success rates