Skip to content

DocumentCRUDV2 Service

Overview

The DocumentCRUDV2 is a robust Azure Function-based microservice that provides comprehensive CRUD (Create, Read, Update, Delete) operations for document management with built-in authentication, event publishing, and dynamic model support. This is the second version of the DocumentCRUD service with enhanced features and improved architecture.

Business Purpose

This service serves as a centralized document management system that: - Handles RESTful CRUD operations for documents across multiple platforms - Provides JWT-based authentication and authorization using Microsoft Service tokens - Integrates with Azure Cosmos DB for data persistence - Publishes events to Azure Event Hub for downstream processing - Supports batch operations for improved performance - Implements dynamic model loading for flexible document schemas - Manages complex nested entity relationships

Architecture

Service Type

  • Platform: Azure Functions (Containerized Kubernetes Microservice)
  • Runtime: Node.js
  • Trigger: HTTP Trigger (Anonymous authentication with JWT validation)
  • Pattern: RESTful API with Event-Driven Architecture

Key Components

graph TD
    A[HTTP Request] --> B[Authentication Layer]
    B --> C{Valid MS Token?}
    C -->|No| D[401 Unauthorized]
    C -->|Yes| E[Handler.js]

    E --> F[URL Parser]
    F --> G[Dynamic Model Loader]
    G --> H[NoSQL Client]

    H --> I{Operation Type?}
    I -->|GET| J[Read Operation]
    I -->|POST| K[Create Operation]
    I -->|PUT| L[Update Operation]
    I -->|DELETE| M[Delete Operation]

    J --> N[Cosmos DB Query]
    K --> O[Document Creation]
    L --> P[Document Update]
    M --> Q[Document Deletion]

    O --> R[Event Publishing]
    P --> R
    M --> R

    R --> S[Event Hub]
    N --> T[Response]
    R --> T

    U[Auth Exchange Service] --> B
    V[Dynamic Models] --> G

Authentication

Microsoft Service Token Authentication

All requests require a valid Microsoft Service (MS) access token in the Authorization header: 

Authorization: Bearer <ms_access_token>

Important: You must use an ms_access_token, not a Delty access token.

Authentication Flow

  1. Service extracts token from Authorization: Bearer <token> header
  2. Forwards token to configured authentication endpoint:
  3. DEV: https://test-publisher.delty.com/kube/authexchange/ms/token
  4. STAGE: https://stage-publisher.delty.com/kube/authexchange/ms/token
  5. PROD: https://publisher.delty.com/kube/authexchange/ms/token
  6. If validation succeeds, request proceeds
  7. If validation fails, returns 401 Unauthorized

Token Requirements

  • Must be a valid Microsoft Service token
  • Must not be expired
  • Must be recognized by the authentication service

API Specification

URL Structure

The service uses a RESTful URL pattern: 

/platform/{platformname}/{entity}/{id?}/{subentity?}/{subid?}

Important: The {id} in the URL represents the document ID, not the partition key. Partition keys are automatically derived from the URL structure.

URL Structure and Data Mapping

Single Entity Level

/platform/{platformname}/{entity}/{id}
- Document ID: {id} (from URL) - Partition Key: {platformname} (automatically set) - Container: {platformname}-{entity}

Nested Entity Level

/platform/{platformname}/{entity1}/{id1}/{entity2}/{id2}
- Document ID: {id2} (last ID in URL) - Partition Key: {id1} (parent entity ID - automatically set) - Container: {platformname}-{entity2}

HTTP Operations

GET Operations

  • Get all documents: GET /platform/{platformname}/{entity}
  • Returns all documents in the partition
  • Get specific document: GET /platform/{platformname}/{entity}/{id}
  • Retrieves document by ID with automatic partition key derivation

POST Operations

  • Create document: POST /platform/{platformname}/{entity}
  • ID auto-generated if not provided in request body
  • Partition key automatically set to {platformname}
  • Create sub-document: POST /platform/{platformname}/{entity}/{id}/{subentity}
  • ID auto-generated if not provided
  • Partition key automatically set to {id} (parent entity)

PUT Operations

  • Update document: PUT /platform/{platformname}/{entity}/{id}
  • Updates existing document with automatic partition key derivation
  • Update sub-document: PUT /platform/{platformname}/{entity}/{id}/{subentity}/{subid}
  • Updates nested entity with parent ID as partition key

DELETE Operations

  • Delete document: DELETE /platform/{platformname}/{entity}/{id}
  • Deletes document with automatic partition key derivation
  • Delete sub-document: DELETE /platform/{platformname}/{entity}/{id}/{subentity}/{subid}
  • Deletes nested entity with parent ID as partition key

Request/Response Format

Request Body (POST/PUT)

{
    "id": "optional-document-id",
    "name": "Document Name",
    "data": { /* document-specific data */ }
}

Response Format

{
    "action": "Create|Update|Delete|Read",
    "collection": "container-name",
    "record": { /* document data */ },
    "status": 200
}

Core Functionality

Dynamic Model Loading

  • Automatically loads models from src/models/ directory
  • Supports custom ID generation per model type
  • Flexible schema validation and processing

Batch Operations

  • Supports processing multiple operations in a single request
  • Maintains transaction consistency where possible
  • Optimized for high-throughput scenarios

Event Publishing

Automatically publishes events to Azure Event Hub for: - Document creation (POST) - Document updates (PUT) - Document deletion (DELETE)

Events include: - Authentication details - Operation metadata - Document data - Unique record ID

Dependencies

External Services

  • Azure Cosmos DB: Primary data storage
  • Azure Event Hub: Event publishing for downstream processing
  • AuthExchange Service: Microsoft Service token validation

Key NPM Packages

  • @azure/cosmos: Azure Cosmos DB client
  • jsonwebtoken: JWT token handling
  • node-fetch: HTTP client for authentication
  • idgen: ID generation utilities

Configuration

Environment-Specific Settings

  • Development: Basic logging and test endpoints
  • Integration: Integration testing environment
  • Production: Production endpoints with enhanced security

Required Configuration

{
    "endpoints": {
        "auth": "https://your-auth-endpoint.com"
    },
    "LogLevel": "debug",
    "PublisherCosmos": {
        "endpoint": "cosmos-db-endpoint",
        "masterKey": "cosmos-db-key",
        "database": "database-name"
    }
}

Performance Characteristics

Scalability Features

  • Horizontal scaling through Azure Functions
  • Partitioned Cosmos DB storage for optimal performance
  • Batch operation support for high-throughput scenarios
  • Dynamic model loading for flexible schema management

Processing Metrics

  • Throughput: ~1000 operations per second
  • Latency: <200ms average response time
  • Availability: 99.9% uptime with multi-region deployment
  • Consistency: Strong consistency for critical operations

Security Considerations

  • Authentication: Microsoft Service token validation
  • Authorization: Role-based access through token claims
  • Data Privacy: Secure handling of sensitive document data
  • Audit Trail: Comprehensive logging for compliance
  • Network Security: HTTPS-only communication

Monitoring and Observability

Logging

  • Structured logging with configurable levels
  • Request/response logging for debugging
  • Authentication event logging
  • Performance metrics logging

Metrics

  • Operation success/failure rates
  • Response time percentiles
  • Authentication success rates
  • Document operation volumes by type

Error Handling

Error Scenarios

  1. Authentication Failures: Invalid or expired tokens (401)
  2. Validation Errors: Malformed requests or data (400)
  3. Not Found: Requested documents don't exist (404)
  4. Server Errors: Database or service failures (500)

Error Response Format

{
    "error": "Error description",
    "status": 400,
    "details": "Additional error context"
}

This service integrates with the broader Publisher ecosystem: - DocumentCacheHandler: Consumes document change events - AuthExchange: Provides token validation services - Publisher Portal: Primary consumer of CRUD operations - Analytics Services: Consume document change events

Development

Local Development Setup

  1. Clone repository
  2. Install dependencies: npm install
  3. Configure environment settings
  4. Set up Cosmos DB connection
  5. Configure Event Hub connection
  6. Run tests: npm test
  7. Start locally: func start

Code Structure

  • index.js: Azure Function entry point with authentication
  • src/Handler.js: Core CRUD operations and request processing
  • src/nosql/NoSQLClient.js: Cosmos DB integration
  • src/models/: Dynamic document models and schemas
  • config/: Environment-specific configurations

Troubleshooting

Common Issues

  1. Authentication Failures: Verify MS token validity and endpoint configuration
  2. Document Not Found: Check partition key derivation and document ID
  3. Performance Issues: Review batch operation usage and indexing
  4. Event Publishing Failures: Verify Event Hub connection and permissions

Debug Steps

  1. Check Application Insights for request traces
  2. Verify authentication endpoint connectivity
  3. Review Cosmos DB query patterns and performance
  4. Monitor Event Hub message flow