Redis Cache Sync Service

Introduction

This service synchronizes data between blob storage and Redis cache using an event-driven architecture. It receives events from an event service and processes them to update the Redis cache. The service can fetch data from URLs mentioned in the events and perform various cache operations like setting, getting, and deleting data.

Architecture

Flow Diagram

Below is a visual representation of the service's architecture and data flow:

flowchart TB
    EventService[Event Service] --> EventReceiver[Event Receiver]

    subgraph CacheSync["Redis Cache Sync Service"]
        EventReceiver --> Validation[Event Validation]
        Validation -->|Valid Event| EventProcessor[Event Processor]
        Validation -->|Invalid Event| ErrorHandler[Error Handler]

        EventProcessor -->|URL in Event| HttpClient[HTTP Client]
        HttpClient -->|Fetch Blob Data| DataProcessor[Data Processor]

        EventProcessor -->|No URL| CacheOps[Cache Operations]
        DataProcessor --> CacheOps

        CacheOps -->|SET| RedisSet[Set in Redis]
        CacheOps -->|GET| RedisGet[Get from Redis]
        CacheOps -->|DELETE| RedisDelete[Delete from Redis]
        CacheOps -->|DELETE_PATTERN| RedisDeletePattern[Delete by Pattern]
    end

    RedisSet --> Redis[(Redis DB)]
    RedisGet --> Redis
    RedisDelete --> Redis
    RedisDeletePattern --> Redis

    BlobStorage[(Blob Storage)] --> HttpClient

Components

1. Event Processor - Receives and validates events from the event service
2. HTTP Client - Fetches data from blob storage URLs provided in events
3. Cache Service - Manages Redis connections and cache operations
4. Health Checks - Provides /live and /ready endpoints for Kubernetes health monitoring

Data Flow

1. Service receives an event from the event service
2. Event is validated and parsed
3. If the event contains blob URLs, the service fetches data from those URLs
4. Data is processed according to event type (SET, GET, DELETE, DELETE_PATTERN)
5. Redis cache is updated with the processed data
6. Service logs the operation and returns appropriate response

Getting Started

Prerequisites

1. Node.js (v14.x or higher)
2. Redis server
3. Kubernetes cluster (for production deployment)

Installation process

1. Clone this repository:

   git clone <repository-url>
   

2. Install dependencies:

   npm install
   

3. Configure environment variables:

4. REDIS_HOST: Redis server hostname (default: "localhost")
5. REDIS_PORT: Redis server port (default: 6379)
6. CLUSTER: Current Kubernetes cluster name
7. LOG_LEVEL: Logging verbosity (debug, info, warn, error)
8. PORT: HTTP port for health checks and metrics

Running locally

npm start

Deployment

The service is containerized and deployed to Kubernetes using CI/CD pipelines.

API References

Event Structure

Events received by the service must conform to the following structure:

{
  "key": "<event-type>",
  "value": {
    "data": "<operation-data>",
    "ttl": "<cache-ttl-in-seconds>"
  }
}

Supported Event Types

• SET: Store data in Redis cache
• GET: Retrieve data from Redis cache
• DELETE: Remove specific keys from Redis cache
• DELETE_PATTERN: Remove keys matching a pattern from Redis cache

Build and Test

Building the project

npm run build

This uses Webpack to bundle the application for production.

Running tests

npm test

Maintenance

Clearing Redis Cache

For maintenance purposes, you might need to clear the Redis cache. Use the following command with caution:

# Preview what would be deleted (dry run)
find . -maxdepth 1 -mindepth 1 -print

# Delete all files and directories in current directory
find . -maxdepth 1 -mindepth 1 -print0 | xargs -0 rm -rf

⚠️ Warning: Always verify your current directory before running delete commands.

Contribute

To contribute to this project:

1. Fork the repository
2. Create a feature branch
3. Implement your changes
4. Write tests for your changes
5. Submit a pull request

Please ensure your code follows the project's coding standards and includes appropriate tests.