plugNflow

plugNflow is a config-driven, FastAPI-based microservice that provides a unified, flexible API integration gateway for interacting with multiple external APIs through a single dynamically-configurable endpoint. plugNflow enables rapid onboarding of new providers via configuration changes only—no code modifications required.

Table of Contents

1. Introduction
2. Core Concepts & Architecture
3. Deployment Overview
4. Provider Configuration Schema
5. Request & Response Flow
6. Dynamic API Documentation
7. Caching Mechanisms
8. Advanced Features
   • User-Defined Functions (UDFs)
   • Hierarchical Provider Configuration
   • Aggregator Providers
9. Example Usage Scenarios
10. Security Considerations
11. Extensibility & Next Steps
12. Appendix: Sample Provider Configuration

Introduction

In a constantly evolving API landscape, integrating diverse third-party services can be challenging due to varying authentication schemes, request/response formats, and business-specific logic.

plugNflow simplifies this by providing:

• A single HTTP POST /api endpoint for all integrations.
• Fully config-driven integration logic with provider configs stored externally.
• Dynamic generation and enforcement of request/response schemas.
• Support for provider-specific authentication and business logic.
• Advanced orchestration capabilities, including caching, hierarchical calls, and aggregators.

This allows organizations to seamlessly add or update API integrations by modifying configuration rather than code, accelerating development while maintaining robustness and observability.

Core Concepts & Architecture

• Single Endpoint, Multiple Providers: All client requests come through the single /api endpoint, specifying the target provider by name in the request payload.
• Provider Configurations: Define how each external integration works — authentication, request construction, response parsing, pre/post processing, caching, etc.
• Dynamic Schemas: OpenAPI schemas for requests and responses are dynamically built from provider configs and surfaced in /api/docs for clear discoverability.
• Request Lifecycle: plugNflow enriches, validates, processes, forwards, and maps requests and responses based solely on config.
• Caching: Multi-layer caching optimizes both plugNflow responses and external service interactions.
• Extensible Processing: Incorporate complex, custom logic with user-defined Python functions plugged into pre/post-processing hooks.

Deployment Overview

• Built on FastAPI, plugNflow is deployable on any standard API/cloud infrastructure.
• Provider configs are stored and managed in a database (e.g., MongoDB Atlas).
• Enabled providers are controlled via environment variables; configs are loaded and validated at service startup.
• API-level authentication (protecting /api) is handled externally in deployment, while external API auth is managed within provider configs.
• Careful use of environment variables ensures secrets are injected only at runtime, never stored in config DB.

Provider Configuration Schema

Provider configs conform to a strongly typed Pydantic schema composed of:

• ProviderConfig: Top-level config for a provider, includes name, description, aggregator flag, and operations.
• Operations:
  • internal: Handles DB operations, response schemas, and hash keys for caching.
  • external: Specifies external API calls including authentication and main core calls, headers, payload formats, and caching of tokens.

Environment variables referenced as OSENV[VAR_NAME] in configs are resolved at runtime, keeping secrets secure.

Request & Response Flow

1. Client sends a POST request to /api specifying:
   • "provider": provider name as configured
   • "transaction_payload": list of input data
2. plugNflow validates the request as per provider schema.
3. If present and configured, the transaction_id is used to fetch additional payload data from internal DB or services.
4. Pre-processing hooks (local or external UDFs) transform the request.
5. plugNflow authenticates to external APIs as configured, managing token caching.
6. External API calls are made with constructed payloads.
7. Post-processing runs to map and transform external responses.
8. The unified response is assembled and returned to client.
9. Caching reduces redundant external calls as configured.

Dynamic API Documentation

• On startup, plugNflow loads enabled providers and dynamically builds the /api/docs OpenAPI spec.
• This includes enumerated provider options, and provider-specific request and response schemas.
• Adding a new provider config and enabling it triggers inclusion in docs after restart.
• This approach maintains up-to-date API contracts usable by client teams without extra manual documentation efforts.

Caching Mechanisms

1. plugNflow Provider-Level Response Caching

• What: Cache responses per provider keyed by hash fields defined in provider config [combined with transaction_id (if present)].
• How: For repeated requests with identical required fields, plugNflow returns cached response rather than calling external APIs again.
• Always enabled: This cache is ON by design.
• Force refresh: Send "force_update": true in request to bypass cache and trigger fresh external call.

2. External Service-Level Caching

• What: Cache intermediate data such as authentication tokens or reusable external service responses, configured inside provider Authentication or Core sections.
• How: Cached tokens are reused until expiry unless invalidated.
• Configurable: Controlled via cacheConfig block in provider config.

Advanced Features

User-Defined Functions (UDFs)

• For complex pre/post processing, plugNflow supports referencing Python functions from a dynamic module (default: external.py).
• Configured preProcess/postProcess steps define a sequence of such function calls.
• Future plan: support user-uploadable or configurable UDF modules referenced by environment variables.

Hierarchical Provider Configuration

• Providers can be arranged parent-child.
• Child provider calls can specify a parent_id referencing entities from parent provider responses.
• plugNflow fetches parent entity data automatically to build child requests if transaction_payload is not provided explicitly.

Aggregator Providers

• Providers can be configured to be "is_aggregator": true. This means that this provider would consolidate the children services reponse too while responding.
• These do NOT automatically call child providers.
• Instead, when responding, plugNflow includes previously fetched and cached data from child providers related to items in the parent response.
• This returns a consolidated dataset combining parent and child data without runtime downstream calls.

Example Usage Scenarios

Scenario 1: Basic Provider Call

Send a request to /api specifying provider child_service_example and payload:

{
  "provider": "child_service_example",
  "transaction_payload": [
    {
      "goods_desc": "cillum tempor quis elit2",
      "hs_code": "68056334",
      "transaction_id": "render_txn_1"
    }
  ]
}

Scenario 2: Transaction ID Lookup

Request includes a transaction_id; plugNflow queries internal DB (transaction_data collection) to build or enrich payload before sending to external APIs.

Scenario 3: Hierarchical Calls

Provider list_vessels returns vessel IDs; client calls provide_vessel_history_info with a parent_id referencing an ID from the previous response. plugNflow auto-fetches cached parent data for child request construction.

Scenario 4: Aggregator Provider

• Client calls a provider configured with "is_aggregator": true, e.g., list_vessels.
• plugNflow uses already cached data from child providers to enrich the response.
• No new downstream API calls to child providers are made at this point.

Security Considerations

• Access to the plugNflow /api endpoint is secured at deployment level (API keys, JWT, Basic Auth)—never handled in provider configs.
• External API credentials are supplied via OSENV[...] to avoid secrets in config storage.
• All requests are validated against strict Pydantic schemas.
• Rate limiting and careful CORS policy are recommended.
• Exercise caution with UDF execution—sandbox or vet code uploaded/run externally.

Extensibility & Next Steps

• Implement hot-reloading for provider configs for zero-downtime config changes.
• Enable secure uploading/configuration of external UDF modules for custom logic.
• Add versioning and rollback for provider configs.
• Enhance aggregator providers with customizable aggregation pipelines.
• Automate client SDK generation from dynamic OpenAPI schema.
• Implement structured logging and tracing for complex, multi-provider flows.

Appendix: Sample Provider Configuration

{
  "name": "child_service_example",
  "description": "plugNflow example child service",
  "is_aggregator": false,
  "operations": {
    "internal": {
      "db_operations": {
        "transactionSource": [],
        "preProcess": [],
        "postProcess": []
      },
      "hashkeys": [],
      "main_response_schema": {
        "success_status": "OPEN",
        "default_status": "OPEN",
        "schema": {
          "title": "SnpFetchDug",
          "type": "object",
          "properties": {
            "id": {
              "title": "Id",
              "type": "string"
            },
            "parentId": {
              "title": "Id",
              "type": "string"
            },
            "service": {
              "title": "service",
              "type": "string"
            },
            "status": {
              "$ref": "#/definitions/Status"
            },
            "result": {
              "$ref": "#/definitions/SnpDugResult"
            }
          },
          "required": [
            "id",
            "status"
          ]
        }
      }
    },
    "external": {
      "Authentication": {
        "authenticationMethod": "Default",
        "httpMethod": "POST",
        "endpoint": "https://sam.ihsmarkit.com/",
        "slug": "sso/oauth2/access_token",
        "headers": {
          "Content-Type": "application/x-www-form-urlencoded"
        },
        "requestPayload": {
          "type": "urlencoded",
          "params": {
            "client_id": "OSENV[plugNflow_EXAMPLE_CHILD_ID]",
            "client_secret": "OSENV[plugNflow_EXAMPLE_CHILD_SECRET]",
            "grant_type": "client_credentials",
            "scope": "openid"
          }
        },
        "responsePayload": {
          "token": "<access_token>"
        },
        "cacheConfig": {
          "enabled": true,
          "expiry": 0
        }
      },
      "Core": {
        "httpMethod": "POST",
        "endpoint": "https://www.dualusegoods.ihsmarkit.com/",
        "slug": "V2/dualusegoodsclassifier",
        "headers": {
          "Authorization": "<token>",
          "accept": "application/json",
          "Content-Type": "application/json"
        },
        "requestPayload": {
          "type": "json",
          "json": {
            "Input_Text": [
              "{hs_code}||{goods_description_compliance}||{goods_desc}"
            ],
            "Regions_Filter": "{region_code}||{region_options}"
          }
        },
        "responsePayload": {
          "type": "json",
          "params": {
            "input_description": "{Input_Description}",
            "dual_usage_code": "{Dual_Usage_Code}",
            "reason_for_match": "{Reason_for_Match}",
            "es_primary_keyword": "{ES_Primary_Keyword}",
            "dual_usage_flag": "{Dual_Usage_Flag}",
            "dual_usage_flag_prob": "{Dual_Usage_Flag_Prob}",
            "commodity_details": [
              {
                "commodity_source_pdf": "{Commodity_Details.Commodity_Source_PDF}",
                "commodity_source": "{Commodity_Details.Commodity_Source}",
                "country": "{Commodity_Details.Country}",
                "regional_code": "{Commodity_Details.Regional_Code}",
                "commodity_category_description": "{Commodity_Details.Commodity_Category_Description}"
              }
            ]
          }
        }
      }
    }
  }
}