Enhanced OpenGraph Data Fetching with Improved Error Handling

Added

  • Robust error handling with retry logic for OpenGraph API calls
  • Parallel processing for screenshot fetches
  • Comprehensive reporting with clear statistics
  • Utility functions for consistent report naming and frontmatter

Enhanced

  • Switched to plain text parsing for safer YAML handling
  • Improved file safety with atomic write operations
  • Added detailed function documentation and cross-references
  • Updated report template with screenshot counts

Fixed

  • Screenshot fetch error handling and propagation
  • Memory management for large directory processing
  • File path formatting in reports
  • Error tracking in statistics

Technical Details

New Utilities

  1. addReportNamingConventions.cjs
    • Generates unique report filenames with auto-incrementing indices
    • Ensures directory existence with proper permissions
    • Format: YYYY-MM-DD_reportName_runIndex.md
  2. addReportFrontmatterTemplate.cjs
    • Provides consistent frontmatter formatting
    • Supports customizable fields with sensible defaults
    • Maintains proper YAML structure

Core Script Improvements

  • Added retry logic with exponential backoff
  • Implemented parallel processing with proper tracking
  • Enhanced error categorization and reporting
  • Added comprehensive statistics tracking

Impact

  • More reliable OpenGraph data fetching
  • Clearer error reporting and tracking
  • Improved maintainability through documentation
  • Better scalability for large directories

Usage Notes

The script can be run with:
bash 
node scripts/build-scripts/runFetchOpenGraphData.cjs
Environment variables:
  • TARGET_DIR: Directory to process (default: '../content/tooling/AI-Toolkit')
  • REPORT_OUTPUT_DIR: Report output location (default: 'src/content/data_site')
  • OPEN_GRAPH_IO_API_KEY: Required API key for OpenGraph.io