Maintain a Proprietary Extended Markdown Flavor Rendering Pipeline

Current Tasks:

Add Internal Embeds:

We may have code in the file:

Extended Markdown Rendering Pipeline Specification

1. System Overview

1.1 Core Technologies

• Astro: Primary framework and build system
• Unified: Text processing framework
• Remark: Markdown processor and plugin system
  • remark-gfm: GitHub Flavored Markdown support
    • Tables with alignment
    • Task lists (- [ ] and - [x])
    • Strikethrough (~~text~~)
    • URL autolinking
    • Footnotes
• Rehype: HTML processor and plugin system
• TypeScript: Type system and development language

1.2 Key Architecture

site/src/layouts/OneArticle.astro
site/src/components/markdown/AstroMarkdown.astro

Important Component Files

site/src/components/markdown/TableOfContents.astro
site/src/components/markdown/ImageGallery.astro
site/src/components/markdown/ToolingGallery.astro

Important Utility Files

site/src/utils/markdown/
├── remark-callout.ts       # Callout block processing
├── remark-citations.ts     # Citation handling
├── remark-images.ts        # Image processing
├── remark-backlinks.ts     # Wiki-link processing
├── remark-codeblocks.ts    # Custom code block handling
├── remark-toc.ts          # Table of contents generation
├── normalizeShellLangs.ts  # Shell language normalization
└── remark-mermaid.ts      # Mermaid diagram support

2. Custom Markdown Extensions

2.1 Callouts

• Syntax: > [!NOTE] or > [!WARNING]
• Processed by: remark-callout.ts
• Rendered by: Custom callout components
• Features:
  • Multiple callout types (NOTE, WARNING, TIP, etc.)
  • Nested content support
  • Custom styling per callout type

2.2 Citations

• Syntax: [^1] or [^citation-key]
• Processed by: remark-citations.ts
• Rendered as: Interactive footnotes with backlinks
• Features:
  • Automatic numbering
  • Reference list generation
  • Backlink support

2.3 iFrames

• Processed by: Custom iframe handling
• Features:
  • Responsive embeds
  • Lazy loading
  • Custom aspect ratios

2.4 Wiki-Links

• Syntax: [[Page Name]] or [[Display Text|Page Name]]
• Processed by: remark-backlinks.ts
• Features:
  • Automatic path resolution
  • Backlink generation
  • Custom display text support

2.5 Custom Code Blocks

• Processed by: remark-codeblocks.ts
• Features:
  • Syntax highlighting
  • Line numbers
  • Copy-to-clipboard
  • Special handling for:
    • Mermaid diagrams
    • Image galleries
    • Tooling galleries

2. Extension Types and Processing Order

2.1 Processing Pipeline Order

1. Container Elements (First Pass)
   • Callouts
   • Alert blocks
   • Custom containers
2. Inline Elements (Second Pass)
   • Wiki-links (backlinks)
   • Citations
   • Custom inline syntax
3. Rich Content (Third Pass)
   • iFrames
   • Images
   • Code blocks
4. Final Processing (Fourth Pass)
   • HTML generation
   • Component mapping
   • Style application

2.2 Supported Extensions

1. Callouts
   • Syntax: > [!NOTE] or similar
   • Types: NOTE, WARNING, INFO, etc.
   • Nested content support
   • Citation compatibility
2. Citations
   • Inline: [1], [2], etc.
   • Section: Citations: header
   • URL support
   • Automatic collection and rendering
3. iFrames
   • Raw HTML support
   • YouTube/Loom optimizations
   • Responsive containers
   • Attribute preservation
4. Wiki-Links
   • Double bracket syntax: [[page]]
   • Title support: [[page|title]]
   • Automatic backlink generation
   • Path resolution
5. Custom Code Blocks
   • Language-specific highlighting
   • Custom block types
   • Metadata support
   • Component mapping

Custom Codeblocks in Production

Ideas for Custom Codeblocks

3. Implementation Details

3.1 AST Processing Strategy

1. Detection Phase
   typescript

   // Example for callouts
   visit(tree, 'blockquote', (node, index, parent) => {
     if (isCallout(node)) {
       const calloutNode = transformToCallout(node);
       replaceNode(parent, index, calloutNode);
     }
   });

2. Transformation Phase
   typescript

   // Common node transformation pattern
   interface TransformedNode extends Parent {
     type: string;
     data: {
       hName: string;
       hProperties: Record<string, unknown>;
     };
   }

3. Component Mapping
   typescript

   // In AstroMarkdown.astro
   const HANDLED_TYPES = {
     callout: ArticleCallout,
     citation: ArticleCitations,
     html: Fragment,
     // ... other mappings
   };

3.2 Type Safety

1. Node Type Definitions
   typescript

   // In site/src/types/mdast.d.ts
   declare module 'mdast' {
     interface StaticPhrasingContent {
       type: string;
       value?: string;
       data?: {
         hName?: string;
         hProperties?: Record<string, unknown>;
       };
     }
   }

2. Plugin Options
   typescript

   interface PluginOptions {
     debug?: boolean;
     preserveOriginal?: boolean;
     // Extension-specific options
   }

3.3 Error Handling

1. Graceful Degradation
   typescript

   try {
     // Process extension
     return transformedNode;
   } catch (error) {
     console.error(`Error processing ${type}:`, error);
     return originalNode; // Preserve original content
   }

2. Debug Output
   typescript

   if (options.debug) {
     astDebugger.writeDebugFile(
       `${phase}-${type}`,
       JSON.stringify(node, null, 2)
     );
   }

4. Component Architecture

4.1 Base Components

1. AstroMarkdown
   • Entry point for rendering
   • Node type routing
   • Error boundary
2. Extension Components
   • Specialized rendering
   • Prop validation
   • Style encapsulation

4.2 Style Management

1. CSS Modules
   css

   /* Extension-specific styles */
   .extension-container {
     width: 100%;
     box-sizing: border-box;
   }
   
   .text-wrapper {
     display: inline-block;
     width: 100%;
     word-wrap: break-word;
     word-break: break-word;
     overflow-wrap: break-word;
     hyphens: auto;
   }

2. Responsive Design
   css

   /* Common responsive patterns */
   .responsive-container {
     position: relative;
     padding-bottom: 56.25%;
     height: 0;
   }
   
   .responsive-iframe {
     position: absolute;
     top: 0;
     left: 0;
     width: 100%;
     height: 100%;
   }

5. Configuration

5.1 Astro Config

// astro.config.mjs
export default defineConfig({
  markdown: {
    remarkPlugins: [
      remarkGfm, // GitHub Flavored Markdown support
      remarkCalloutHandler,
      remarkBacklinks,
      remarkImages,
      remarkCitations
    ],
    remarkRehype: {
      allowDangerousHtml: true,
      handlers: defListHastHandlers
    },
    rehypePlugins: [
      rehypeRaw
    ]
  }
});

5.2 Plugin Registration

// Plugin registration pattern
export default function remarkExtension(options: Options = {}) {
  return (tree: Root) => {
    // 1. Detection
    const nodes = detectNodes(tree);
    
    // 2. Transformation
    const transformed = transformNodes(nodes);
    
    // 3. Insertion
    return insertNodes(tree, transformed);
  };
}

6. Testing and Validation

6.1 Test Files

1. Basic Features
   • content/vocabulary/agile.md
   • Tests common extensions
2. Complex Cases
   • Nested extensions
   • Mixed content types
   • Edge cases

6.2 Debug Process

1. AST Inspection
   • Debug files per phase
   • Node structure validation
   • Type checking
2. Visual Validation
   • Component rendering
   • Style application
   • Responsive behavior

7. References

1. Handle iFrames with our AST Rendering Pipeline
2. Rendering Extended Markdown through AST
3. Handle Custom Codeblocks in Astro
4. Handle Citations in Markdown Content
5. Handle Callouts in Markdown Content
6. Remark Plugin Implementation
7. Rendering Extended Markdown like Astro Big Doc

File Structure of Working Code

Core Processing

site/src/utils/markdown/
├── debug/
│   └── markdown-debugger.ts      # AST debugging utilities
├── plugins/
│   ├── remark-callout.ts        # Callout processing
│   ├── remark-backlinks.ts      # Wiki-link handling
│   ├── remark-citations.ts      # Citation processing
│   └── remark-images.ts         # Image handling
└── types/
    ├── mdast.d.ts              # Core AST types
    ├── mdast-callout.d.ts      # Callout types
    └── mdast-citation.d.ts     # Citation types

Component Structure

site/src/components/markdown/
├── AstroMarkdown.astro         # Main renderer
├── callouts/
│   └── ArticleCallout.astro    # Callout component
├── citations/
│   └── ArticleCitations.astro  # Citations component
└── common/
    └── ResponsiveContainer.astro # Shared components

Image Handling

Wiki-Style Image Syntax

1. Complex path normalization between assets/ and visuals/ directories
2. AST transformation issues with maintaining proper node types
3. Performance concerns with local image serving
4. Lack of image optimization capabilities

• Simplified markdown processing
• CDN-based image delivery
• Built-in image optimization
• Automatic format conversion
• Responsive images support