Auto Model Documentation

​

Overview

​

Key Capabilities

​

Two Access Methods

• Web UI: A browser-based interface for interactive configuration and job monitoring
• CLI: A command-line tool for scripting and automation

​

Getting Started

​

Prerequisites

• Python 3.10 or higher installed
• Access to an LLM API (Anthropic Claude or OpenAI GPT-4)
• MLflow tracking server access (if documenting registered models)
• Your ML codebase accessible from the local filesystem

​

Installation

# Clone or download the repository
cd auto_model_docs

# Install in development mode
pip install -e .

# Or install dependencies directly
pip install -r requirements.txt

​

Environment Setup

# LLM API Keys (set one or both)
ANTHROPIC_API_KEY=your-anthropic-api-key
OPENAI_API_KEY=your-openai-api-key

# Optional: Custom OpenAI-compatible endpoint
OPENAI_BASE_URL=https://api.example.com/v1

# MLflow Configuration (optional)
MLFLOW_TRACKING_URI=https://your-mlflow-server.com

​

Web UI Guide

​

Accessing the Web Interface

python web_app.py

​

Configuration Panel

​

Options Panel

​

Advanced Options

​

Artifact Filtering

​

Running a Job

1. Configure your settings in the form panels
2. Click Generate Documentation to start the job
3. Monitor progress in the Logs panel

• Scanning: Analyzing code and extracting MLflow artifacts
• Planning: LLM planning content blocks for each section
• Generating: Creating narratives, tables, and charts
• Building: Assembling the final Word document

​

CLI Usage

​

Basic Command Structure

python main.py --spec <path-to-spec.yaml> [options]

​

Key Arguments

​

Filtering Options

​

Advanced Options

​

Example Commands

python main.py --spec doc_spec.yaml --provider anthropic

python main.py --spec doc_spec.yaml \
    --provider openai \
    --notebook \
    --verbose

python main.py --spec doc_spec.yaml \
    --provider anthropic \
    --models "churn_model*,fraud_detector" \
    --experiments "production_*" \
    --latest-only

python main.py --spec custom_spec.yaml \
    --code-root /path/to/ml/project \
    --output /path/to/output \
    --generation-workers 8 \
    --planning-workers 4 \
    --max-files 100

python main.py --spec doc_spec.yaml \
    --notebook-from-cache \
    --notebook-path ./updated_notebook.ipynb

​

Document Specification (YAML)

​

Spec File Structure

# Document metadata
title: "Machine Learning Model Documentation"
authors: "Data Science Team"

# Sections to include
sections:
  - Executive Summary
  - Data Overview
  - Model Architecture
  - Model Performance
  - Conclusion

# Optional hints for content guidance
hints:
  "Executive Summary": >
    Focus on business impact and key metrics.

​

Section Configuration

sections:
  - Executive Summary
  - Data Overview
  - Feature Engineering
  - Model Architecture
  - Training Pipeline
  - Model Performance
  - Validation Results
  - Deployment Considerations
  - Conclusion
  - Appendix

​

Per-Model Sections

sections:
  - Executive Summary
  - Data Overview
  - "Model Performance: per_model"  # Creates subsection for each model
  - Conclusion

• 5. Model Performance
• 5.1 Model Performance: churn_predictor
• 5.2 Model Performance: fraud_detector

​

Hints for Guiding Content

hints:
  "Executive Summary": >
    Focus on business impact, key metrics, and high-level model capabilities.
    Keep it suitable for non-technical stakeholders.

  "Data Overview": >
    Describe data sources, volume, quality considerations, and any data
    preprocessing steps applied before feature engineering.

  "Model Performance": >
    Present key metrics (accuracy, AUC, F1, etc.), confusion matrix
    insights, and performance analysis across different segments.

  "Validation Results": >
    Document the validation approach based on what is found in the code.
    Do not describe validation methods that were not actually implemented.

  "Conclusion": >
    Summarize key model capabilities and performance highlights.
    Do not introduce new technical details - synthesize what was covered.

​

Example Spec File

# Document Specification for Auto Model Documentation
title: "Machine Learning Model Documentation"
authors: "Data Science Team"

sections:
  - Executive Summary
  - Data Overview
  - Feature Engineering
  - Model Architecture
  - Training Pipeline
  - "Model Performance: per_model"
  - Validation Results
  - Deployment Considerations
  - Conclusion
  - Appendix

hints:
  "Executive Summary": >
    Focus on business impact, key metrics, and high-level model capabilities.
    Keep it suitable for non-technical stakeholders.

  "Data Overview": >
    Describe data sources, volume, quality considerations, and any data
    preprocessing steps applied before feature engineering.

  "Feature Engineering": >
    Document feature transformations, encoding methods, scaling approaches,
    and feature selection rationale.

  "Model Architecture": >
    Include model type, hyperparameters, training configuration,
    and architectural decisions.

  "Model Performance": >
    Present key metrics (accuracy, AUC, F1, etc.), confusion matrix
    insights, and performance analysis across different segments.

  "Validation Results": >
    Document the validation approach used based on what is found in the code.
    If cross-validation was implemented, cover those results.
    If only holdout/test set validation was used, document that approach.

  "Deployment Considerations": >
    Address inference latency, resource requirements, monitoring strategy,
    and model refresh cadence.

  "Conclusion": >
    Summarize the key model capabilities and performance highlights.
    Describe recommended monitoring approach based on the model type.

​

Pipeline Phases

​

Scanning Phase

• Scans Python files in the code root directory
• Extracts function definitions, class structures, and docstrings
• Identifies ML-related patterns (model training, feature engineering)
• Respects max_files limit to control scope

• Connects to MLflow tracking server
• Queries registered models and experiment runs
• Extracts metrics, parameters, tags, and artifact metadata
• Applies model/experiment filters if specified

​

Planning Phase

• Analyzes the scanned context (code + artifacts)
• Determines what content blocks are needed for each section
• Plans narratives, tables, charts, and other elements
• Considers hints from the spec file
• Runs in parallel with configurable worker count

​

Generation Phase

• Narratives: Natural language descriptions and explanations
• Tables: Structured data like metrics, parameters, feature lists
• Charts: Visualizations of metrics, performance trends, distributions
• Processes blocks in parallel for efficiency

​

Building Phase

• Creates Word document with proper formatting and styles
• Inserts tables, charts, and images
• Generates Jupyter notebook if requested
• Saves cache file for future notebook regeneration

​

Output Formats

​

Word Document (.docx)

​

Jupyter Notebook (.ipynb)

​

Cache File

​

Configuration Reference

​

Environment Variables

​

LLM Provider Settings

​

MLflow Settings

​

Parallelization Settings

​

Best Practices

​

Tips for Better Documentation

1. Write descriptive docstrings in your ML code. The scanner extracts these for context.
2. Use meaningful MLflow experiment and model names. These appear in the generated documentation.
3. Log comprehensive metrics to MLflow. The more metrics available, the richer the documentation.
4. Organize code logically. Separate feature engineering, model training, and evaluation into distinct modules.
5. Use clear function and variable names. The LLM uses these to understand code intent.

​

Recommended YAML Spec Patterns

1. Start broad, then specific:
   sections:
     - Executive Summary      # High-level overview
     - Technical Deep Dive    # Detailed technical content
     - Appendix               # Supporting details
   
2. Use per_model for multi-model projects:
   sections:
     - "Model Performance: per_model"
     - "Deployment Guide: per_model"
   
3. Provide detailed hints for critical sections:
   hints:
     "Executive Summary": >
       This section is for executive stakeholders.
       Focus on business value and ROI.
       Avoid technical jargon.
   

​

Performance Optimization

1. Limit file scanning for large codebases:
   --max-files 30
   
2. Use model filtering to focus on specific models:
   --models "production_*" --latest-only
   
3. Increase workers if you have API rate limit headroom:
   --generation-workers 8 --planning-workers 4
   
4. Use notebook caching for iterative refinement to the notebook generation without calling the LLM during development:
   # First run: full generation
   python main.py --spec doc_spec.yaml --notebook
   
   # Subsequent runs: fast notebook rebuild
   python main.py --spec doc_spec.yaml --notebook-from-cache
   
5. Enable verbose mode for debugging:
   python main.py --spec doc_spec.yaml --verbose