feat: add AI documentation check workflow for pull requests #20328
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This workflow automatically analyzes PRs to identify documentation updates needed. Features: - Triggers on PR opened/synchronize/reopened events - Posts initial comment when starting analysis - Creates Coder task to analyze code changes against existing docs - Identifies updates needed, deprecated docs, and new docs required - Updates PR comment with actionable documentation recommendations Includes: - .github/workflows/documentation-check.yaml: GitHub Actions workflow - scripts/documentation-check.sh: Helper script for task management
Fixes YAML validation errors by using proper heredoc syntax (cat <<EOF) instead of raw multi-line strings in bash run blocks.
- Add comprehensive 'Determine Inputs' step matching triage workflow - Extract GitHub user ID lookup to separate step - Update all step references to use determine-inputs outputs - Fix heredoc formatting to properly handle backticks and variable expansion - Maintain PR comment posting flow (initial + update)
- Replace heredoc with simple quoted strings for comment bodies - Ensures proper variable expansion for GITHUB_REPOSITORY and RUN_ID - Maintains consistency with bash best practices in GitHub Actions
- All heredoc content must be indented to match bash script context - Prevents YAML parser from interpreting bash script content as YAML keys - Validates successfully with PyYAML parser
- Only pass PR URL to task instead of full diff, title, body, files - Instructs Claude Code to fetch PR details using GitHub CLI/API - Prevents token limit issues with large PRs - Reduces workflow complexity and execution time - Claude Code in task has full access to GitHub tools anyway
David got frustrated with having to scroll so much so see the og comment below |
- Track start/end time of task execution - Calculate and format duration (minutes and seconds) - Display execution time in final PR comment with clock emoji - Helps users understand task performance
David got frustrated with having to scroll so much so see the og comment below |
- Add clear delimiter markers (---DOCUMENTATION-ANALYSIS-START/END---) to prompt - Update script to extract only content between markers - Provides cleaner output: just the bulleted recommendations or 'no changes needed' - Removes all the tool calls, debugging, and conversation from PR comment - Falls back to task link if markers not found
|
🤖 Documentation Check Complete ⏱️ Execution time: 3m 26s |
- Fetch original comment before updating - Append results below a horizontal divider - Keeps 'Started' message visible for full timeline - Add emojis to metadata (⏱️ 📋 🔗) for better visual separation
|
🤖 Documentation Check Started Analyzing PR changes to determine if documentation updates are needed... ⏱️ Execution time: 4m 47s |
- Update comment immediately after task creation - Shows 'Analysis in progress...' with task link - Allows users to follow along in real-time - Three-step progression: Started -> In Progress (with link) -> Complete
|
🤖 Documentation Check Started Analyzing PR changes to determine if documentation updates are needed... 🔄 Analysis in progress... 📋 Task: https://dev.coder.com/tasks/david-fraley/doccheck-pr-20328-18543452075 ⏱️ Execution time: 5m 47s |
Use PR number only (not RUN_ID) for task naming to ensure consistent task reuse across workflow re-runs. This prevents mismatched task URLs in PR comments and allows the task to accumulate conversation history for the same PR.
|
🤖 Documentation Check Started Analyzing PR changes to determine if documentation updates are needed... 🔄 Analysis in progress... 📋 Task: https://dev.coder.com/tasks/david-fraley/doccheck-pr-20328 ⏱️ Execution time: 6m 7s |
Remove the automatic task deletion step to match traiage workflow behavior. This allows users to review the task logs after the workflow completes, since the results are already posted to the PR comment.
|
🤖 Documentation Check Started Analyzing PR changes to determine if documentation updates are needed... |
johnstcn
reviewed
Oct 16, 2025
| [View workflow run](https://github.com/${GITHUB_REPOSITORY}/actions/runs/${RUN_ID}) | ||
| EOF | ||
| ) | ||
| COMMENT_ID=$(gh pr comment "${PR_NUMBER}" --body "${COMMENT_BODY}" --repo "${GITHUB_REPOSITORY}" | grep -oP 'https://github.com/.*/pull/.*#issuecomment-\K\d+') |
There was a problem hiding this comment.
suggestion: --create-if-none --edit-last
Implement Cian's suggestion to use --create-if-none and --edit-last flags instead of manually tracking comment IDs. This simplifies the workflow and ensures only one comment is created/updated per PR. Benefits: - Simpler code (no need to track COMMENT_ID) - No need to fetch and preserve original comment - Guarantees single comment per PR - Cleaner PR comment history
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds an automated workflow that uses Coder Tasks and Claude Code to analyze pull requests and identify potential documentation updates that might be needed.
How It Works
Implementation Details
Workflow:
.github/workflows/documentation-check.yamltraiage.yamlworkflowScript:
scripts/documentation-check.shcreate,wait,summary,deletetraiage.shKey Features
✅ Automatic triggering on PR events
✅ User-friendly comments with status updates
✅ Scalable - only passes PR URL, not full diff (prevents token limits)
✅ Actionable recommendations with specific file paths
✅ Proper cleanup of resources after completion
✅ Access control - verifies push permissions
✅ Manual triggering via workflow_dispatch for testing
Testing
The workflow can be manually triggered using:
Files Changed
.github/workflows/documentation-check.yaml- Main workflow definitionscripts/documentation-check.sh- Task management scriptNext Steps
🤖 This workflow will help ensure documentation stays in sync with code changes!