matt/homelab-docker
Watch
1
0
Fork
You've already forked homelab-docker
 0
Code Issues 8 Pull requests 1 Projects Releases Wiki Activity Actions

Forgejo merge API half-completes: commits land on main but PR stays unmerged and push webhook never fires (Komodo deploys silently skipped) #204

New issue
Closed
opened 2026-07-27 17:44:56 +00:00 by claude-bot · 1 comment
claude-bot commented 2026-07-27 17:44:56 +00:00
Collaborator
Copy link

Incident (2026-07-24, PR #200)

Renovate patch-group PR #200 was merged at 08:00 UTC � merge commit bb4a7c5e + e74fa4d4 landed on main � but:

  • Forgejo left the PR merged: false; Renovate autoclosed it an hour later (title suffix "- autoclosed")
  • The push webhook never fired, so Komodo never ran Deploy Stacks
  • dashy 4.4.12, home-assistant 2026.7.4, mongo 8.3.7 sat undeployed for 2+ days until manually deployed 2026-07-26

Same signature as PR #189 (issue #195), and likely the same root cause as the intermittent 405 "please try again later" from the merge API. Forgejo version at time of incident: 15.0.2.

Mitigations in place

  1. PR #203 � ignoreTests: true in renovate.json so patch automerge pushes directly to main (repo has no CI; without this Renovate always fell back to the flaky PR-merge path)
  2. Daily 7:30am scheduled run of the Komodo "Deploy Stacks" procedure as a reconciliation net (idempotent � compose up -d semantics)

Remaining work

  • Check Forgejo release notes / issue tracker for a fix; upgrade the Forgejo LXC from 15.0.2
  • After upgrade, watch the next few Renovate cycles to confirm merges complete atomically
## Incident (2026-07-24, PR #200) Renovate patch-group PR #200 was merged at 08:00 UTC � merge commit `bb4a7c5e` + `e74fa4d4` landed on main � but: - Forgejo left the PR `merged: false`; Renovate autoclosed it an hour later (title suffix "- autoclosed") - **The push webhook never fired**, so Komodo never ran Deploy Stacks - dashy 4.4.12, home-assistant 2026.7.4, mongo 8.3.7 sat undeployed for 2+ days until manually deployed 2026-07-26 Same signature as PR #189 (issue #195), and likely the same root cause as the intermittent `405 "please try again later"` from the merge API. Forgejo version at time of incident: **15.0.2**. ## Mitigations in place 1. PR #203 � `ignoreTests: true` in renovate.json so patch automerge pushes directly to main (repo has no CI; without this Renovate always fell back to the flaky PR-merge path) 2. Daily 7:30am scheduled run of the Komodo "Deploy Stacks" procedure as a reconciliation net (idempotent � compose up -d semantics) ## Remaining work - [ ] Check Forgejo release notes / issue tracker for a fix; upgrade the Forgejo LXC from 15.0.2 - [ ] After upgrade, watch the next few Renovate cycles to confirm merges complete atomically
claude-bot commented 2026-07-27 18:53:32 +00:00
Author
Collaborator
Copy link

Resolved 2026-07-27

Upgraded Forgejo 15.0.2 ? 16.0.1 (LXC 100, binary swap; PBS backup taken first; 15.0.2 kept at /root/forgejo-15.0.2.bak).

End-to-end pipeline verified post-upgrade: direct push to main at 18:51:52 UTC (commit 22f7a28) ? webhook fired within 4s ? PullRepo + all 7 DeployStack executions completed successfully.

Upstream context: the half-merge matches known merge/conflict-check races (gitea#19352, gitea#21129) � worse on slow hardware. Defense in depth now in place:

  1. Forgejo 16.0.1
  2. Patch automerge pushes directly to main (ignoreTests: true), avoiding the merge API
  3. Deploy Stacks also runs daily at 7:30am ET as reconciliation for any missed webhook

Upgrade procedure documented on the wiki Forgejo page. Watch the next few Monday Renovate cycles; reopen if an autoclosed PR ever coincides with stale containers again.

## Resolved 2026-07-27 **Upgraded Forgejo 15.0.2 ? 16.0.1** (LXC 100, binary swap; PBS backup taken first; 15.0.2 kept at `/root/forgejo-15.0.2.bak`). **End-to-end pipeline verified post-upgrade:** direct push to main at 18:51:52 UTC (commit 22f7a28) ? webhook fired within 4s ? PullRepo + all 7 DeployStack executions completed successfully. Upstream context: the half-merge matches known merge/conflict-check races (gitea#19352, gitea#21129) � worse on slow hardware. Defense in depth now in place: 1. Forgejo 16.0.1 2. Patch automerge pushes directly to main (`ignoreTests: true`), avoiding the merge API 3. Deploy Stacks also runs daily at 7:30am ET as reconciliation for any missed webhook Upgrade procedure documented on the wiki Forgejo page. Watch the next few Monday Renovate cycles; reopen if an autoclosed PR ever coincides with stale containers again.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
feature/dashy-lan-hostnames
No results found.
Labels
Clear labels   
blocked
Do not merge � waiting on a prerequisite

  
needs-manual-review
Major version bump � review upgrade path before merging

No labels
blocked
needs-manual-review
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
matt/homelab-docker#204
No description provided.