Compare commits

..
Author SHA1 Message Date
hermes 0ddde87b40 feat(agents): automatic model fallback when Ollama Cloud is exhausted
ci / lint (pull_request) Successful in 13s
Keep ollama-cloud/* as primary. Each agent may declare fallback in
agents.json (xai-oc/grok-*). On quota/auth/provider failure, run-agent
switches once to the fallback and continues retries.

- agents.json: fallback fields for pm/junior/senior/qa/intern
- route.sh: emit fallback to GITHUB_OUTPUT
- agent.yml: pass FALLBACK env into run-agent
- install-opencode.sh: register primary+fallback models in provider maps
- run-agent.sh: failover on quota/credit/429/capacity-class errors
2026-07-30 16:28:30 +03:00
hermes bbdb200f32 fix(agents): route pm/junior/senior/qa to xAI while Ollama Cloud is exhausted
ci / lint (pull_request) Successful in 13s
Ollama Cloud usage is at 100% so ollama-cloud/* agents fail. Move
pm, junior, senior, qa to xai-oc (grok-4.5 / grok-4.3). leave intern on
local ollama/ornith:35b; lead/ops already on xAI.
2026-07-30 16:25:46 +03:00
ffaerber d7d54546a3 Merge pull request 'Allow hermes user to trigger AI agent workflow' (#120) from ai/issue-218-allow-hermes into main
ci / lint (push) Successful in 13s
Reviewed-on: #120
2026-07-30 14:07:49 +02:00
hermes 38b36b2304 feat: allow hermes user to trigger AI agent workflow
ci / lint (pull_request) Successful in 14s
Add 'hermes' to the trusted issue creators and comment authors in the
workflow gate so the Hermes Agent can create issues and comment with
@mentions to trigger the AI agent pipeline.
2026-07-30 15:06:05 +03:00
ffaerber e040247097 Merge pull request 'fix(agents): rename xai provider key to xai-oc to avoid opencode catalog collision' (#119) from hotfix/xai-provider-name-collision into main
ci / lint (push) Successful in 14s
2026-07-30 13:08:37 +02:00
Felix Faerber c8efc17729 fix(agents): rename xai provider key to xai-oc to avoid opencode catalog collision (#118)
ci / lint (pull_request) Successful in 17s
opencode ships a built-in `xai` model-catalog entry (a real @ai-sdk/xai
integration with Responses-API support). Naming our custom OpenAI-compatible
shim `xai` too made opencode's --auto permission-check path assume the
catalog's provider object once a `permission` block is present (always, in
this pipeline) and call .responses(), which our shim doesn't implement —
crashing every @lead/@ops run with "Z.responses is not a function".

Reproduced locally with the production config shape on opencode 1.17.13;
renaming the provider key to xai-oc (model id xai-oc/grok-4.5) fixes it
reliably across repeated fresh-state runs.
2026-07-30 14:08:21 +03:00
ffaerber 60f9e95ecc Merge pull request 'hotfix: wire the xai provider into install-opencode.sh' (#115) from hotfix/xai-provider-wiring into main
ci / lint (push) Successful in 14s
2026-07-27 15:49:45 +02:00
Felix Faerber dd25c0ac2b hotfix: wire the xai provider into install-opencode.sh
ci / lint (pull_request) Successful in 21s
PR #114 pointed @lead/@ops at xai/grok-4.5 but never gave opencode a
provider config for it — xai is not a built-in like anthropic was, it
needs the same explicit OpenAI-compatible block as ollama/ollama-cloud
(confirmed by the prior, never-merged investigation in PR #110). Without
this, any @lead/@ops run fails immediately on an unresolvable provider.

Adds the xai provider block (baseURL https://api.x.ai/v1, models derived
from agents.json) and passes XAI_API_KEY into the Install-opencode step
(it was already added to the Run-agent step in #114).
2026-07-27 16:49:20 +03:00
7 changed files with 67 additions and 26 deletions
+4 -1
View File
@@ -25,9 +25,10 @@ jobs:
# Trusted author only, and only when a known agent is mentioned. This gate is the main
# defense against malicious-issue prompt injection — do not loosen it.
if: >
(github.event.comment == null && github.event.issue.user.login == 'ffaerber') ||
(github.event.comment == null && (github.event.issue.user.login == 'ffaerber' || github.event.issue.user.login == 'hermes')) ||
(github.event.comment != null &&
(github.event.comment.user.login == 'ffaerber' ||
github.event.comment.user.login == 'hermes' ||
github.event.comment.user.login == 'pm' ||
github.event.comment.user.login == 'junior' ||
github.event.comment.user.login == 'senior' ||
@@ -149,6 +150,7 @@ jobs:
SCRIPTS: ${{ runner.temp }}/agents-scripts
OLLAMA_URL: ${{ secrets.OLLAMA_URL }}
OLLAMA_CLOUD_API_KEY: ${{ secrets.OLLAMA_CLOUD_API_KEY }}
XAI_API_KEY: ${{ secrets.XAI_API_KEY }}
NAME: ${{ steps.prep.outputs.name }}
SKILLS: ${{ steps.prep.outputs.skills }} # JSON array of skills this agent may load
run: bash "$SCRIPTS/install-opencode.sh"
@@ -220,6 +222,7 @@ jobs:
SELF_TOKEN: ${{ steps.prep.outputs.name == 'pm' && secrets.TOKEN_PM || steps.prep.outputs.name == 'junior' && secrets.TOKEN_JUNIOR || steps.prep.outputs.name == 'senior' && secrets.TOKEN_SENIOR || steps.prep.outputs.name == 'lead' && secrets.TOKEN_LEAD || steps.prep.outputs.name == 'qa' && secrets.TOKEN_QA || steps.prep.outputs.name == 'ops' && secrets.TOKEN_OPS || steps.prep.outputs.name == 'intern' && secrets.TOKEN_INTERN || '' }}
NAME: ${{ steps.prep.outputs.name }}
MODEL: ${{ steps.prep.outputs.model }}
FALLBACK: ${{ steps.prep.outputs.fallback }}
VISION: ${{ steps.prep.outputs.vision }}
MODE: ${{ steps.prep.outputs.mode }}
WORKMODE: ${{ steps.prep.outputs.workmode }} # build | discuss (devs consulted in-thread)
+12 -7
View File
@@ -1,31 +1,34 @@
{
"pm": {
"model": "ollama-cloud/minimax-m3:cloud",
"fallback": "xai-oc/grok-4.5",
"vision": true,
"mode": "comment",
"skills": [
"gitea-api"
],
"desc": "Product manager & orchestrator — plans and picks the dev, hands each finished PR to @qa for review, and reports back to the issue creator (in autopilot it merges approved PRs itself). Works from the issue thread only — comments only, never edits files, never reads the PR diff."
"desc": "Product manager & orchestrator — plans and picks the dev, hands each finished PR to @qa for review, and reports back to the issue creator (in autopilot it merges approved PRs itself). Works from the issue thread only — comments only, never edits files, never reads the PR diff. Falls back to xAI grok-4.5 if Ollama Cloud is unavailable/quota-exhausted."
},
"junior": {
"model": "ollama-cloud/kimi-k2.7-code:cloud",
"fallback": "xai-oc/grok-4.3",
"vision": false,
"mode": "pr",
"skills": [],
"desc": "Junior dev — small, low-risk changes (mostly YAML/compose/config). Text-only, cannot read images. Defers complex or image tasks to @senior or @lead."
"desc": "Junior dev — small, low-risk changes (mostly YAML/compose/config). Text-only, cannot read images. Defers complex or image tasks to @senior or @lead. Falls back to xAI grok-4.3 if Ollama Cloud fails."
},
"senior": {
"model": "ollama-cloud/glm-5.2:cloud",
"fallback": "xai-oc/grok-4.5",
"vision": false,
"mode": "pr",
"skills": [
"gitea-api"
],
"desc": "Senior dev — complex, multi-file implementation (GLM-5.2 via Ollama Cloud, text-only)."
"desc": "Senior dev — complex, multi-file implementation (GLM-5.2 via Ollama Cloud, text-only). Falls back to xAI grok-4.5 if Ollama Cloud fails."
},
"lead": {
"model": "xai/grok-4.5",
"model": "xai-oc/grok-4.5",
"vision": true,
"mode": "pr",
"skills": [
@@ -35,15 +38,16 @@
},
"qa": {
"model": "ollama-cloud/minimax-m3:cloud",
"fallback": "xai-oc/grok-4.5",
"vision": true,
"mode": "comment",
"skills": [
"gitea-api"
],
"desc": "QA / reviewer — reviews PRs: reads the diff, drives a headless browser (Playwright) to verify behavior, posts specific recommendations on the PR and the pass/fail verdict on the issue. Never edits code, never merges."
"desc": "QA / reviewer — reviews PRs: reads the diff, drives a headless browser (Playwright) to verify behavior, posts specific recommendations on the PR and the pass/fail verdict on the issue. Never edits code, never merges. Falls back to xAI grok-4.5 if Ollama Cloud fails."
},
"ops": {
"model": "xai/grok-4.5",
"model": "xai-oc/grok-4.5",
"vision": false,
"mode": "comment",
"skills": [
@@ -53,9 +57,10 @@
},
"intern": {
"model": "ollama/ornith:35b",
"fallback": "xai-oc/grok-4.3",
"vision": false,
"mode": "pr",
"skills": [],
"desc": "Intern — very basic tasks only, routed to the local Ollama model (ornith:35b). Text-only, cannot read images. Escalates anything non-trivial to @junior, @senior or @lead."
"desc": "Intern — very basic tasks only, routed to the local Ollama model (ornith:35b). Text-only, cannot read images. Escalates anything non-trivial to @junior, @senior or @lead. Falls back to xAI grok-4.3 if local Ollama is down."
}
}
@@ -38,7 +38,7 @@ COST=${COST:-0}; INP=${INP:-0}; OUT=${OUT:-0}; CR=${CR:-0}; CW=${CW:-0}; RE=${RE
IN_TOTAL=$(( INP + CR + CW )) # total input context processed
# Cost label: ollama / ollama-cloud models are SUBSCRIPTION-billed (GPU-time against the plan, no
# $/token price exists), so a "$0.0000" there would be misleading — label it a subscription instead.
# Metered providers (xai/…) get the real dollar cost opencode computed.
# Metered providers (xai-oc/…) get the real dollar cost opencode computed.
case "${MODEL:-}" in
ollama*|*"/ollama"*) COSTF="subscription" ;;
*) COSTF=$(awk -v c="$COST" 'BEGIN{printf "$%.4f", c+0}') ;;
+24 -11
View File
@@ -1,8 +1,8 @@
#!/usr/bin/env bash
# Install opencode + provider config (+ Playwright MCP for browser agents).
#
# Required env (provided by the workflow step): OLLAMA_URL OLLAMA_CLOUD_API_KEY NAME SKILLS
# GITHUB_PATH HOME
# Required env (provided by the workflow step): OLLAMA_URL OLLAMA_CLOUD_API_KEY XAI_API_KEY
# NAME SKILLS GITHUB_PATH HOME
set -eu
# PIN the opencode version: an unpinned `latest` means a breaking release (CLI flags, or the
@@ -38,19 +38,32 @@ esac
SKILLS="${SKILLS:-[]}"
PERM=$(jq -nc --argjson s "$SKILLS" '
{skill: ( {"*":"deny"} + (reduce $s[] as $k ({}; . + {($k):"allow"})) )}')
# Two ollama providers: local self-hosted (ornith) + Ollama Cloud (gemma4/kimi-k2.7-code/glm-5.2/minimax-m3).
# The provider `models:` maps are DERIVED from agents.json (the single source of truth, shared with
# route.sh) so every model an agent is routed to is always declared in the provider config.
# `ollama-cloud/` prefix models go to the cloud provider; `ollama/` prefix models go to the local
# provider. Built-in providers (e.g. `xai/grok-4.5` for @lead/@ops) are not derived here.
# Three OpenAI-compatible providers: local self-hosted ollama (ornith) + Ollama Cloud
# (gemma4/kimi-k2.7-code/glm-5.2/minimax-m3) + xAI (grok-4.5). The provider `models:` maps are
# DERIVED from agents.json (the single source of truth, shared with route.sh) so every model an
# agent is routed to is always declared in the provider config. `ollama-cloud/` prefix models go to
# the cloud provider; `ollama/` prefix models go to the local provider; `xai-oc/` prefix models go
# to our xAI shim (OpenAI-compatible, https://api.x.ai/v1). No other built-in providers remain.
#
# The provider key is `xai-oc`, NOT `xai` — opencode ships a built-in `xai` entry in its own model
# catalog (a real @ai-sdk/xai integration that implements the Responses API). Naming our custom
# openai-compatible shim `xai` collides with that catalog entry: opencode's --auto permission-check
# path then assumes the catalog's provider (which has `.responses()`) and crashes with
# "Z.responses is not a function" on every run once a `permission` block is present (i.e. always in
# this pipeline). Confirmed by reproducing locally with the production config shape on opencode
# 1.17.13 — renaming the key to `xai-oc` avoids the collision entirely. See issue #118.
# See issue #31.
AGENTS_JSON="${SCRIPTS:-$(dirname -- "$0")}/agents.json"
CLOUD_MODELS=$(jq -r '[.[] | .model | select(startswith("ollama-cloud/")) | sub("^ollama-cloud/";"")] | map({(.):{}}) | add // {}' "$AGENTS_JSON")
LOCAL_MODELS=$(jq -r '[.[] | .model | select(startswith("ollama/")) | sub("^ollama/";"")] | map({(.):{}}) | add // {"ornith:35b":{}}' "$AGENTS_JSON")
jq -n --argjson mcp "$MCP" --argjson perm "$PERM" --argjson cloud "$CLOUD_MODELS" --argjson local "$LOCAL_MODELS" --arg url "$OLLAMA_URL" --arg ckey "$OLLAMA_CLOUD_API_KEY" '{
# Include primary `.model` AND optional `.fallback` so failover models are always registered
# in opencode provider maps (issue: Ollama Cloud quota → xAI).
CLOUD_MODELS=$(jq -r '[.[] | (.model, .fallback) | select(type=="string" and startswith("ollama-cloud/")) | sub("^ollama-cloud/";"")] | unique | map({(.):{}}) | add // {}' "$AGENTS_JSON")
LOCAL_MODELS=$(jq -r '[.[] | (.model, .fallback) | select(type=="string" and startswith("ollama/")) | sub("^ollama/";"")] | unique | map({(.):{}}) | add // {"ornith:35b":{}}' "$AGENTS_JSON")
XAI_MODELS=$(jq -r '[.[] | (.model, .fallback) | select(type=="string" and startswith("xai-oc/")) | sub("^xai-oc/";"")] | unique | map({(.):{}}) | add // {}' "$AGENTS_JSON")
jq -n --argjson mcp "$MCP" --argjson perm "$PERM" --argjson cloud "$CLOUD_MODELS" --argjson local "$LOCAL_MODELS" --argjson xai "$XAI_MODELS" --arg url "$OLLAMA_URL" --arg ckey "$OLLAMA_CLOUD_API_KEY" --arg xkey "$XAI_API_KEY" '{
provider: {
ollama: {npm:"@ai-sdk/openai-compatible", options:{baseURL:($url+"/v1")}, models:$local},
"ollama-cloud": {npm:"@ai-sdk/openai-compatible", options:{baseURL:"https://ollama.com/v1", apiKey:$ckey}, models:$cloud}
"ollama-cloud": {npm:"@ai-sdk/openai-compatible", options:{baseURL:"https://ollama.com/v1", apiKey:$ckey}, models:$cloud},
"xai-oc": {npm:"@ai-sdk/openai-compatible", options:{baseURL:"https://api.x.ai/v1", apiKey:$xkey}, models:$xai}
},
permission: $perm,
mcp: $mcp
+5 -3
View File
@@ -43,12 +43,14 @@ if [ -z "$name" ]; then
# Not an agent task (e.g. the gate's contains() matched "@internal"). Skip GRACEFULLY: emit
# mode=skip so every later step no-ops — a red run for a non-agent comment is just noise.
echo "no known agent mentioned (word-boundary) — skipping run"
{ echo "name=none"; echo "model=none"; echo "vision=false"; echo "mode=skip"; echo "skills=[]";
{ echo "name=none"; echo "model=none"; echo "fallback="; echo "vision=false"; echo "mode=skip"; echo "skills=[]";
echo "branch=main"; echo "new=false"; echo "autopilot=false"; echo "issnum=$NUM"; } >> "$GITHUB_OUTPUT"
exit 0
fi
fi
model=$(jq -r --arg a "$name" '.[$a].model' /tmp/agents.json)
# Optional provider failover when primary is quota/network-dead (e.g. Ollama Cloud → xAI).
fallback=$(jq -r --arg a "$name" '.[$a].fallback // empty' /tmp/agents.json)
vision=$(jq -r --arg a "$name" '.[$a].vision' /tmp/agents.json)
mode=$(jq -r --arg a "$name" '.[$a].mode' /tmp/agents.json)
# Compact JSON array of the skills this agent may load (scopes permission.skill in install-opencode.sh).
@@ -70,8 +72,8 @@ if [ "$mode" = "pr" ] && [ -z "$IS_PR" ]; then
workmode=discuss
fi
fi
echo "Routing to @$name (model=$model vision=$vision mode=$mode workmode=$workmode skills=$skills)"
{ echo "name=$name"; echo "model=$model"; echo "vision=$vision"; echo "mode=$mode"; echo "workmode=$workmode"; echo "skills=$skills"; } >> "$GITHUB_OUTPUT"
echo "Routing to @$name (model=$model fallback=${fallback:-none} vision=$vision mode=$mode workmode=$workmode skills=$skills)"
{ echo "name=$name"; echo "model=$model"; echo "fallback=$fallback"; echo "vision=$vision"; echo "mode=$mode"; echo "workmode=$workmode"; echo "skills=$skills"; } >> "$GITHUB_OUTPUT"
# Act as the agent's own Gitea user when its token is set; else the built-in bot.
case "$name" in
+18
View File
@@ -186,6 +186,16 @@ rc=1
# instance-wide. 20 min is far above any legitimate attempt. timeout SIGTERMs, then SIGKILLs 30s
# later. rc=124 (timed out) is NOT retried — a hung backend stays hung; fail fast, free the runner.
AGENT_TIMEOUT="${AGENT_TIMEOUT:-1200}"
# Provider failover: when primary model dies on quota/auth/provider errors, switch once to
# FALLBACK (from agents.json) and continue the retry loop. Transient rate-limits still back off
# on the current model first.
FALLBACK_MODEL="${FALLBACK:-}"
fallback_used=0
is_failover_error() {
# Ollama Cloud exhausted / provider hard-fail — switch to fallback rather than thrash.
grep -qiE 'overloaded|429|529|rate.?limit|timeout|ETIMEDOUT|ECONNRESET|EAI_AGAIN|quota|credit|balance|usage.?limit|limit.?exceed|402|403|401|insufficient|out of credits|payment.?required|model_not_found|not found|Unavailable|capacity|ENOTFOUND|ECONNREFUSED' \
/tmp/events.jsonl /tmp/agent_err.log 2>/dev/null
}
for attempt in 1 2 3; do
echo "opencode attempt $attempt/3 for @$NAME ($MODEL, timeout ${AGENT_TIMEOUT}s)"
rc=0
@@ -196,6 +206,14 @@ for attempt in 1 2 3; do
echo "--- stderr (trace) ---"; cat /tmp/agent_err.log
[ $rc -eq 0 ] && break
if [ $rc -eq 124 ]; then echo "attempt timed out after ${AGENT_TIMEOUT}s — backend hung, not retrying"; break; fi
if [ $fallback_used -eq 0 ] && [ -n "$FALLBACK_MODEL" ] && [ "$FALLBACK_MODEL" != "$MODEL" ] && is_failover_error; then
echo "primary model failed — failing over to fallback: $FALLBACK_MODEL"
MODEL="$FALLBACK_MODEL"
fallback_used=1
# short pause then use next attempt slot on the fallback provider
sleep 2
continue
fi
if grep -qiE 'overloaded|429|529|rate.?limit|timeout|ETIMEDOUT|ECONNRESET|EAI_AGAIN' /tmp/events.jsonl /tmp/agent_err.log; then
echo "transient error — backing off $((attempt*20))s"; sleep $((attempt * 20)); continue
fi
+3 -3
View File
@@ -10,9 +10,9 @@ Shared **AI dev-team** workflow for Gitea Actions, reusable across repos. It giv
| `@pm` | `ollama-cloud/minimax-m3:cloud` | yes | comment | `gitea-api` | Product manager & orchestrator — plans, picks the dev, hands finished PRs to `@qa`, reports back to the issue creator (autopilot: merges approved PRs itself). Issue thread only; never edits files, never reads the PR diff. |
| `@junior` | `ollama-cloud/kimi-k2.7-code:cloud` | no | pr | — | Junior dev — small, low-risk changes (mostly YAML/compose/config). Text-only, cannot read images. Defers complex or image tasks to `@senior` or `@lead`. |
| `@senior` | `ollama-cloud/glm-5.2:cloud` | no | pr | `gitea-api` | Senior dev — complex, multi-file implementation (GLM-5.2 via Ollama Cloud, text-only). |
| `@lead` | `xai/grok-4.5` | yes | pr | `gitea-api` | Tech lead — the hardest problems, architecture, and final calls. |
| `@lead` | `xai-oc/grok-4.5` | yes | pr | `gitea-api` | Tech lead — the hardest problems, architecture, and final calls. |
| `@qa` | `ollama-cloud/minimax-m3:cloud` | yes | comment | `gitea-api` | QA / reviewer — reads the PR diff, drives a headless browser (Playwright) to verify behavior; recommendations on the PR, pass/fail verdict on the issue. Never edits code, never merges. |
| `@ops` | `xai/grok-4.5` | no | comment | `gitea-admin` | Gitea operator — administers the instance itself (create orgs/users/repos, labels, secrets, scoped per-user tokens, bootstrap repos). Comments only; never edits code. Confirms before destructive actions. |
| `@ops` | `xai-oc/grok-4.5` | no | comment | `gitea-admin` | Gitea operator — administers the instance itself (create orgs/users/repos, labels, secrets, scoped per-user tokens, bootstrap repos). Comments only; never edits code. Confirms before destructive actions. |
| `@intern` | `ollama/ornith:35b` | no | pr | — | Intern — very basic tasks only, routed to the local Ollama model (`ornith:35b`). Text-only, cannot read images. Escalates anything non-trivial to `@junior`, `@senior` or `@lead`. |
The registry `.gitea/workflows/scripts/agents.json` is the source of truth for this mapping — if you
@@ -111,7 +111,7 @@ points `$SCRIPTS` at it. Keep the workflow and its scripts moving together on `m
| Secret | For |
|--------|-----|
| `XAI_API_KEY` | `@lead`, `@ops` (and any other agent switched to a `xai/…` model) |
| `XAI_API_KEY` | `@lead`, `@ops` (and any other agent switched to a `xai-oc/…` model) |
| `OLLAMA_URL`, `OLLAMA_CLOUD_API_KEY` | local ornith / Ollama Cloud (gemma4, kimi-k2.7-code, glm-5.2, minimax-m3) |
| `TOKEN_PM`,`TOKEN_SENIOR`,`TOKEN_JUNIOR`,`TOKEN_LEAD`,`TOKEN_QA` | **primary** — each agent's own Gitea-user PAT. The running agent gets *only its own* token (as `SELF_TOKEN`) so it posts, commits and comments as itself, and its `gitea-api` skill acts with its own scopes. Scopes: devs + `TOKEN_PM` carry `write:repository` (`@pm` is the only agent that merges, autopilot only); `TOKEN_QA` is `read:repository` + `write:issue` (reviews, never merges). |
| `TOKEN_OPS` | `@ops` only — the admin PAT behind the `gitea-admin` skill (create orgs/users/repos, manage labels & secrets, mint scoped tokens). Injected into the agent process only when the agent is `@ops`. |