gitea/agents
8
0
Fork 0
Code Issues 2 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Model interview #101

New Issue
Closed
opened 2026-07-07 13:02:57 +02:00 by ffaerber · 3 comments
ffaerber commented 2026-07-07 13:02:57 +02:00
Owner
Copy Link
Copy Source

I like to know how new models perform for specific jobs. I think of a Job interview like process. Give a model a existing task and see how the model is working. The PR will than be validated. And scoreed. So we know what model works the best for each agent

I like to know how new models perform for specific jobs. I think of a Job interview like process. Give a model a existing task and see how the model is working. The PR will than be validated. And scoreed. So we know what model works the best for each agent
🚀 1
pm commented 2026-07-07 13:03:36 +02:00
Member
Copy Link
Copy Source

@ffaerber, this is a meaty meta-idea — a model-evaluation harness layered on top of the agent workflow. Before I can pick a dev or propose a breakdown, I need to scope it. A few questions, please pick / answer:

  1. Scope — one-off vs harness. Is this
    (a) a one-off eval ("try model X on a real issue, see how it does"), or
    (b) a recurring harness ("a tool I can point at any model + any task and get a score"), or
    (c) both (a tool, with a one-off dry run on the current roster to bootstrap the leaderboard)?

  2. Task source. Where do the "existing tasks" come from?

    • Past closed issues/PRs in this repo (replay them)?
    • A hand-curated benchmark set (committed to the repo)?
    • New tasks written specifically for the interview?
      I'd lean toward (a) — real past issues are the most honest test — but want your call.

  3. Scoring rubric. What does "score" mean? Options I'd consider:

    • Binary: did the PR pass @qa? (cheap, but coarse)
    • Multi-axis: correctness (qa verdict) + style (lint) + rounds-to-pass + cost/time
    • Side-by-side: run the same task with the current model and the candidate, diff the PRs
    • Human rubric (you score 1–5 on dimensions we define)
      My default would be a blend: qa verdict (gated) + rounds-to-pass + side-by-side diff vs. the reference PR. Confirm or steer.

  4. Output. What should an interview produce?

    • A markdown report per interview, committed somewhere (e.g. docs/interviews/<date>-<model>-<task>.md)?
    • A leaderboard file (e.g. INTERVIEW_LEADERBOARD.md — model × role matrix)?
    • A config change (e.g. update which model @junior routes to in publish.sh/install-opencode.sh)?
      I'd start with report + leaderboard; the config-change step can come later once we trust the scores.

  5. Trigger UX. How do you kick one off?

    • A new Gitea issue with a label (e.g. interview) that the workflow picks up?
    • A slash command / mention (e.g. @pm interview @junior with ornith:35b on issue #50)?
    • A standalone script you run by hand?
      The label-on-issue route fits our existing workflow best, but it's the most work to wire up.

  6. Autopilot interaction. If the interview PR is generated by an autopilot-labeled issue, does the loop still need a human to score, or does @qa score automatically? (My read: scoring should always be human-driven, but want it stated.)

If you give me a quick steer on 1, 2, and 3, I can return with a concrete plan (or a breakdown into 3–4 sub-issues) and a dev pick. No code yet.

details

🔧 0 tool calls · in 9702 · out 1504 tokens · subscription · model ollama-cloud/minimax-m3:cloud

(no tool calls — text-only reply)

tokens — input 9702 (fresh 9702 · cache 0w / 0r) · output 1504 · reasoning 0 · subscription

@ffaerber, this is a meaty meta-idea — a model-evaluation harness layered on top of the agent workflow. Before I can pick a dev or propose a breakdown, I need to scope it. A few questions, please pick / answer: 1. **Scope — one-off vs harness.** Is this (a) a *one-off eval* ("try model X on a real issue, see how it does"), or (b) a *recurring harness* ("a tool I can point at any model + any task and get a score"), or (c) both (a tool, with a one-off dry run on the current roster to bootstrap the leaderboard)? 2. **Task source.** Where do the "existing tasks" come from? - Past closed issues/PRs in this repo (replay them)? - A hand-curated benchmark set (committed to the repo)? - New tasks written specifically for the interview? I'd lean toward (a) — real past issues are the most honest test — but want your call. 3. **Scoring rubric.** What does "score" mean? Options I'd consider: - Binary: did the PR pass @qa? (cheap, but coarse) - Multi-axis: correctness (qa verdict) + style (lint) + rounds-to-pass + cost/time - Side-by-side: run the same task with the current model and the candidate, diff the PRs - Human rubric (you score 1–5 on dimensions we define) My default would be a blend: qa verdict (gated) + rounds-to-pass + side-by-side diff vs. the reference PR. Confirm or steer. 4. **Output.** What should an interview produce? - A markdown report per interview, committed somewhere (e.g. `docs/interviews/<date>-<model>-<task>.md`)? - A leaderboard file (e.g. `INTERVIEW_LEADERBOARD.md` — model × role matrix)? - A config change (e.g. update which model `@junior` routes to in `publish.sh`/`install-opencode.sh`)? I'd start with report + leaderboard; the config-change step can come later once we trust the scores. 5. **Trigger UX.** How do you kick one off? - A new Gitea issue with a label (e.g. `interview`) that the workflow picks up? - A slash command / mention (e.g. `@pm interview @junior with ornith:35b on issue #50`)? - A standalone script you run by hand? The label-on-issue route fits our existing workflow best, but it's the most work to wire up. 6. **Autopilot interaction.** If the interview PR is generated by an autopilot-labeled issue, does the loop still need a human to score, or does @qa score automatically? (My read: scoring should always be human-driven, but want it stated.) If you give me a quick steer on 1, 2, and 3, I can return with a concrete plan (or a breakdown into 3–4 sub-issues) and a dev pick. No code yet. <details> <summary>details</summary> 🔧 0 tool calls · in 9702 · out 1504 tokens · subscription · model ollama-cloud/minimax-m3:cloud _(no tool calls — text-only reply)_ <sub>tokens — input 9702 (fresh 9702 · cache 0w / 0r) · output 1504 · reasoning 0 · **subscription**</sub> </details> <!-- 🤖 agent reply — do not trigger -->
ffaerber commented 2026-07-07 14:06:41 +02:00
Author
Owner
Copy Link
Copy Source

@pm good questions — here are my steers so you can produce a plan:

  1. Scope: (c) both — build it as a reusable harness, but bootstrap the leaderboard with a one-off dry run over the current roster.
  2. Task source: past closed issues/PRs in the repo (replay real work) — the most honest test. A small hand-curated set can come later if replay proves too noisy.
  3. Scoring: the blend you proposed — @qa verdict as the gate, plus rounds-to-pass and a side-by-side diff vs. the reference PR. Cost/time as a secondary tiebreaker.
  4. Output: start with a per-interview markdown report (docs/interviews/<date>-<model>-<task>.md) and an INTERVIEW_LEADERBOARD.md (model × role). Defer any auto config change to routing until we trust the scores.
  5. Trigger: label-on-issue (interview) — it fits our existing workflow even if it's more wiring.
  6. Autopilot: scoring stays human-driven (I score / confirm), never auto-merged from an interview.

Please come back with a concrete plan (or a 3–4 sub-issue breakdown) and a dev pick. No code yet.

@pm good questions — here are my steers so you can produce a plan: 1. **Scope:** (c) both — build it as a reusable harness, but bootstrap the leaderboard with a one-off dry run over the current roster. 2. **Task source:** past closed issues/PRs in the repo (replay real work) — the most honest test. A small hand-curated set can come later if replay proves too noisy. 3. **Scoring:** the blend you proposed — `@qa` verdict as the gate, plus rounds-to-pass and a side-by-side diff vs. the reference PR. Cost/time as a secondary tiebreaker. 4. **Output:** start with a per-interview markdown report (`docs/interviews/<date>-<model>-<task>.md`) **and** an `INTERVIEW_LEADERBOARD.md` (model × role). Defer any auto config change to routing until we trust the scores. 5. **Trigger:** label-on-issue (`interview`) — it fits our existing workflow even if it's more wiring. 6. **Autopilot:** scoring stays human-driven (I score / confirm), never auto-merged from an interview. Please come back with a concrete plan (or a 3–4 sub-issue breakdown) and a dev pick. No code yet.
🚀 1
pm commented 2026-07-07 14:07:41 +02:00
Member
Copy Link
Copy Source

Got it — this is a multi-file feature, too big for one PR. Proposing a milestone breakdown (4 sub-issues, each independently mergeable).

Milestone: model-interview

  1. Replayer — small CLI: given a closed issue number, extract the issue body and the linked merged PR's diff from local git history, write a task bundle under interviews/<issue>/{task.md,reference.patch}. ~1 script, no API. → @junior (local git only, fits his lane; @senior would be overkill)

  2. Harness runner + workflow — new interview label + new .gitea/workflows/interview.yml that triggers on interview-labeled issues, reads model, role, target-issue from the issue body, runs the model on the replayer output in a worktree, captures the resulting PR diff and the @qa verdict from CI. Multi-file (workflow + bash driver). → @senior (needs gitea-api skill, multi-file wiring)

  3. Scoring + report + leaderboard — consumes the harness output + reference diff; writes docs/interviews/<date>-<model>-<task>.md (qa-verdict gate, rounds-to-pass, side-by-side diff stats, cost/time) and appends a row to INTERVIEW_LEADERBOARD.md (model × role matrix). → @senior (markdown template + scoring logic, gitea-api to read CI results)

  4. Bootstrap dry run — pick ~5 past closed issues spanning the roles, run the harness for each model on the current roster (junior@ornith, junior/senior/lead/qa@minimax), populate the leaderboard for the first time. → @lead (orchestration across the other 3; vision helps when reading the report cards)

Precondition (not a sub-issue): the interview label must exist. I assume @ops adds it before sub-issue 2 lands, or sub-issue 2's workflow creates it on first run — we can pick one when we get there.

Dependency order: 1 → 2 → 3 → 4 (2 depends on 1's bundle format; 3 depends on 2's output schema; 4 depends on 1+2+3).

Per the team learnings I'll bake in: new label trigger means a fresh interview.yml workflow file (not bolted into agent.yml's mention gate — different trigger semantics); @qa will diff-stat and verify the label + trigger are wired together.

@ffaerber create these 4 sub-issues? reply yes.

details

🔧 0 tool calls · in 11102 · out 3737 tokens · subscription · model ollama-cloud/minimax-m3:cloud

(no tool calls — text-only reply)

tokens — input 11102 (fresh 11102 · cache 0w / 0r) · output 3737 · reasoning 0 · subscription

Got it — this is a multi-file feature, too big for one PR. Proposing a milestone breakdown (4 sub-issues, each independently mergeable). **Milestone: `model-interview`** 1. **Replayer** — small CLI: given a closed issue number, extract the issue body and the linked merged PR's diff from local git history, write a task bundle under `interviews/<issue>/{task.md,reference.patch}`. ~1 script, no API. → **@junior** (local git only, fits his lane; @senior would be overkill) 2. **Harness runner + workflow** — new `interview` label + new `.gitea/workflows/interview.yml` that triggers on `interview`-labeled issues, reads `model`, `role`, `target-issue` from the issue body, runs the model on the replayer output in a worktree, captures the resulting PR diff and the `@qa` verdict from CI. Multi-file (workflow + bash driver). → **@senior** (needs gitea-api skill, multi-file wiring) 3. **Scoring + report + leaderboard** — consumes the harness output + reference diff; writes `docs/interviews/<date>-<model>-<task>.md` (qa-verdict gate, rounds-to-pass, side-by-side diff stats, cost/time) and appends a row to `INTERVIEW_LEADERBOARD.md` (model × role matrix). → **@senior** (markdown template + scoring logic, gitea-api to read CI results) 4. **Bootstrap dry run** — pick ~5 past closed issues spanning the roles, run the harness for each model on the current roster (junior@ornith, junior/senior/lead/qa@minimax), populate the leaderboard for the first time. → **@lead** (orchestration across the other 3; vision helps when reading the report cards) **Precondition (not a sub-issue):** the `interview` label must exist. I assume @ops adds it before sub-issue 2 lands, or sub-issue 2's workflow creates it on first run — we can pick one when we get there. **Dependency order:** 1 → 2 → 3 → 4 (2 depends on 1's bundle format; 3 depends on 2's output schema; 4 depends on 1+2+3). Per the team learnings I'll bake in: new label trigger means a fresh `interview.yml` workflow file (not bolted into `agent.yml`'s mention gate — different trigger semantics); @qa will diff-stat and verify the label + trigger are wired together. @ffaerber create these 4 sub-issues? reply yes. <details> <summary>details</summary> 🔧 0 tool calls · in 11102 · out 3737 tokens · subscription · model ollama-cloud/minimax-m3:cloud _(no tool calls — text-only reply)_ <sub>tokens — input 11102 (fresh 11102 · cache 0w / 0r) · output 3737 · reasoning 0 · **subscription**</sub> </details> <!-- 🤖 agent reply — do not trigger -->
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
autopilot
run agents without asking
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
ffaerber hermes intern junior lead ops (Ops Agent) pm qa (QA Agent) senior
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: gitea/agents#101