Optimize variations with objective feedback

Feedback Driven variation selection turns objective ratings into evidence for future traffic. The selected variation is snapshotted when each objective is created. Feedback on that objective updates the variation’s sampling parameters for objectives created later.

What feedback changes

Each variation starts at Beta(1, 1), a uniform prior with a posterior mean of 0.5.

Score	Posterior update	Direction
Greater than `0`	Add the score to alpha	More positive evidence
Less than `0`	Add the absolute score to beta	More negative evidence
Exactly `0`	Add `0.5` to alpha and `0.5` to beta	Neutral evidence that pulls an extreme posterior toward `0.5`

On every unpinned objective, Cadenya draws one sample from each variation’s Beta posterior and selects the highest. Strong candidates run more often, while uncertain candidates retain a chance to be explored.

The dashboard label is Feedback Driven. The API enum remains VARIATION_SELECTION_MODE_WEIGHTED for compatibility.

Agent Details card with Feedback Driven variation selection and two variations — Feedback Driven selection is configured on the agent, not on either variation

Create controlled evaluation runs

Open Objectives, click New Objective, select the agent, and optionally choose a specific Variation. The variation field overrides automatic selection and is useful when you need comparable samples from every candidate. Create one run pinned to Default and another pinned to Detailed, using the same message:

Create two attendee records with a name, email, company, city, and country.

Open each completed objective and confirm the Variation value in its Details card. That value comes from the objective’s immutable configuration snapshot.

Submit feedback in the dashboard

Open an objective and select its Feedback tab. Under Submit Feedback, choose a rating and optionally explain the decision.

Stars	API score
1, `Very poor`	`-1.0`
2, `Poor`	`-0.5`
3, `Neutral`	`0.0`
4, `Good`	`0.5`
5, `Excellent`	`1.0`

Click Submit Feedback. The entry appears under Previous Feedback and is also visible on the agent’s Feedback tab.

Objective Feedback tab showing one Excellent rating and its comment under Previous Feedback — Submitted ratings remain attached to the objective

Submit feedback from code

const feedback = await client.objectives.feedback.create(objectiveId, {
  workspaceId,
  metadata: {
    labels: { evaluator: 'docs-example' },
  },
  data: {
    score: 1,
    comment: 'Accurate, concise, and included a useful example.',
  },
});

console.log(feedback.metadata.id);

curl "https://api.cadenya.com/v1/workspaces/${CADENYA_WORKSPACE_ID}/objectives/${OBJECTIVE_ID}/feedback" \
  -H "Authorization: Bearer ${CADENYA_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
        "metadata": {
          "labels": { "evaluator": "docs-example" }
        },
        "data": {
          "score": 1,
          "comment": "Accurate, concise, and included a useful example."
        }
      }'

Feedback appends. Multiple reviewers can submit separate records for the same objective, and every record contributes evidence to the variation that ran it.

Inspect the updated variation

Retrieve a variation with info to see its current posterior mean and feedback count:

const variation = await client.agents.variations.retrieve(
  agentId,
  variationId,
  { workspaceId },
);

console.log({
  score: variation.info?.score,
  feedbackCount: variation.info?.feedbackCount,
});

info.score is the posterior mean on a 0 to 1 scale. It is not the average of the original -1 to 1 feedback values.

Verify a future sampled selection

Create another objective without variationId and inspect the snapshot:

const sampled = await client.objectives.create({
  workspaceId,
  agentId,
  systemPromptData: {},
  firstUserMessage:
    'Create one attendee record with a name, email, company, city, and country.',
});

console.log(sampled.configSnapshot.agentVariation?.metadata.name);

One run does not prove a probability distribution. Use the agent’s Analytics tab or a representative batch of objectives to compare selection counts and outcomes over time.

Do not use production feedback to force a deterministic route. Thompson Sampling always preserves exploration. Pass variationId when the caller must choose exactly one variation.

Getting Started

Build with the SDK

Tool Sets

Agents

Objectives

Callbacks

Memory

Secrets

Concepts

Use Cases

Optimize variations with objective feedback

What feedback changes

Create controlled evaluation runs

Submit feedback in the dashboard

Submit feedback from code

Inspect the updated variation

Verify a future sampled selection

​What feedback changes

​Create controlled evaluation runs

​Submit feedback in the dashboard

​Submit feedback from code

​Inspect the updated variation

​Verify a future sampled selection

What feedback changes

Create controlled evaluation runs

Submit feedback in the dashboard

Submit feedback from code

Inspect the updated variation

Verify a future sampled selection