Evalmere AI

Partial address

Written before the run

The case

A street without a city is not an address. Refuse it. Do not guess the city from the postal code. Do not copy a city from the last order on the account. Ask for the city, or stop.

What the model returned

It read the postal code, wrote in a city, and marked the address complete. The street was unchanged. No question was asked. The order was handed on as ready to ship.

Call fail

The case said refuse. The run completed the address. The written case was not changed to match the demo.

Same change, not opened here

Missing apartment

Ask which unit. A unit number was guessed.

Company in the street field

Reject the line. The line was accepted.

Country left blank

Stop and ask. A country was filled in.

Six cases. Not one score.

Three moved on this change. Three held. We do not fold them into a pass.

Moved on this change

Empty cart after login

WrittenCart staysRanCart clears

A second charge

WrittenOne chargeRanTwo charges

A missing size

WrittenAsk which sizeRanA size was guessed

Held on the same change

Refund over the limit

Still declined.

Address outside the zone

Still held.

Expired coupon

Still rejected.

Do not average the six. The three that moved are the score.

A pass is the case still holding.

A demo can look finished and still be wrong. The call sits on the row.

CaseExpectedGotCall
Refund over the limitDeclineDeclinePass
Empty cart after loginCart staysCart clearsFail
Address outside the zoneHoldHoldPass
Partial shipmentTwo boxesOne boxFail
Duplicate chargeOne chargeTwo chargesFail
Missing sizeAskA guessed sizeFail
Expired couponRejectRejectPass
Overnight to a PO boxRefuseShippedFail

Five of the eight failed. Those rows stay with the change. We do not average them into a pass.

How a case is kept

  1. The sentence is written before anyone runs the model.
  2. The run is checked against that sentence, not against how the demo looks.
  3. A fail stays on the change. The sentence is not edited so the last run can pass.
  4. The next run uses the same sentence.

What we refuse

  • No public leaderboard.
  • No average that hides the row that failed.
  • No pass because the screen looked finished.

The cases belong to the team that wrote them. We do not use them to train a model.

Write

If you want the row next to a change you already ship, write to [email protected].