Everyone agrees they should evaluate their prompts and nobody has enough test cases, because writing them is dull. Being able to describe a scenario and have the case land in the project removes the exact friction that keeps eval suites thin. Watch the loop though: an agent generating the cases that judge its own output is a closed circle worth breaking with a few cases written by a human who knows what failure looks like.
Scorecard's endpoint over their LLM evaluation platform — creating evaluation projects, configuring what is measured, and running it.
- Evaluation projects, and test cases organised by scenario
- Metric configuration such as accuracy and helpfulness
- Running a model evaluation and reading the results
OAuth. Their endpoint advertises protected-resource metadata, so a compliant client discovers the authorisation server itself. A local install is offered as an alternative.
Paste a URL, then authorize — add the endpoint to your client, then approve the OAuth consent screen