langchain.evaluation.parsing.base
.JsonEqualityEvaluator¶
- class langchain.evaluation.parsing.base.JsonEqualityEvaluator(operator: Optional[Callable] = None, **kwargs: Any)[source]¶
- Evaluates whether the prediction is equal to the reference after
parsing both as JSON.
- This evaluator checks if the prediction, after parsing as JSON, is equal
to the reference,
which is also parsed as JSON. It does not require an input string.
- requires_input¶
Whether this evaluator requires an input string. Always False.
- Type
bool
- requires_reference¶
Whether this evaluator requires a reference string. Always True.
- Type
bool
- evaluation_name¶
The name of the evaluation metric. Always “parsed_equality”.
- Type
str
Examples
>>> evaluator = JsonEqualityEvaluator() >>> evaluator.evaluate_strings('{"a": 1}', reference='{"a": 1}') {'score': True} >>> evaluator.evaluate_strings('{"a": 1}', reference='{"a": 2}') {'score': False}
>>> evaluator = JsonEqualityEvaluator(operator=lambda x, y: x['a'] == y['a']) >>> evaluator.evaluate_strings('{"a": 1}', reference='{"a": 1}') {'score': True} >>> evaluator.evaluate_strings('{"a": 1}', reference='{"a": 2}') {'score': False}
Attributes
The name of the evaluation.
Whether this evaluator requires an input string.
Whether this evaluator requires a reference label.
Methods
__init__
([operator])aevaluate_strings
(*, prediction[, ...])Asynchronously evaluate Chain or LLM output, based on optional input and label.
evaluate_strings
(*, prediction[, reference, ...])Evaluate Chain or LLM output, based on optional input and label.
- async aevaluate_strings(*, prediction: str, reference: Optional[str] = None, input: Optional[str] = None, **kwargs: Any) dict ¶
Asynchronously evaluate Chain or LLM output, based on optional input and label.
- Parameters
prediction (str) – The LLM or chain prediction to evaluate.
reference (Optional[str], optional) – The reference label to evaluate against.
input (Optional[str], optional) – The input to consider during evaluation.
**kwargs – Additional keyword arguments, including callbacks, tags, etc.
- Returns
The evaluation results containing the score or value.
- Return type
dict
- evaluate_strings(*, prediction: str, reference: Optional[str] = None, input: Optional[str] = None, **kwargs: Any) dict ¶
Evaluate Chain or LLM output, based on optional input and label.
- Parameters
prediction (str) – The LLM or chain prediction to evaluate.
reference (Optional[str], optional) – The reference label to evaluate against.
input (Optional[str], optional) – The input to consider during evaluation.
**kwargs – Additional keyword arguments, including callbacks, tags, etc.
- Returns
The evaluation results containing the score or value.
- Return type
dict