`langchain_community.callbacks.argilla_callback`.ArgillaCallbackHandler¶

class langchain_community.callbacks.argilla_callback.ArgillaCallbackHandler(dataset_name: str, workspace_name: Optional[str] = None, api_url: Optional[str] = None, api_key: Optional[str] = None)[source]¶

Callback Handler that logs into Argilla.

Parameters

dataset_name (str) – name of the FeedbackDataset in Argilla. Note that it must exist in advance. If you need help on how to create a FeedbackDataset in Argilla, please visit https://docs.argilla.io/en/latest/tutorials_and_integrations/integrations/use_argilla_callback_in_langchain.html.
workspace_name (Optional[str]) – name of the workspace in Argilla where the specified FeedbackDataset lives in. Defaults to None, which means that the default workspace will be used.
api_url (Optional[str]) – URL of the Argilla Server that we want to use, and where the FeedbackDataset lives in. Defaults to None, which means that either ARGILLA_API_URL environment variable or the default will be used.
api_key (Optional[str]) – API Key to connect to the Argilla Server. Defaults to None, which means that either ARGILLA_API_KEY environment variable or the default will be used.

Raises

ImportError – if the argilla package is not installed.
ConnectionError – if the connection to Argilla fails.
FileNotFoundError – if the FeedbackDataset retrieval from Argilla fails.

Examples

>>> from langchain_community.llms import OpenAI
>>> from langchain_community.callbacks import ArgillaCallbackHandler
>>> argilla_callback = ArgillaCallbackHandler(
...     dataset_name="my-dataset",
...     workspace_name="my-workspace",
...     api_url="http://localhost:6900",
...     api_key="argilla.apikey",
... )
>>> llm = OpenAI(
...     temperature=0,
...     callbacks=[argilla_callback],
...     verbose=True,
...     openai_api_key="API_KEY_HERE",
... )
>>> llm.generate([
...     "What is the best NLP-annotation tool out there? (no bias at all)",
... ])
"Argilla, no doubt about it."

Initializes the ArgillaCallbackHandler.

Parameters

dataset_name (str) – name of the FeedbackDataset in Argilla. Note that it must exist in advance. If you need help on how to create a FeedbackDataset in Argilla, please visit https://docs.argilla.io/en/latest/tutorials_and_integrations/integrations/use_argilla_callback_in_langchain.html.
workspace_name (Optional[str]) – name of the workspace in Argilla where the specified FeedbackDataset lives in. Defaults to None, which means that the default workspace will be used.
api_url (Optional[str]) – URL of the Argilla Server that we want to use, and where the FeedbackDataset lives in. Defaults to None, which means that either ARGILLA_API_URL environment variable or the default will be used.
api_key (Optional[str]) – API Key to connect to the Argilla Server. Defaults to None, which means that either ARGILLA_API_KEY environment variable or the default will be used.

Raises

ImportError – if the argilla package is not installed.
ConnectionError – if the connection to Argilla fails.
FileNotFoundError – if the FeedbackDataset retrieval from Argilla fails.

Attributes

`BLOG_URL`
`DEFAULT_API_URL`
`ISSUES_URL`
`REPO_URL`
`ignore_agent`	Whether to ignore agent callbacks.
`ignore_chain`	Whether to ignore chain callbacks.
`ignore_chat_model`	Whether to ignore chat model callbacks.
`ignore_llm`	Whether to ignore LLM callbacks.
`ignore_retriever`	Whether to ignore retriever callbacks.
`ignore_retry`	Whether to ignore retry callbacks.
`raise_error`
`run_inline`

Methods

`__init__`(dataset_name[, workspace_name, ...])	Initializes the ArgillaCallbackHandler.
`on_agent_action`(action, **kwargs)	Do nothing when agent takes a specific action.
`on_agent_finish`(finish, **kwargs)	Do nothing
`on_chain_end`(outputs, **kwargs)	If either the parent_run_id or the run_id is in self.prompts, then log the outputs to Argilla, and pop the run from self.prompts.
`on_chain_error`(error, **kwargs)	Do nothing when LLM chain outputs an error.
`on_chain_start`(serialized, inputs, **kwargs)	If the key input is in inputs, then save it in self.prompts using either the parent_run_id or the run_id as the key.
`on_chat_model_start`(serialized, messages, *, ...)	Run when a chat model starts running.
`on_llm_end`(response, **kwargs)	Log records to Argilla when an LLM ends.
`on_llm_error`(error, **kwargs)	Do nothing when LLM outputs an error.
`on_llm_new_token`(token, **kwargs)	Do nothing when a new token is generated.
`on_llm_start`(serialized, prompts, **kwargs)	Save the prompts in memory when an LLM starts.
`on_retriever_end`(documents, *, run_id[, ...])	Run when Retriever ends running.
`on_retriever_error`(error, *, run_id[, ...])	Run when Retriever errors.
`on_retriever_start`(serialized, query, *, run_id)	Run when Retriever starts running.
`on_retry`(retry_state, *, run_id[, parent_run_id])	Run on a retry event.
`on_text`(text, **kwargs)	Do nothing
`on_tool_end`(output[, observation_prefix, ...])	Do nothing when tool ends.
`on_tool_error`(error, **kwargs)	Do nothing when tool outputs an error.
`on_tool_start`(serialized, input_str, **kwargs)	Do nothing when tool starts.

__init__(dataset_name: str, workspace_name: Optional[str] = None, api_url: Optional[str] = None, api_key: Optional[str] = None) → None[source]¶

Initializes the ArgillaCallbackHandler.

Parameters

dataset_name (str) – name of the FeedbackDataset in Argilla. Note that it must exist in advance. If you need help on how to create a FeedbackDataset in Argilla, please visit https://docs.argilla.io/en/latest/tutorials_and_integrations/integrations/use_argilla_callback_in_langchain.html.
workspace_name (Optional[str]) – name of the workspace in Argilla where the specified FeedbackDataset lives in. Defaults to None, which means that the default workspace will be used.
api_url (Optional[str]) – URL of the Argilla Server that we want to use, and where the FeedbackDataset lives in. Defaults to None, which means that either ARGILLA_API_URL environment variable or the default will be used.
api_key (Optional[str]) – API Key to connect to the Argilla Server. Defaults to None, which means that either ARGILLA_API_KEY environment variable or the default will be used.

Raises

ImportError – if the argilla package is not installed.
ConnectionError – if the connection to Argilla fails.
FileNotFoundError – if the FeedbackDataset retrieval from Argilla fails.

Return type

None

on_agent_action(action: AgentAction, **kwargs: Any) → Any[source]¶

Do nothing when agent takes a specific action.

Parameters

action (AgentAction) –
kwargs (Any) –

Return type

Any

on_agent_finish(finish: AgentFinish, **kwargs: Any) → None[source]¶

Do nothing

Parameters

finish (AgentFinish) –
kwargs (Any) –

Return type

None

on_chain_end(outputs: Dict[str, Any], **kwargs: Any) → None[source]¶

If either the parent_run_id or the run_id is in self.prompts, then log the outputs to Argilla, and pop the run from self.prompts. The behavior differs if the output is a list or not.

Parameters

outputs (Dict[str, Any]) –
kwargs (Any) –

Return type

None

on_chain_error(error: BaseException, **kwargs: Any) → None[source]¶

Do nothing when LLM chain outputs an error.

Parameters

error (BaseException) –
kwargs (Any) –

Return type

None

on_chain_start(serialized: Dict[str, Any], inputs: Dict[str, Any], **kwargs: Any) → None[source]¶

If the key input is in inputs, then save it in self.prompts using either the parent_run_id or the run_id as the key. This is done so that we don’t log the same input prompt twice, once when the LLM starts and once when the chain starts.

Parameters

serialized (Dict[str, Any]) –
inputs (Dict[str, Any]) –
kwargs (Any) –

Return type

None

on_chat_model_start(serialized: Dict[str, Any], messages: List[List[BaseMessage]], *, run_id: UUID, parent_run_id: Optional[UUID] = None, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, **kwargs: Any) → Any¶

Run when a chat model starts running.

ATTENTION: This method is called for chat models. If you’re implementing: a handler for a non-chat model, you should use on_llm_start instead.

Parameters

serialized (Dict[str, Any]) –
messages (List[List[BaseMessage]]) –
run_id (UUID) –
parent_run_id (Optional[UUID]) –
tags (Optional[List[str]]) –
metadata (Optional[Dict[str, Any]]) –
kwargs (Any) –

Return type

Any

on_llm_end(response: LLMResult, **kwargs: Any) → None[source]¶

Log records to Argilla when an LLM ends.

Parameters

response (LLMResult) –
kwargs (Any) –

Return type

None

on_llm_error(error: BaseException, **kwargs: Any) → None[source]¶

Do nothing when LLM outputs an error.

Parameters

error (BaseException) –
kwargs (Any) –

Return type

None

on_llm_new_token(token: str, **kwargs: Any) → None[source]¶

Do nothing when a new token is generated.

Parameters

token (str) –
kwargs (Any) –

Return type

None

on_llm_start(serialized: Dict[str, Any], prompts: List[str], **kwargs: Any) → None[source]¶

Save the prompts in memory when an LLM starts.

Parameters

serialized (Dict[str, Any]) –
prompts (List[str]) –
kwargs (Any) –

Return type

None

on_retriever_end(documents: Sequence[Document], *, run_id: UUID, parent_run_id: Optional[UUID] = None, **kwargs: Any) → Any¶

Run when Retriever ends running.

Parameters

documents (Sequence[Document]) –
run_id (UUID) –
parent_run_id (Optional[UUID]) –
kwargs (Any) –

Return type

Any

on_retriever_error(error: BaseException, *, run_id: UUID, parent_run_id: Optional[UUID] = None, **kwargs: Any) → Any¶

Run when Retriever errors.

Parameters

error (BaseException) –
run_id (UUID) –
parent_run_id (Optional[UUID]) –
kwargs (Any) –

Return type

Any

on_retriever_start(serialized: Dict[str, Any], query: str, *, run_id: UUID, parent_run_id: Optional[UUID] = None, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, **kwargs: Any) → Any¶

Run when Retriever starts running.

Parameters

serialized (Dict[str, Any]) –
query (str) –
run_id (UUID) –
parent_run_id (Optional[UUID]) –
tags (Optional[List[str]]) –
metadata (Optional[Dict[str, Any]]) –
kwargs (Any) –

Return type

Any

on_retry(retry_state: RetryCallState, *, run_id: UUID, parent_run_id: Optional[UUID] = None, **kwargs: Any) → Any¶

Run on a retry event.

Parameters

retry_state (RetryCallState) –
run_id (UUID) –
parent_run_id (Optional[UUID]) –
kwargs (Any) –

Return type

Any

on_text(text: str, **kwargs: Any) → None[source]¶

Do nothing

Parameters

text (str) –
kwargs (Any) –

Return type

None

on_tool_end(output: Any, observation_prefix: Optional[str] = None, llm_prefix: Optional[str] = None, **kwargs: Any) → None[source]¶

Do nothing when tool ends.

Parameters

output (Any) –
observation_prefix (Optional[str]) –
llm_prefix (Optional[str]) –
kwargs (Any) –

Return type

None

on_tool_error(error: BaseException, **kwargs: Any) → None[source]¶

Do nothing when tool outputs an error.

Parameters

error (BaseException) –
kwargs (Any) –

Return type

None

on_tool_start(serialized: Dict[str, Any], input_str: str, **kwargs: Any) → None[source]¶

Do nothing when tool starts.

Parameters

serialized (Dict[str, Any]) –
input_str (str) –
kwargs (Any) –

Return type

None

Examples using ArgillaCallbackHandler¶

Argilla

langchain_community.callbacks.argilla_callback.ArgillaCallbackHandler¶

Examples using ArgillaCallbackHandler¶

`langchain_community.callbacks.argilla_callback`.ArgillaCallbackHandler¶