FeedbackCollector Streamlit Integration

The FeedbackCollector takes user feedback from within an app and saves it to Trubrics.

Install

To get started with Streamlit, install the additional dependency:

pip install "trubrics[streamlit]"

Streamlit Example Apps

Once you have created an account with Trubrics, you can try our deployed example Streamlit apps that use the integration to save feedback:

• LLM chat - deployed app | code : A chatbot that queries OpenAI's API and allows users to leave feedback.
• LLM single answer - deployed app | code : An LLM app that queries OpenAI's API and allows users to leave feedback on single text generations.

The code for these apps can be viewed in the trubrics-sdk, and may be run by cloning the repo and running:

LLM chatLLM single answer

Tip

To run this app, you are required to have your own OpenAI API key.

Install openai:

pip install openai

Then save your OpenAI API key with OPENAI_API_KEY='your_openai_key' in st.secrets, and run:

streamlit run examples/feedback/streamlit/llm_chatbot.py

Tip

To run this app, you are required to have your own OpenAI API key.

Install openai:

pip install openai

Then save your OpenAI API key with OPENAI_API_KEY='your_openai_key' in st.secrets, and run:

streamlit run examples/feedback/streamlit/llm_app.py

Add the FeedbackCollector to your App

Here is a complete example to log user prompts and feedback from a simple streamlit application:

examples/streamlit/basic_app.py

import streamlit as st
from trubrics.integrations.streamlit import FeedbackCollector

if "logged_prompt" not in st.session_state:
    st.session_state.logged_prompt = None
if "feedback_key" not in st.session_state:
    st.session_state.feedback_key = 0

# 1. authenticate with trubrics
collector = FeedbackCollector(
    email=st.secrets.TRUBRICS_EMAIL,
    password=st.secrets.TRUBRICS_PASSWORD,
    project="default"
)

if st.button("Refresh"):
    st.session_state.feedback_key += 1
    st.session_state.logged_prompt = None
    st.experimental_rerun()

prompt = "Tell me a joke"
generation = "Why did the chicken cross the road? To get to the other side."
st.write(f"#### :orange[Example user prompt: {prompt}]")


if st.button("Generate response"):
    # 2. log a user prompt & model response
    st.session_state.logged_prompt = collector.log_prompt(
        config_model={"model": "gpt-3.5-turbo"},
        prompt=prompt,
        generation=generation,
    )

if st.session_state.logged_prompt:
    st.write(f"#### :blue[Example model generation: {generation}]")
    # 3. log some user feedback
    user_feedback = collector.st_feedback(
        component="default",
        feedback_type="thumbs",
        open_feedback_label="[Optional] Provide additional feedback",
        model=st.session_state.logged_prompt.config_model.model,
        prompt_id=st.session_state.logged_prompt.id,
        key=st.session_state.feedback_key,
        align="flex-start",
    )

What's going on here? Let's break down this snippet:

1. FeedbackCollector()

Tip

The authentication token is cached already, but to optimise your app further, wrap the FeedbackCollector in @st.cache_data.

FeedbackCollector object

Parameters:

Name	Type	Description	Default
project	Optional[str]	a Trubrics project name	required
email	Optional[str]	a Trubrics account email	required
password	Optional[str]	a Trubrics account password	required

Source code in trubrics/integrations/streamlit/collect.py

def __init__(
    self,
    project: Optional[str],
    email: Optional[str],
    password: Optional[str],
    firebase_api_key: Optional[str] = None,
    firebase_project_id: Optional[str] = None,
):
    """
    Args:
        project: a Trubrics project name
        email: a Trubrics account email
        password: a Trubrics account password
    """

    if email and password and project:
        super().__init__(
            email=email,
            password=password,
            project=project,
            firebase_api_key=firebase_api_key,
            firebase_project_id=firebase_project_id,
        )

2. collector.log_prompt()

.log_prompt() parameters

Log user prompts to Trubrics.

Parameters:

Name	Type	Description	Default
config_model	dict	model configuration with fields "model", "prompt_template", "temperature"	required
prompt	str	user prompt to the model	required
generation	str	model generation	required
user_id	Optional[str]	user id	None
session_id	Optional[str]	session id, for example for a chatbot conversation	None
tags	list	feedback tags	[]
metadata	dict	any feedback metadata	{}

Source code in trubrics/platform/__init__.py

def log_prompt(
    self,
    config_model: dict,
    prompt: str,
    generation: str,
    user_id: Optional[str] = None,
    session_id: Optional[str] = None,
    tags: list = [],
    metadata: dict = {},
) -> Optional[Prompt]:
    """
    Log user prompts to Trubrics.

    Parameters:
        config_model: model configuration with fields "model", "prompt_template", "temperature"
        prompt: user prompt to the model
        generation: model generation
        user_id: user id
        session_id: session id, for example for a chatbot conversation
        tags: feedback tags
        metadata: any feedback metadata
    """
    config_model = ModelConfig(**config_model)
    prompt = Prompt(
        config_model=config_model,
        prompt=prompt,
        generation=generation,
        user_id=user_id,
        session_id=session_id,
        tags=tags,
        metadata=metadata,
    )
    auth = get_trubrics_auth_token(
        self.config.firebase_api_key,
        self.config.email,
        self.config.password.get_secret_value(),
        rerun=expire_after_n_seconds(),
    )
    res = save_document_to_collection(
        auth,
        firestore_api_url=self.config.firestore_api_url,
        project=self.config.project,
        collection="prompts",
        document=prompt,
    )
    if "error" in res:
        logger.error(res["error"])
        return None
    else:
        logger.info("User prompt saved to Trubrics.")
        prompt.id = res["name"].split("/")[-1]
        return prompt

3. collector.st_feedback()

.st_feedback() parameters

Collect ML model user feedback with UI components from a Streamlit app.

Parameters:

Name	Type	Description	Default
component	str	component name. Create a new component directly in Trubrics.	required
feedback_type	str	type of feedback to be collected textbox: open textbox feedback thumbs: 👍 / 👎 UI buttons faces: 😞 / 🙁 / 😐 / 🙂 / 😀 UI buttons	required
textbox_type	str	if textbox selected as feedback_type, the type of textbox to use ["text-input", "text-area"]	'text-input'
model	str	the model used - can be a model version, a link to the saved model artifact in cloud storage, etc	required
prompt_id	Optional[str]	id of the prompt object	None
tags	list	a list of tags for the feedback	[]
metadata	dict	any data to save with the feedback	{}
user_id	Optional[str]	an optional reference to a user, for example a username if there is a login, a cookie ID, etc	None
key	Optional[str]	a key for the streamlit components (necessary if calling this method multiple times)	None
open_feedback_label	Optional[str]	label of optional text_input for "faces" or "thumbs" feedback_type	None
save_to_trubrics	bool	whether to save the feedback to Trubrics, or just to return the feedback object	True
disable_with_score	Optional[str]	an optional score to disable the component. Must be a "thumbs" emoji or a "faces" emoji. Can be used to pass state from one component to another.	None
align	str	where to align the feedback component ["flex-end", "center", "flex-start"]	'flex-end'
success_fail_message	bool	whether to display an st.toast message on feedback submission.	True

Source code in trubrics/integrations/streamlit/collect.py

def st_feedback(
    self,
    component: str,
    feedback_type: str,
    model: str,
    textbox_type: str = "text-input",
    prompt_id: Optional[str] = None,
    tags: list = [],
    metadata: dict = {},
    user_id: Optional[str] = None,
    key: Optional[str] = None,
    open_feedback_label: Optional[str] = None,
    save_to_trubrics: bool = True,
    align: str = "flex-end",
    disable_with_score: Optional[str] = None,
    success_fail_message: bool = True,
) -> Optional[dict]:
    """
    Collect ML model user feedback with UI components from a Streamlit app.

    Args:
        component: component name. Create a new component directly in Trubrics.
        feedback_type: type of feedback to be collected

            - textbox: open textbox feedback
            - thumbs: 👍 / 👎 UI buttons
            - faces: 😞 / 🙁 / 😐 / 🙂 / 😀 UI buttons
        textbox_type: if textbox selected as feedback_type, the type of textbox to use ["text-input", "text-area"]
        model: the model used - can be a model version, a link to the saved model artifact in cloud storage, etc
        prompt_id: id of the prompt object
        tags: a list of tags for the feedback
        metadata: any data to save with the feedback
        user_id: an optional reference to a user, for example a username if there is a login, a cookie ID, etc
        key: a key for the streamlit components (necessary if calling this method multiple times)
        open_feedback_label: label of optional text_input for "faces" or "thumbs" feedback_type
        save_to_trubrics: whether to save the feedback to Trubrics, or just to return the feedback object
        disable_with_score: an optional score to disable the component. Must be a "thumbs" emoji or a "faces" emoji.
            Can be used to pass state from one component to another.
        align: where to align the feedback component ["flex-end", "center", "flex-start"]
        success_fail_message: whether to display an st.toast message on feedback submission.
    """
    if key is None:
        key = feedback_type
    if feedback_type == "textbox":
        text = self.st_textbox_ui(type=textbox_type, key=key, label=open_feedback_label)
        if text:
            user_response = {"type": feedback_type, "score": None, "text": text}
            if save_to_trubrics:
                feedback = self.log_feedback(
                    component=component,
                    user_response=user_response,
                    model=model,
                    prompt_id=prompt_id,
                    metadata=metadata,
                    tags=tags,
                    user_id=user_id,
                )
                if feedback is None:
                    error_msg = "Error in pushing feedback issue to Trubrics."
                    if success_fail_message:
                        st.error(error_msg)
                else:
                    if success_fail_message:
                        st.success("Feedback saved to Trubrics.")
                    return self._pydantic_to_dict(feedback)
            else:
                user_response = Response(**user_response)
                feedback = Feedback(
                    component=component,
                    model=model,
                    user_response=user_response,
                    prompt_id=prompt_id,
                    user_id=user_id,
                    tags=tags,
                    metadata=metadata,
                )
                return self._pydantic_to_dict(feedback)
    elif feedback_type in ("thumbs", "faces"):

        def _log_feedback_trubrics(user_response, **kwargs):
            feedback = self.log_feedback(user_response=user_response, **kwargs)
            if success_fail_message:
                if feedback:
                    st.toast("Feedback saved to [Trubrics](https://trubrics.streamlit.app/).", icon="✅")
                    return self._pydantic_to_dict(feedback)
                else:
                    st.toast("Error in saving feedback to [Trubrics](https://trubrics.streamlit.app/).", icon="❌")

        user_response = streamlit_feedback(
            feedback_type=feedback_type,
            optional_text_label=open_feedback_label,
            disable_with_score=disable_with_score,
            on_submit=_log_feedback_trubrics if save_to_trubrics else None,
            kwargs={
                "component": component,
                "model": model,
                "prompt_id": prompt_id,
                "metadata": metadata,
                "tags": tags,
                "user_id": user_id,
            },
            align=align,
            key=key,
        )
        if save_to_trubrics is False and user_response:
            user_response = Response(**user_response)
            feedback = Feedback(
                component=component,
                model=model,
                user_response=user_response,
                prompt_id=prompt_id,
                user_id=user_id,
                tags=tags,
                metadata=metadata,
            )
            return self._pydantic_to_dict(feedback)
        return user_response
    else:
        raise ValueError("feedback_type must be one of ['textbox', 'faces', 'thumbs'].")
    return None

---

Last update: November 15, 2023
Created: November 15, 2023