Smart Flagging: Streamline Your AI Quality Assurance

Understanding Smart Flagging

What is Smart Flagging?

Smart Flagging is Alhena's intelligent monitoring system that automatically identifies AI responses that may require human review. Rather than requiring administrators to manually review every conversation, the system uses advanced algorithms to detect specific signals that warrant attention.

The feature works by scanning conversations in real-time, analyzing AI responses against your approved knowledge base, and flagging only those that meet certain criteria for human review.

This targeted approach transforms your quality assurance process from reviewing 100% of conversations to focusing only on the small fraction (typically less than 1%) that might need attention.

How Smart Flagging Works

The Smart Flagging system uses several detection mechanisms to identify potential issues in AI responses:

1. Outside Knowledge Detection: Identifies when the AI references information that isn't directly inferred from your approved knowledge base or documentation

2. Low Confidence Responses: Detects when the AI shows signs of uncertainty in its responses

3. Customer Frustration Signals: Recognizes interactions where sentiment analysis detects potential user dissatisfaction

4. Policy and Compliance Topics: Flags discussions involving sensitive areas like refunds, warranties, or other topics where accuracy is critical

5. System Fallbacks: Identifies cases where the AI couldn't generate an appropriate response

When any of these conditions are met, the conversation is automatically flagged for review, with a clear explanation of why the flag was raised. This allows your QA team to focus exclusively on conversations that truly need attention, rather than sifting through hundreds of routine interactions.

Accessing and Using Smart Flagging

Identifying Flagged Conversations

When using Alhena's Smart Flagging feature, you'll notice visual indicators that help you quickly identify conversations that require attention:

Orange Flag Indicators in Conversation Lists

In your conversation list, flagged messages are marked with a orange flag icon. This makes it easy to spot which conversations have been automatically identified as potentially needing review.

Figure 1: Yellow flag icons indicate conversations that have been automatically flagged for review.

Filtering for Flagged Conversations

To view only flagged conversations:

1. Navigate to the "Conversations" section in your Alhena dashboard

2. Click on the three-dot menu (⋮) in the top-right corner of the conversation list

3. Select "Flagged" from the dropdown menu

This filtering option allows you to focus exclusively on conversations that require your attention, rather than scrolling through all conversations.

Figure 2: Use the dropdown menu to filter and view only flagged conversations.

Reviewing Flagged Content

When you open a flagged conversation, you'll see a notification banner at the top of the message that says "This message has been flagged. Learn why." This banner provides access to detailed information about why the message was flagged.

Understanding Flag Reasons

To see why a message was flagged:

1. Click on the "This message has been flagged. Learn why." banner

2. The banner will expand to show the specific reason(s) for flagging

3. Review the explanation, which typically identifies statements that are not directly inferred from your knowledge base

Figure 3: Expanded view showing detailed explanation of why the message was flagged.

In the example above, the system has identified specific statements that are not directly inferred from the ingested knowledge. This transparency helps you quickly understand what potential issues might exist in the AI's response.

Providing Feedback on Flags

After reviewing a flagged message, you can provide feedback to help improve the system:

1. At the bottom of the expanded flag explanation, you'll see the question "Was this message flagged correctly?"

2. Click the thumbs-up (👍) button if you agree with the flag

3. Click the thumbs-down (👎) button if you believe the flag was unnecessary

This feedback mechanism helps train the system to better understand what should and shouldn't be flagged in the future, continuously improving the accuracy of the Smart Flagging feature.

Best Practices for Smart Flagging

Maximizing the Value of Smart Flagging

To get the most out of Alhena's Smart Flagging feature, consider implementing these best practices in your quality assurance workflow:

Establish a Regular Review Cadence

Consistency is key to effective quality assurance:

• Daily Quick Reviews: Spend 15-30 minutes each day reviewing newly flagged conversations

• Weekly Deep Dives: Schedule a weekly session to analyze patterns and trends in flagged content

• Monthly Reporting: Track metrics on flag frequency, accuracy, and resolution to measure improvement over time

A regular cadence ensures that flags are addressed promptly while also providing insights into longer-term trends.

Use Flags as Learning Opportunities

Each flagged conversation is an opportunity to improve your AI:

• Update Knowledge Base: When you encounter legitimate flags, update your AI's knowledge base to address the gap

• Refine Guidelines: Use patterns in flagged content to refine your AI's guidelines and personality settings

• Document Common Issues: Maintain a log of frequently flagged issues and their resolutions for team reference

Provide Consistent Feedback

The feedback mechanism is crucial for improving the system:

• Always Respond: Make it a practice to provide thumbs-up or thumbs-down feedback for every flag you review

• Be Consistent: Establish clear criteria for what constitutes a "good" flag to ensure consistency across your team

• Explain Disagreements: When marking a flag as incorrect, note why for your team's reference