Chatbot Intent and Sentiment Annotation: Designing Reliable Conversation Ground Truth

A customer can ask the same question in very different emotional states. “Can you cancel my subscription?” may be a routine request. Annotation is driving customer support chatbot capabilities. See how annotation is revolutionizing customer support chatbots. May be a routine request. “Cancel this now. I have asked three times” expresses the same core goal with a very different conversational context.

That is why chatbot intent and sentiment annotation should not collapse everything into one label. Alongside intent and sentiment, slot filling extracts the structured parameters that let systems act on user requests — see the nuances of slot filling in conversational speech for how that process works in practice. Intent describes what the customer wants to accomplish. Sentiment describes the evaluative tone expressed in the message or conversation. Emotion, urgency, escalation, and topic can be separate signals again.

When those dimensions are defined independently and connected through a clear annotation schema, conversational AI teams can build training data that represents real customer conversations without forcing complex language into overly simple categories.

This guide explains how to design that multi-layer ground truth for customer-support chatbots, virtual assistants, and conversational AI systems.

Separate Intent, Sentiment, Emotion, and Urgency

The first design decision is deciding which conversational signals belong in separate annotation layers.

Consider:

“I was charged twice again. Please refund the duplicate charge today.”

Several labels may be relevant:

Separating these layers makes the dataset easier to interpret and prevents one label from carrying several meanings.

Annotera provides dedicated intent classification services and sentiment annotation services. A conversational AI program can use either layer independently or combine them where the application requires both.

Build the Intent Taxonomy Around Customer Goals

An intent taxonomy should describe what customers are trying to accomplish.

For a commerce support chatbot, possible intents might include:

• Track_Order;
• Cancel_Order;
• Change_Shipping_Address;
• Request_Refund;
• Report_Damaged_Item;
• Update_Payment_Method;
• Cancel_Subscription;
• Reset_Password; and
• Request_Human_Agent.

The exact taxonomy should come from the actions, workflows, or information needs the conversational system is designed to support.

Every intent should have:

• a clear definition;
• in-scope examples;
• out-of-scope examples;
• confusable neighboring intents;
• entity or slot requirements where relevant;
• multi-intent rules;
• context rules; and
• an escalation procedure for ambiguous messages.

For a deeper intent-specific framework, see Annotera’s guide to intent annotation for conversational AI.

Choose the Right Intent Granularity

An intent taxonomy can fail because its categories are too broad or because they are so narrow that humans cannot apply them consistently.

Hierarchical taxonomies can help when broad and specific distinctions are both useful.

For example:

Billing → Charge Problem → Duplicate Charge

The hierarchy should support the downstream system rather than add complexity for its own sake.

Define How Multi-Intent Messages Are Labeled

Customers frequently ask for more than one thing in the same message.

“Where is my order, and can I change the delivery address before it arrives?”

This contains at least two potential intents:

• Track_Order; and
• Change_Shipping_Address.

The annotation specification needs to decide whether the dataset supports:

• a single primary intent;
• primary and secondary intent;
• multi-label intent annotation;
• separate intent spans within the message; or
• message splitting before classification.

Annotera’s intent-classification framework includes support for multi-intent annotation, making this an important distinction to preserve when the conversational system must respond to several customer goals in one turn.

Use Conversation Context When Intent Depends on Earlier Turns

Some customer utterances cannot be classified correctly in isolation.

Consider:

Bot: Would you like to cancel the order or change the delivery date?
Customer: Cancel it.

The phrase “Cancel it” does not identify the object without the preceding turn.

If the production chatbot uses dialogue context, the annotation process should provide annotators with enough surrounding conversation to make the same interpretation.

The guideline should define:

• how many prior turns annotators can inspect;
• whether agent or bot messages are included;
• whether context can override the literal wording of the current turn;
• how unresolved references are labeled; and
• how conversation resets or topic changes are handled.

Track Intent Shifts Across a Conversation

A customer’s goal can change during the same support session.

For example:

Assigning one intent to the whole conversation would erase that progression.

For systems that respond turn by turn, utterance-level intent labels plus conversation identifiers preserve this change more effectively.

Define the Sentiment Schema Separately

Sentiment annotation represents evaluative tone rather than customer action.

A basic polarity schema might contain:

• Positive;
• Neutral;
• Negative; and
• Mixed.

A more detailed project may add sentiment intensity or separate emotion categories.

Before adding more classes, decide whether annotators can distinguish those states consistently from the available text.

Annotera’s sentiment analysis annotation services support polarity, fine-grained sentiment, emotion categories, sarcasm, aspect-based sentiment, and contextual interpretation.

Do Not Confuse Sentiment With Emotion or Urgency

Sentiment, emotion, and urgency can correlate, but they answer different questions.

For example:

“My card expires next month. Can you send the replacement now?”

The request may be urgent without expressing negative sentiment.

Conversely:

“I really dislike the new app, but I don’t need help right now.”

This can express negative sentiment without urgency.

Separate labels preserve those distinctions.

Create Rules for Mixed and Aspect-Specific Sentiment

Customer messages do not always contain one uniform sentiment.

“The product is excellent, but delivery was terrible.”

A document-level positive or negative label loses important information.

Depending on the use case, the project can annotate:

• overall sentiment: Mixed;
• product sentiment: Positive; and
• delivery sentiment: Negative.

Aspect-based sentiment becomes useful when the downstream system needs to connect opinions to particular products, services, features, agents, or stages of the customer journey.

Handle Sarcasm, Negation, and Indirect Language

Literal words are not always reliable sentiment signals.

Consider:

“Great. Another failed payment.”

The word great is positive in isolation, but the full utterance is likely expressing dissatisfaction.

Other difficult structures include:

• “Not bad at all.”
• “I wouldn’t say I’m happy.”
• “Thanks for making me wait another hour.”
• “Exactly what I needed… another error.”
• “Could have been worse.”

The guidelines should include examples of negation, sarcasm, understatement, intensifiers, and indirect complaints when those patterns occur in the target data.

Annotate Sentiment Change Across Dialogue Turns

Customer sentiment can change during a successful or unsuccessful support conversation.

A single conversation-level label such as “Positive” would hide the original frustration. A single “Negative” label would hide the successful recovery.

For use cases that need conversation dynamics, annotate sentiment at the utterance or segment level and derive conversation-level summaries separately.

Annotera’s guide to real-time sentiment tagging in customer support focuses specifically on this sentiment-change use case.

Allow Legitimate Intent-Sentiment Combinations

Intent and sentiment should be analyzed together, but one should not automatically dictate the other.

Consider the intent:

Cancel_Subscription

It can appear with several sentiments:

All three can be valid.

Cross-layer QA can still identify unusual combinations for review, but unexpected does not mean incorrect.

Keep Escalation Signals Separate From Sentiment

Negative sentiment does not automatically mean a customer should be escalated, and neutral language does not mean escalation is unnecessary.

Compare:

“This is frustrating, but I’ll try the reset again.”

with:

“Please connect me to a supervisor.”

The first expresses frustration but does not necessarily request escalation. The second explicitly requests escalation even though its tone is linguistically neutral.

If escalation is important to the conversational system, treat it as its own intent, event, or operational annotation layer.

Potential labels might include:

• Escalation_Not_Requested;
• Human_Agent_Requested;
• Supervisor_Requested;
• Repeated_Unresolved_Issue; or
• Escalation_Review_Required.

The exact categories should match the support workflow rather than be inferred automatically from sentiment polarity.

Define Ambiguous, Unknown, and Out-of-Scope Cases

Not every message can be assigned confidently to a supported intent or sentiment class.

Common situations include:

• message is too vague;
• required context is missing;
• customer changes topic mid-message;
• request falls outside the chatbot’s supported scope;
• several intents are equally plausible;
• sentiment cannot be established reliably;
• text is incomplete;
• message contains only an attachment or URL; or
• language is not supported by the current annotation team.

The schema should distinguish useful states such as:

These labels are preferable to forcing uncertain examples into the nearest available class.

Cover Real Customer Language

Customer-support language differs from polished product documentation.

Datasets may contain:

• typos;
• abbreviations;
• sentence fragments;
• slang;
• code-switching;
• regional vocabulary;
• non-native phrasing;
• emojis;
• all-caps emphasis;
• repeated punctuation;
• product-specific terminology; and
• new language that appears after launches or policy changes.

Intent coverage should include multiple ways of expressing the same customer goal.

Sentiment coverage should likewise include direct and indirect ways customers express approval, dissatisfaction, uncertainty, relief, or frustration.

For intent-specific guidance on regional and linguistic variation, see why high-quality intent annotation matters for chatbot performance.

Measure Intent and Sentiment Annotation Quality Separately

A combined intent-and-sentiment project should not report only one overall annotation-accuracy number.

Different error types require different corrective actions.

Results should also be broken down by intent class, sentiment class, language, channel, product area, and conversation type where those dimensions affect difficulty.

Build a Conversational Annotation Calibration Set

Before scaling annotation, give several annotators the same representative set of customer conversations.

The calibration set should deliberately include:

• clear single-intent messages;
• multi-intent messages;
• context-dependent replies;
• intent shifts;
• neutral sentiment;
• strong positive and negative sentiment;
• mixed sentiment;
• sarcasm;
• negation;
• ambiguous intent;
• explicit human-agent requests;
• urgent but neutral requests;
• frustrated messages without escalation requests;
• slang and abbreviations;
• regional phrasing;
• typos and sentence fragments;
• out-of-scope requests; and
• examples that should remain uncertain.

Disagreement should be analyzed by cause.

If annotators repeatedly confuse two intents, the taxonomy may overlap. If they agree on intent but disagree on sentiment, the sentiment guideline may need better examples. If context-dependent turns produce disagreement, annotators may need more conversation history.

Update the Dataset as Customer Language Changes

Customer-support taxonomies are rarely static.

New language can appear because of:

• new products;
• policy changes;
• new payment methods;
• new delivery options;
• service incidents;
• new chatbot capabilities;
• regional expansion;
• new slang or abbreviations; and
• previously unseen customer issues.

Production logs can help identify messages with low model confidence, repeated fallback behavior, emerging intent clusters, or frequent human reclassification.

Those examples can enter a controlled review process before new taxonomy categories or guideline changes are introduced.

Taxonomy and guideline versions should be recorded so teams know which labeling rules were used for each dataset release.

Intent and Sentiment Annotation Checklist

Before a conversational AI dataset moves into production, answer these questions:

• Which conversational signals are separate annotation layers?
• What does each intent represent?
• Are intent categories mutually distinguishable?
• Is there an intent hierarchy?
• Can one message contain multiple intents?
• How is the primary intent selected, if required?
• How much conversation context can annotators use?
• How are intent changes across turns represented?
• What sentiment taxonomy is being used?
• Is mixed sentiment supported?
• Are emotion labels separate from sentiment?
• Is urgency separate from sentiment?
• Is escalation represented as a separate signal?
• How is sarcasm handled?
• How is negation handled?
• How are ambiguous intents labeled?
• How are out-of-scope requests labeled?
• Can annotators mark sentiment as unclear?
• Does the data represent real customer language variation?
• Are intent and sentiment scored separately during QA?
• Are difficult examples independently adjudicated?
• How will new intents and language patterns be identified?
• How are taxonomy and guideline versions tracked?

How Annotera Supports Conversational AI Annotation

Annotera supports conversational AI teams with text annotation services that can combine intent classification, sentiment labeling, semantic annotation, entity extraction, categorization, and other NLP requirements.

Intent projects can include custom taxonomies, hierarchical categories, multi-intent annotation, contextual intent resolution, conversation-level labeling, calibration datasets, QA, and adjudication.

Sentiment projects can include polarity, mixed sentiment, emotion categories, aspect-level labels, sarcasm handling, contextual interpretation, and gold-standard review sets.

Where both dimensions are required, Annotera can structure them as separate but connected annotation layers so teams can analyze what customers want without losing how the conversation is developing.

For voice-based customer interactions, where tone, pitch, pace, and other acoustic information introduce additional signals, see Annotera’s speech sentiment annotation services.

Conclusion: Preserve the Different Signals Inside a Conversation

A customer-support message can contain a goal, an opinion, an emotion, a degree of urgency, and a request for escalation at the same time.

The annotation schema should preserve those distinctions rather than compress them into one broad idea of customer “intent” or “sentiment.”

Start with a clear intent taxonomy. Define sentiment independently. Add emotion, urgency, escalation, and aspect-level labels only when the downstream system needs them. Then define multi-intent rules, contextual labeling, ambiguity handling, conversation trajectories, and QA criteria before annotation scales.

The result is conversational ground truth that represents not only what customers say, but the different signals an AI system may need to interpret correctly.

Building a customer-support chatbot or conversational AI dataset? Talk to Annotera about your intent and sentiment annotation requirements and design the taxonomy, contextual rules, QA framework, and dataset workflow around your application.