E-commerce Product Video Segmentation: How to Build Accurate Product Masks

A product can look very different from one frame of an e-commerce video to the next. It may rotate, deform, pass behind a hand, reflect studio lights, move in and out of focus, or briefly disappear behind another product.

That makes e-commerce product video segmentation more than a matter of tracing a clean outline. Before annotation begins, teams must decide what the product mask actually represents and how that definition should remain consistent throughout the video.

Should a mask include only the visible part of a handbag when a model’s hand covers the handle? Should the empty space inside the handle be background? How should a transparent bottle be outlined? Should a loose garment be segmented differently when it is folded, hanging, or worn by a model?

These decisions determine whether product masks remain useful across visual search, product isolation, video merchandising, tracking, catalog enrichment, and other retail computer vision workflows.

This guide explains how to define product-mask ground truth across video frames, including boundary rules, occlusion, transparency, reflections, deformation, product identity, temporal stability, and quality assurance.

Start by Defining What the Product Mask Represents

The term product segmentation can describe several different ground-truth objectives.

One project may need the visible product silhouette for background separation. Another may need individual instances separated during product tracking. A third may need a product region only to improve downstream recognition.

Before selecting annotation rules, define what the model is expected to learn.

Annotera’s polygon video annotation services support contour-based object segmentation across video sequences. The exact boundary convention should be selected from the model objective rather than assumed to be universal.

Choose Between Visible and Amodal Product Masks

Occlusion introduces one of the most important decisions in product segmentation.

Suppose a shopper’s hand covers part of a handbag.

A visible mask labels only the handbag pixels that can actually be seen.

An amodal mask attempts to represent the full expected object shape, including the portion hidden behind the hand.

Neither approach is automatically better.

The critical requirement is to choose one according to the downstream task and avoid switching between visible and inferred boundaries from frame to frame.

Define Product Versus Background Precisely

Many product-boundary errors come from uncertainty about where the product ends and the scene begins.

An annotation guide should address details such as:

• outer product silhouette;
• handles;
• straps;
• shoelaces;
• tags attached to the product;
• packaging;
• hangers;
• stands;
• mannequin parts;
• display fixtures;
• promotional stickers; and
• detachable accessories.

A handbag’s removable shoulder strap, for example, may be part of the target product in one dataset and a separate accessory instance in another.

The correct decision depends on the product ontology and intended model output.

Decide How to Handle Holes and Openings

Many retail products contain interior areas through which background remains visible.

Examples include:

• handbag handles;
• shoe straps;
• rings and bracelets;
• open-frame furniture;
• bicycle frames;
• wire baskets;
• eyeglass frames; and
• decorative cutouts.

The project must decide whether these interior openings remain background or are absorbed into one simplified product region.

If the downstream task requires an accurate silhouette for compositing or instance segmentation, interior holes may matter significantly.

If the mask is only being used for coarse product localization, that level of geometric detail may be unnecessary.

Create Separate Rules for Transparent and Semi-Transparent Products

Transparent products create a boundary problem because background pixels remain visible through the object.

Common examples include:

• glassware;
• transparent bottles;
• clear plastic packaging;
• eyeglass lenses;
• cosmetic containers;
• transparent furniture;
• mesh or sheer fabrics; and
• semi-transparent accessories.

A binary polygon contour may represent the object’s external silhouette, but that does not automatically capture degrees of transparency inside the contour.

Therefore, the annotation specification should answer:

• Is the outer silhouette enough?
• Should transparent interior regions remain part of the object?
• Are holes and transparent material treated differently?
• Does the project need opacity information?
• Should uncertain edges be reviewed by a specialist?

If the application needs realistic foreground extraction or compositing around transparent materials, a polygon alone may not provide the required representation.

Treat Reflections as Appearance, Not New Geometry

Reflective products create a different challenge.

Metal appliances, glossy packaging, jewelry, mirrors, polished shoes, electronics, and glass containers can reflect objects and lights from the surrounding scene.

A reflection appearing inside the physical product boundary normally does not create a new object boundary.

The annotation guide should help reviewers distinguish:

• physical product edge;
• surface reflection;
• cast highlight;
• transparent region;
• background object visible through the material; and
• genuine separate object overlap.

Without examples, annotators can mistakenly follow bright reflections or color changes rather than the product’s geometry.

Define Whether Shadows Belong to the Product

Studio and lifestyle product videos frequently contain shadows.

For most object-instance segmentation tasks, a cast shadow is a scene effect rather than part of the physical product. But the project specification should state this explicitly.

Teams should distinguish between:

• product pixels;
• cast shadows;
• self-shadowing within the product;
• reflections; and
• dark background areas adjacent to the object.

Clear examples are particularly useful when products are black, glossy, or photographed against dark backgrounds.

Handle Hands, Models, and Mannequins Consistently

E-commerce products are often demonstrated rather than filmed in isolation.

A person may:

• hold a product;
• wear apparel;
• carry a bag;
• put on glasses;
• rotate footwear;
• open packaging;
• demonstrate an appliance; or
• touch several products at once.

The annotation schema needs to define whether human pixels are excluded from the product mask and how occluded product boundaries are handled.

For a visible-only handbag mask, the hand covering the handle would typically interrupt the visible product region.

For an amodal-object task, the hidden part may instead be represented according to the project’s inference rules.

The same distinction applies to products worn on models. Apparel segmentation, human parsing, and complete garment reconstruction can require different annotation targets.

Account for Deformable Products

Rigid products usually retain a similar shape as they rotate. Deformable products do not.

Examples include:

• shirts;
• dresses;
• scarves;
• soft handbags;
• blankets;
• shoes with flexible uppers;
• soft packaging; and
• other textiles.

A garment can be flat in one frame, folded in another, and worn by a person later in the sequence.

For these products, annotations should follow the actual target boundary in each frame rather than trying to preserve one rigid template.

This also affects interpolation. Boundaries generated between two accurate keyframes can still drift when the object’s shape changes nonlinearly.

Create Rules for Motion Blur and Soft Boundaries

Fast product movement can make the edge of an object visually uncertain.

A shoe being rotated quickly, a garment swinging, or a product moved across the camera can create blur that extends beyond the sharp physical outline.

The annotation guide should define whether the polygon follows:

• the estimated physical object boundary;
• the visible blur envelope;
• a conservative visible region; or
• another project-specific convention.

There is no universal rule suitable for every model.

The important requirement is that boundary treatment remains consistent across annotators and frames.

Preserve Product Identity Across Frames

When the downstream task includes video object segmentation or product tracking, geometric masks alone are not enough.

The same physical product should retain the same instance or track identifier as it:

• moves;
• rotates;
• changes scale;
• passes behind a hand;
• becomes partially occluded;
• temporarily leaves the center of the frame; or
• changes appearance because of lighting.

The project should define when a track ends and when a reappearing product receives the same or a new ID.

This becomes especially important in videos containing visually identical products.

Separate Multiple and Overlapping Products

Retail videos frequently contain several products in one scene.

They may:

• touch;
• overlap;
• stack;
• sit inside other products;
• share similar colors;
• belong to the same SKU; or
• move together temporarily.

If the downstream task requires instance-level understanding, each physical object should retain a separate instance identity even when products belong to the same class.

The annotation guidelines should also specify whether packaging components count as separate objects and how grouped or bundled products are treated.

Align Masks With the Product Taxonomy

A perfect contour can still have the wrong semantic label.

An e-commerce segmentation project may classify masks at different levels:

• product;
• department;
• category;
• subcategory;
• brand;
• product family;
• variant;
• SKU; or
• product component.

The correct granularity depends on what the model will do with the segmented object.

If the model only needs to isolate footwear from the background, SKU-level labels may be unnecessary. If the segmented video supports product recognition, finer product identity may be required.

The taxonomy should also include explicit handling for unknown, discontinued, new, or visually indistinguishable products.

Use Keyframes and Interpolation Carefully

Video annotation tools can propagate or interpolate polygons between manually labeled frames.

This can reduce repetitive work when the object’s motion and shape change predictably.

However, interpolated masks need closer review when:

• the product rotates quickly;
• the product deforms;
• the camera cuts or changes angle;
• another object creates occlusion;
• the product enters or exits the frame;
• motion blur appears;
• fine appendages change position; or
• the visible silhouette changes abruptly.

For those frames, manual correction may be necessary even when interpolation works well elsewhere in the sequence.

Annotera’s broader guide to polygon annotation techniques for video focuses on motion, keyframes, interpolation, occlusion, and temporal consistency across computer vision applications.

Set Practical Polygon Vertex Rules

Adding more polygon points does not automatically make an annotation better.

Too few vertices can fail to represent meaningful product curvature. Too many can introduce unnecessary annotation effort and make the contour harder to maintain consistently across video.

Annotators should add vertices where they are needed to represent:

• changes in curvature;
• corners;
• thin protrusions;
• straps and handles;
• irregular contours;
• product openings; and
• other geometry required by the task.

Project guidelines should include examples of acceptable and excessive vertex density for the product categories being labeled.

Know When Polygon Annotation Is Not Enough

Polygon annotation is useful when a product can be represented effectively through a contour made from connected vertices.

But another representation may be more appropriate when the application needs:

• dense pixel-level class assignment;
• very fine hair-like structures;
• complex transparency;
• soft edges;
• opacity values;
• extremely detailed boundaries; or
• full-scene semantic segmentation.

Likewise, a bounding box may be sufficient if precise contours do not contribute to the model objective.

The goal is not to use the most detailed annotation available. It is to choose the representation that captures the information the target model requires.

For that decision, see Annotera’s Polygon vs. Segmentation: Choosing the Right Mask.

Build E-commerce Video Variation Into the Dataset

Product masks should be tested across the visual conditions the deployed system is expected to encounter.

Relevant coverage can include:

• studio product videos;
• lifestyle videos;
• mobile-phone footage;
• livestream commerce;
• products held in hand;
• products worn by models;
• rotating turntable videos;
• bright and low lighting;
• high-contrast scenes;
• busy backgrounds;
• similar-colored backgrounds;
• transparent products;
• reflective products;
• dark products;
• fine straps and accessories;
• rigid products;
• deformable products;
• multiple overlapping products; and
• different camera resolutions and compression levels.

The required mix should reflect the actual deployment scenario rather than an abstract idea of a “complete” retail dataset.

Measure Product Video Segmentation Quality

One overall annotation-accuracy score can hide important product-mask problems.

Where the final output is a rasterized segmentation mask, teams can also evaluate mask overlap against a validated reference set using an appropriate segmentation metric.

The QA framework should still preserve error categories because the same overall overlap score can result from very different problems, such as missing a strap, including a hand, or shifting the entire product boundary.

Build a Retail Product Segmentation Calibration Set

Before scaling annotation, give several annotators the same deliberately difficult product-video sample.

The calibration set should include examples such as:

• simple rigid products;
• products with handles;
• products with holes;
• transparent bottles;
• reflective packaging;
• jewelry;
• eyeglasses;
• fine straps;
• flowing apparel;
• dark products on dark backgrounds;
• products held by hands;
• products worn by models;
• partial occlusion;
• heavy occlusion;
• multiple identical products;
• rapid rotation;
• motion blur;
• lighting changes;
• interpolation drift; and
• examples whose correct boundary should remain uncertain.

Disagreement should then be analyzed by error type.

If annotators disagree on whether the empty space inside a handle belongs to the product mask, the hole convention needs clarification. If the same product repeatedly receives different IDs after an occlusion, the tracking rule needs clarification.

The calibration stage should improve the annotation specification before those differences spread across thousands of frames.

E-commerce Product Video Segmentation Checklist

Before production begins, the project owner should be able to answer these questions:

• What exactly does the product mask represent?
• Is the task visible-only or amodal segmentation?
• Are product shadows excluded?
• How are reflections handled?
• How are transparent materials represented?
• How are semi-transparent materials represented?
• Do internal product holes remain background?
• Are straps, handles, and fine appendages included?
• Are detachable accessories separate instances?
• Are packaging and product annotated together or separately?
• How are hands treated?
• How are mannequins treated?
• How are worn products separated from the person?
• How are deformable products handled?
• What boundary rule applies during motion blur?
• Does each product require a persistent instance ID?
• When does an object track end?
• How are identical overlapping products distinguished?
• At what product-taxonomy level are labels assigned?
• How are unknown products represented?
• Can polygons be interpolated between keyframes?
• Which frames require manual correction?
• What is acceptable polygon vertex density?
• When should another annotation method replace polygon labeling?
• Which product and video conditions must the dataset cover?
• Which QA measures are reported separately?
• How are annotation-rule changes version controlled?

How Annotera Supports E-commerce Video Segmentation

Annotera supports retail and e-commerce computer vision programs with polygon video annotation services designed for irregular objects, moving products, overlapping instances, and frame-to-frame boundary consistency.

Projects can include custom product taxonomies, visible or amodal boundary rules, instance IDs, occlusion conventions, product-specific edge cases, keyframe workflows, interpolation review, multi-stage QA, and versioned dataset delivery.

For broader retail AI programs, Annotera also supports retail data annotation across visual product recognition, catalog intelligence, checkout systems, inventory, merchandising, customer analytics, and other computer vision workflows.

Teams deciding whether polygon contours or another mask representation fit their model should first review Polygon vs. Segmentation: Choosing the Right Mask.

For annotation mechanics across moving objects more generally, Training AI to Recognize Irregular Shapes in Motion covers temporal consistency, keyframes, interpolation, and occlusion beyond the e-commerce use case.

Conclusion: Define the Product Boundary Before Scaling the Dataset

The difficult part of e-commerce product video segmentation is not drawing a polygon around a clear product on a white background.

The difficult part is deciding what happens when the product is transparent, reflective, deformable, partially hidden, moving quickly, touching another product, being worn by a person, or changing shape from one frame to the next.

Those situations require explicit ground-truth rules.

Define whether masks are visible or amodal. Decide how holes, shadows, reflections, transparency, hands, packaging, motion blur, instance identity, and product taxonomy are handled. Then test those rules on difficult video before scaling production.

When those conventions are consistent, product segmentation becomes much more than a set of frame-level outlines. It becomes reliable video ground truth that preserves product identity and geometry across real retail visual conditions.

Building a retail or e-commerce video dataset? Talk to Annotera about your product video segmentation requirements and design the mask rules, product taxonomy, temporal workflow, edge-case handling, and QA framework around your computer vision model.