Reconstruction reads a silhouette. A background is a source of edges, shadows and colours that are not part of the object, and every one of them is a chance to get the shape wrong.
What a background does to a mesh
- False edges. A strong line behind the object — a table edge, a skirting board — can be read as part of the silhouette and reconstructed as geometry.
- Colour bleed. Ambient colour from the surroundings ends up in the texture. A product photographed on a warm wooden table comes out slightly orange all over.
- Contact shadows. The shadow where the object meets the surface reads as dark geometry, and often reconstructs as a skirt or a thickened base.
- Depth confusion. Anything in focus behind the object competes for the reconstructor's attention.
When to remove it
Almost always, for photographs. The exception is when the object is genuinely hard to separate — fine hair, fur, foliage, transparent material — where an aggressive cut-out removes parts of the object along with the backdrop. In those cases a careful manual mask beats an automatic one.
Renders are easier
If your reference is a render rather than a photograph, export it with a transparent background in the first place. That is a cleaner cut than any removal algorithm produces, because nothing is being inferred.
It matters for texturing too
A reference image driving a retexture has the same problem: the backdrop's colours bleed into the inferred material. Removing the background before using an image as a texture reference is worth doing by default.