The component is getting dragged when i try to select the text – React

<LabelCard
      identifier={identifier}
      type={LabelType.Default}
      defaultPosition={defaultPosition}
      $expandedOffset={expandedOffset}
      locked={locked}
      selectedToolTypeProp={selectedToolType}
      isExpanded={isExpanded}
      originalPosRef={originalPosRef}
      cardRef={cardRef}
    >
      <Content ref={cardRef}>
        <Title selected={isSelected}>{title}</Title>
        {!isMobile && <SubTitle selected={isSelected}>({subtitle})</SubTitle>}
        {isMobile && isExpandable && (
          <InfoButton onClick={onExpand}>
            <Icon name={Icons.Info} selected={isExpanded} />
          </InfoButton>
        )}
        {description.length > 0 && (
          <DescriptionContent
            ref={descriptionRef}
            expanded={isExpanded}
            expandedHeight={expandedHeight}
          >
            <Description>{description}</Description>
          </DescriptionContent>
        )}
      </Content>
</LabelCard>

This is my component where we render LabelCard inside which i have a hook responsible for dragging.
inside this hook i have mouseDown function:

function mouseDown(e: { clientX: number; clientY: number }) {
      if (!card) {
        return;
      }

      startX = e.clientX;
      startY = e.clientY;
      initialX = card.offsetLeft;
      initialY = card.offsetTop;

      document.addEventListener('mousemove', mouseMove);
      document.addEventListener('mouseup', mouseUp);
    }

I tried to check:

if (cardRef?.current && cardRef.current.contains(e.target as Node)) {
        return;
      }

This prevents dragging even when i click on some blank space. This blank space is typically part of the text itself like whitespace so maybe thats why i can not drag the label altogether even when i am not clicking on text.
Is there any solution to prevent dragging when i am selecting the text? while also being able to drag when i click on the blank space?
(I also tried to check if click was on TEXT NODE but it seems as it is not able to differentiate between text and space).