How to use either this type or another based on props only? React Typescript Discriminating Unions

I have a componenet:

type Base = {
    color: string
}

type Button = {
    to: string
} & Base

type Link = {
    link: string
    linkNewTab: boolean
} & Base

type ComponentProps = Button | Link 

export const Button: React.FC<ComponentProps> = (props) => {
    return (<>
        {props.link ? <Link newTab={props.linkNewTab}>...</Link> : props.to && <Button>...</Button>}
    </>)
}

• There is a base type with props which both types have in common.
• The Component should either has Button or Link type based on the given props. But if it has Link, these props are preferred.

Typescript Error:

Property 'link' does not exist on type 'PropsWithChildren<ComponentProps>'.
  Property 'link' does not exist on type '{ to: string; } & Base & { children?: ReactNode; }'.ts(2339)

What I don’t want:

• I can solve the problem by adding a type to the Base type and decide from their which props are allowed. I would like to automatically decide that based on props.

Information: 4.5.4 TypeScript Version