Yup validation – How to validate an object of values only if another value is set?

I’m working on a register feature.
There’s a schema for the validation:

const invoiceAddressSchema = {
  title: string(),
  firstName: string(),
  familyName: string(),
  street: string().required(),
  postalCode: string().required(),
  city: string().required(),
  floor: string().required(),
  country: string().required(),
  phonePrefix: number().required(),
  phoneNumber: number().required(),
};

const schemaSecondStep = object({
  companyName: string().min(2).required(),
  title: string(),
  firstName: string(),
  familyName: string(),
  street: string().required(),
  postalCode: string().required(),
  city: string().required(),
  floor: string().required(),
  country: string().required(),
  deliveryInstruction: string().required(),
  phonePrefix: number().required(),
  phoneNumber: number().required(),
  subscribe: boolean(),
  invoiceAddress: object()
    .shape(invoiceAddressSchema)
    .nullable()
    .when([], {
      is: () => useDifferentInvoiceAddressSelection.value === "same",
      then: () => object().nullable().default(null),
      otherwise: () => object().shape(invoiceAddressSchema).required(),
    }),
});

The base object contains a delivery address. If useDifferentInvoiceAddressSelection equals to same, it means that the invoice address should be equal to the delivery address, and therefore invoiceAddress should be allowed to be “null”. If its not set to same, it should be an object with the values provided.

In the way I did it here, it doesn’t seem to be working correctly.
For example if the phoneNumber of invoiceAddress is set, but the phoneNumber on the delivery address is not set, both fields fail validation.

What is the correct way?