Material UI React JS Select component with multiple value unable to deselect when more than 1 item is selected

So i am trying to implement a multiselect dropdown in material ui with the following conditions
The dropdown will have [0 Apples, 1 Oranges, 2 Grapes]

By default 0 Apples should be selected
None of the options can be unselected.

If 0 Apples is selected and the user selects 1 Oranges or 2 Grapes then 0 Apples will be unselected and either 1 Orange or 2 Grapes will be selected depending on what he selected.

If 1 Oranges is selected then 0 Apples will get unselected.
If 2 Grapes Is selected then 0 Apples will get unselected.

If 1 Oranges is selected 2 Grapes is selected then both of them will get selected.
If 2 Grapes is selected 1 Oranges is selected then both the will get selected.

I am having problems trying to implement the unselect logic for the following cases:
1 Oranges and 2 Grapes is selected then unselect either of them.

And i am not sure where it is not working. The variable actualNewSelectionByUser is empty because he is unselecting so i will read from actualSelected as in value of mui input. I will use this to delete the value in state. But this is not working.

Here is the code sandbox link

link to code sandbox