Preventing form submission with dynamic nested forms

I have a set up where users can add answer options to quizzes with nested forms. Each quiz has a quiz_type which I’m using together with stimulus controller to let the user have the option to either add a multiple choice or long answer question. The way I’ve set this up is that the relevant part of the form is as follows

<%= f.fields_for :quiz do |quiz_form| %>
    <%= quiz_form.text_field :problem_statement %>
    <%= quiz_form.select :quiz_type, options_for_select(Quiz.quiz_type_select, selected: quiz_form.object.quiz_type), { } %>
    <div data-dynamic-select-target="choice">
        <div data-nested-form-target="add_item"></div>
        <template data-nested-form-target="template">
            <%= quiz_form.fields_for :answer_options, AnswerOption.new, child_index: 'ANSWER_RECORD' do |answer_option_fields| %>
              <%= render 'answer_option_form', f: answer_option_fields %>
            <% end %>
        </template>
        
        <%= quiz_form.fields_for :answer_options do |answer_option_fields| %>
          <%= render 'answer_option_form', f: answer_option_fields %>
        <% end %>
    </div>
    <div data-dynamic-select-target="long">
    <%= quiz_form.fields_for :answer_options, AnswerOption.new do |answer_option| %>
      <%= answer_option.hidden_field :correct_answer, value: true %>
       <%= answer_option.text_field :answer_text %>
      <% end %>
    </div>
.
.
.
.

and the stimulus controller is as follows

import { Controller } from "@hotwired/stimulus";

export default class extends Controller {
  static targets = ["select", "choice", "long"];

  connect() {
    this.selected();
  }

  selected() {
    this.hideFields();
    switch (this.selectTarget.value) {
      case "single_choice":
        this.choiceTarget.classList.remove("hidden");
        break;
      case "long_answer":
        this.longTarget.classList.remove("hidden");
        break;
    }
  }

  hideFields() {
    this.choiceTarget.classList.add("hidden");
    this.longTarget.classList.add("hidden");
  }
}

now the problem is that when choosing either one of the single_choice or long_answer options the form will still submit the other one causing it to submit empty records. So for example if I choose to add the singe_choice questions I can add multiple choices from which a single one is correct, but as the form is currently structured the part

<div data-dynamic-select-target="long">
    <%= quiz_form.fields_for :answer_options, AnswerOption.new do |answer_option| %>
      <%= answer_option.hidden_field :correct_answer, value: true %>
       <%= answer_option.text_field :answer_text %>
    <% end %>
</div>

will also be submitted as it’s only hidden, but not actually removed. I also cannot just remove this since there could be a scenario where the user will switch between the quiz_types.

Any suggestions on what can I do to make this work? It would seem that I should somehow render the forms conditionally with the dynamic select choices.