Symfony access_token.token_handler triggers before access_control. What is the best way to skip it for public routes?

I use token_handler to validate token. But I don’t need it for PUBLIC_ACCESS routes. The thing is it triggers before access_control.

Can I check somehow for a PUBLIC_ACCESS first? Or is there a better way to make public routes avoid this token_handler? What is the best way? (Symfony 7)

# AccessTokenHandler.php

readonly class AccessTokenHandler implements AccessTokenHandlerInterface
{
    public function __construct(private ApiTokenRepository $repository)
    {
    }

    public function getUserBadgeFrom(string $accessToken): UserBadge
    {
        $accessToken = $this->repository->findOneByValue($accessToken);
        if (is_null($accessToken) || !$accessToken->isValid()) {
            throw new BadCredentialsException('Invalid credentials.');
        }

        return new UserBadge($accessToken->getUser()->getUserIdentifier());
    }
}

# security.yaml

firewalls:
    main:
        json_login:
            check_path: sign_in
            username_path: email
            password_path: password
        access_token:
            token_handler: AppSecurityAccessTokenHandler
access_control:
    - { path: ^/auth, roles: PUBLIC_ACCESS }
    - { path: ^/*, roles: ROLE_USER }