How to load sapui5 xml fragment in custom library that gets loaded from another app?

We have a custom library for controlls that we use across different sapui5 apps.
For js controls this works fine but now im trying to get it work with a xml fragment.

The library is deployed like an app and contains js files that extends sap/ui/base/Object and works fine.
We call it from other apps via

sap.ui.loader.config({
    paths: {
        'zfiorilibrary': sLibraryPath
    }
});

then we can access it from controller of the app by loading it as you load any other objects from sapui5 declaring it at the top in sap.ui.define()

now i tried the same with an xml file that is in the library and it wont load and gives me cors error in console.
this._oAttachmentsMangerDialog = sap.ui.xmlfragment("library-AttachmentsManager", "zfiorilibrary.fragments.AttachmentsManager", this);

jquery-dbg.js:9325 Access to XMLHttpRequest at 'https://ourserver/sap/bc/ui5_ui5/sap/zfiorilibrary/fragments/AttachmentsManager.fragment.xml' from origin 'http://localhost:8080' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.

when i click the url the xml loads fine

we never have cors errors when loading the js files.
What can i do to load xml files from the library?

We are using sapui5 version 1.71.x (always the newest 1.71 available)