VSCode node debugging looking in build/src for source maps

I’m trying to set up a node attach config in VSCode that allows me to debug a running node program, but it cannot find the sources for the js. The file structure is the following:

project
  | build
    | app.js
    | app.map.js
  | src
    | app.ts

My current launch config is this:

{
    "name": "Attach by Process ID",
    "processId": "${command:PickProcess}",
    "request": "attach",
    "skipFiles": [
        "<node_internals>/**"
    ],
    "type": "node"
}

My tsconfig is:

{
  "compilerOptions": {
    "target": "es2019",
    "lib": ["es2019", "esnext.asynciterable"],
    "typeRoots": ["./src/types", "./node_modules/@types"],
    "allowSyntheticDefaultImports": true,
    "experimentalDecorators": true,
    "emitDecoratorMetadata": true,
    "forceConsistentCasingInFileNames": true,
    "moduleResolution": "node",
    "module": "commonjs",
    "resolveJsonModule": true,
    "pretty": true,
    "sourceMap": true,
    "outDir": "./build",
    "sourceRoot": "./src",
    "allowJs": true,
    "noEmit": false,
    "esModuleInterop": true,
    "skipLibCheck": true,
    "paths": {}
  },
  "include": ["./src/**/*"],
  "exclude": ["node_modules", "tests"]
}

When I debug with --inspect-brk specified, it shows that it’s looking for /build/src/app.ts.' Is there a way to fix this? I have tried all the sourceMap related props in VSCodes config, but honestly I can't seem to get them to have any effect. The one thing that works is setting sourceRoot` to be the absolute path to my project source, but this is on a shared project, so id rather not have a solution that causes these files to be local.