Fabricjs: incorrect polygon display after point update and expansion/rotation

I have a polygon in fabric.js. After updating its points everything works fine regarding moving the polygon, but when I try to stretch/rotate it then the polygon shows incorrectly.

    const canvas = new Canvas('canvas');

    const points = [
      {
        x: 3,
        y: 4,
      },
      {
        x: 16,
        y: 3,
      },
      {
        x: 30,
        y: 5,
      },
      {
        x: 25,
        y: 55,
      },
      {
        x: 19,
        y: 44,
      },
      {
        x: 15,
        y: 30,
      },
      {
        x: 15,
        y: 55,
      },
      {
        x: 9,
        y: 55,
      },
      {
        x: 6,
        y: 53,
      },
      {
        x: -2,
        y: 55,
      },
      {
        x: -4,
        y: 40,
      },
      {
        x: 0,
        y: 20,
      },
    ];

    const poly = new Polygon(points, {
      left: 200,
      top: 50,

      fill: 'yellow',
      strokeWidth: 1,
      stroke: 'grey',

      objectCaching: false,
      transparentCorners: false,
      cornerColor: 'blue',
    });

    canvas.add(poly);

    poly.on('modified', () => {
      const points = RecalculatePointsInPolygon(poly);
 
      poly.set({ points });
 
     poly.setCoords();
     poly.setBoundingBox(true);
    });

function RecalculatePointsInPolygon(polygon: Polygon): Point[] {
  const matrix = polygon.calcTransformMatrix();

  return polygon
    .get('points')
    .map(
      (p: Point) =>
        new Point(p.x - polygon.pathOffset.x, p.y - polygon.pathOffset.y),
    )
    .map((p: Point) => util.transformPoint(p, matrix));
}

Before i resize i have correct shape:

but after resize (and rotate also):

It look like boundingBox is correctly implementent, but content inside is bigger. I also noted that points from RecalculatePointsInPolygon are counted corrected.

Please help i have no idea where i do mistake.

I expect correct shape after resize/rotate.

EDIT. I managed to make the polygon stretch correctly after calculating the points. Unfortunately I have no idea what to do with the rotation.

    poly.on('modified', () => {
      var width = poly.width;
      var height = poly.height;
      var scaleX = poly.scaleX;
      var scaleY = poly.scaleY;

      const points = RecalculatePointsInPolygon(poly);

      poly.set({
        points,

        width: width * scaleX,
        height: height * scaleY,
        scaleY: 1,
        scaleX: 1,
      });

      poly.setBoundingBox(true);
    });

With that code you can update points in polygon and modify this polygon without problem (except rotating).
So now after resize its look like should look:

But still there is a problem with rotation: