With separate canvases, the problem disappears, I'm 99% sure that rescaling is an inexpensive operation itself, but everything on a canvas was marked dirty and that operation took hundred times longer
maybe it's possible to avoid rebuilds altogether if I will rescale not svg, but the container with canvas
Working with Unity built-in components is like alchemy - you can get unexpected things when mixing them or not mixing