last day (15 days later) » 

Feeds
Feeds
15:38
0
Q: Why is QGIS slow on an Azure VM compared to laptop?

Prashant KumarI have got an Azure VM setup which has QGIS 3.28.12-Firenze installed, the VM has 32 virtual processors(2.79GHz base speed) and 64 GB RAM. I have another QGIS 3.28.6-Firenze installed on my work laptop with 20 logical processors (2.4GHz base speed) and 32 GB RAM. I run the same python script on b...

qgis qgis-processing geoprocessing parallel-processing azure
Louis Cottereau
Louis Cottereau
Simple Python scripts run single-core (realpython.com/python-gil) unless you do explicit multiprocessing.
The reasons why the VM is slower could be multiple. It could just be that your specific VM was overloaded at the time, is just slower (because it can't boost its clock speed as much for example) or something else entirely.
Prashant Kumar
Prashant Kumar
@Kalak even if I use the QGIS python console to control QGIS and do the intersection on QGIS via python?
@Kalak I check VM's performance data when I ran the QGIS python script, the VM had no other process running and the CPU utilization was barely 25%
Vince
Vince
I/O-intensive operation are often much worse on a VM. You haven't adequately documented the problem if you don't include the I/O subsystem.
Louis Cottereau
Louis Cottereau
@PrashantKumar the intersection will not be run in parallel unless it is implemented to be that way. As far as I know, even the geoprocessing intersection isn't.
Borgh
Borgh
@Vince +1 for I/O problem. Azure might be efficient storage but it's not particularly fast.
Ian Turton
Ian Turton
15:38
also do you have the same indexes on both machines?
Prashant Kumar
Prashant Kumar
@IanTurton yes, I am using same shapefiles to calculate runtime
Louis Cottereau
Louis Cottereau
For a specific issue where I/O might be an issue using a shapefile is quite bad it takes a lot of disk space compared to geopackage in most cases. (besides being a pretty obsolete format)
Prashant Kumar
Prashant Kumar
@Kalak, sorry can you please elaborate I/O issue? I don't understand. The Azure VM has 50 GB storage (HDD space??) where I am storing the file shapefiles and the QGIS-python script is pointing to that path to load the file into QGIS for intersection
@Vince is I/O issue not a function of no. of processors on VM or processor speed? Knowing that the intersection process would be very intensive, I asked for a 32 process on the VM
Vince
Vince
No, I/O is completely independent of CPU. You can have 10,000 CPUs, but if your data is on a USB 1.0 thumb drive, you'll be waiting on data access until the cows come home.
Prashant Kumar
Prashant Kumar
@Vince is there a recommended way to measure I/O speed in that case? will copying 1 GB file and timing it give a rough measure of that?
Vince
Vince
15:38
This is wandering out of GIS into pure computer architecture, but both mean read time and mean write time are useful benchmarks. To emulate shapefile access you'd need three concurrent I/O channels, reading/writing .shp and .dbf by feature and the .shx in 100-400 byte chunks.
Ian Turton
Ian Turton
But depending on your shapefile, QGis may try to write a new better index (.qpx)
Prashant Kumar
Prashant Kumar
16:18
Okay, Thankyou @Vince. I will do a test and compare.
@IanTurton won't the same happen in both QGIS (VM and laptop) effectively affecting both runtime equally?
 
1 hour later…
Ian Turton
Ian Turton
17:21
only if QGIS has write permission to the data directory
 
3 hours later…
Mapperz
Mapperz
20:40
non-gis but latency on the connection try tracert to see how many hops its taking to the server and see how many milliseconds (smaller is better)

last day (15 days later) »