Someone care to check my math(s)? A 500GiB file, is 536871 MB. Assuming a 5200 RPM drive at 50 MB/s, is it really going take almost 3 hours to read that (assuming no other disk contention, sequential reads, etc)
@billinkc The unit conversions make sense but 50 MB/sec seems quite low for sequential throughput with a large block size, even for a single 5400 RPM drive.
@ypercube yeah I thought I would ask but it seems quite movie-plot and unlikely
@billinkc I think it's a valid point. You could get much worse (a few MB/sec) if it's purely random I/O with small block sizes on a small number of disks, though.
Is it common to get a 500 GB XML file as a data source to load?
You have two choices, you can basically read each row (as they're doing) and just assuming that it's good or maybe it reads until it closes the tag. The other is DOM parsing where it has to read the whole thing to make sense of it
I bet it's an export from a database and someone just said we'll use XML
