last day (15 days later) » 

1:14 PM
-1
Q: Importing a very large (160gb) sql dump file into mysql workbench

GabrielusI'm trying to import the following dataset: https://steam.internet.byu.edu/ After extracting the file , the total file size is roughly 160gb. When I try to run the sql file, I first run out of RAM, then I start running out of storage space (pagefile on C:) . Thus I'm trying to split up the file ...

 
So don't open it in mysql workbench but rather directly import it?
 
@Seth directly importing the structure is taking ages (let alone the data itself). It's been 3 hours and the progress bar hasn't moved.
 
So you don't intend to do any actual work with this? I don't really get what your goal is then. Having it in a DB to do actual queries and what not seems to be rather beneficial in comparison to just opening it MySQL workbench. Did you try to use a regular file splitter?
 
My goal is to import it. I'm a bit new to this. I thought that in order to import the data into the server, I had to use MySql workbench>server>data import. Seeing as there is no progress I was hoping you could suggest a utility to split the file
 
How long did you let it run? How beefy is the machine your SQL server is running on?
 
1:14 PM
Ryzen 1700, 16gb ram, 512gb NVME SSD. I left it running for 3 hours on the "Dump structure" setting and the progress bar hasn't moved. The CPU, RAM and Storage aren't being fully utalised so there doesn't seem to be a hardware bottleneck (less than 10% utilization)
 
Just noticed but isn't feature supposed to be used to import a live database? So you already did import the dataset to live mysql server?
 
The data I'm trying to import is a dump from a MariaDB database from what I can understand
It doesn't seem to ask whether it's live or not, I can simply point it to my file using the "Import from self-contained File" setting
 
Yes I just checked the documentation dev.mysql.com/doc/workbench/en/… it's really just to import into mysql workbench
What I'm still confused about is why you're trying to do that (I'm not really using MySQL workbench). Why not import it to a mysql server instance in order to have data available and be queryable? See for example How to import using cmd.
 
I didn't realise they were seperate - I thought mysql workbench was just a GUI for the server
My most sincere apologies
 
The table structure seems to be rather simple (the overview seems to have all the information).
It's fine.
I haven't used it in ages, it might be less apparent than it was back then.
With a 160 GB dump it's still going to take quite a bit of time to import it.
 
1:21 PM
The workbench seems to be doing a similar command in the log
Running: mysql.exe --defaults-file="c:\users\gabri\appdata\local\temp\tmpjqfdry.cnf" --protocol=tcp --host=localhost --user=root --port=3306 --default-character-set=utf8 --comments --database=steam < "D:\\steam.sql"
And in the SO page:
mysql -u username -p database_name < file.sql
I'll give it a shot anyways
 
The mentioned question has a bit about turning off autocommit beforehand If that is on, it could be that the DB commits for every single instruction in that dump file.
 
That would make sense since the disk usage seems to be jumping up and down
 
So give it a shot. ;D
 
The command I just tried is
SET autocommit=0 ; source D:\\steam.sql ; COMMIT ;
CPU usage is 9% and disk is basically 0
anything else I can do to speed it up?
it's importing 10788 records at a time
It seems to be using one core only for some reason
 
You could try to check this question. Could be quiet dated though. From just reading it, -max_allowed_packet could be worth a shot.
If you want to "hard split" the files you might try the splitter from Total Commander or just a simple PowerShell script or similar.
 
1:36 PM
What would a safe value for max_allowed_packet be for my config?
he set it to 1073741824
but I'm not sure whether its a bit excessive
 
dev.mysql.com/doc/refman/5.7/en/… but it looks like it might actually not be that helpful.
This could help you to get a better understanding and some more parameters to fiddle with.
 
Thanks :)
 
No problem, as you can see I'm not entirely sure either. But it's part of what I'd try.
If you're wondering why your question gets downvotes, it's likely because it's somewhat muddy and doesn't show a lot of the things you might have tried or what the actual error is. Currently, if you carefully read it, it doesn't actually contain a question.
 
1:56 PM
I see
I will reword it then
Reworded now
Thanks for the help :)
still trying the commands you last sent
 

last day (15 days later) »