last day (28 days later) » 

9:33 PM
0
Q: Laggy slave OK to have only one system user in processlist?

names_are_easyIn all the examples of a processlist on a mysql slave I've seen, there are two system users. I have a slave that is very far behind the master. When I do show a processlist, there is only one system user (State: "Waiting for master to send event"), the event scheduler, and a trigger user. Is t...

 
Is Relay_Log_Space increasing each time you run SHOW SLAVE STATUS\G ?
 
Yes, Relay_Log_Space is increasing each time.
 
Please look at the Last Paragraph of my answer. Then, please answer all of the following questions: 1) Is Seconds_Behind_Master 0 ? 2) Are there two system user connections in the Processlist ? 3) Is Last Errno 0 ? 4) Does any text appear after Last Error ? 5) Did you run LOAD DATA INFILE on the Master ?
 
1)No it is more than 0; 2)No, only one; 3)Last_IO_Errno: 0; 4)No text; 5)I did not run LOAD DATA INFILE;
 
Please post your SHOW SLAVE STATUS\G and your SHOW PROCESSLIST; into the question.
Please post the SHOW PROCESSLIST; as well
Is Seconds_Behind_Master steadily increasing or is it stuck on 245653 ?
 
9:33 PM
It is most definately (and most unfortunately) increasing. Last count was: 247259
 
On the Master, go run SHOW BINARY LOGS;. Is master-bin.004515 still there ?
Another question: Is Exec_Master_Log_Pos moving ?
 
good question, yes binlog 004515 is still there.
Exec_Master_Log_Pos is changing.
 
Your replication is not in a normal state. Please join me in chat
 
Hi Rolando
So you think replication is working OK?
 
Not really. The SQL thread should be in the processlist;
Let's try to get it back in the PROCESSLIST;
 
9:42 PM
OK
 
Run STOP SLAVE; please
 
SHOW SLAVE STATUS\G
 
done
Slave_IO_Running: No
Slave_SQL_Running: No
 
What is Relay_Master_Log_File and Exec_Master_Log_Pos ?
 
9:44 PM
Relay_Master_Log_File: master-bin.004516
Exec_Master_Log_Pos: 123739411
 
CHANGE MASTER TO master_log_file='master-bin.004516',master_log_pos=123739411;
This will erase the 60GB of relay logs, so there might be a slight pause.
 
does it generate the older relay logs?
will I be able to recover from this if it does not work? I really don't want to start replication over
 
It will regenerate all those logs
Don't run it yet
 
I have not run it yet. this is customer facing data, so I have to be sure.
 
good
If replication is almost three days old, you are much better off reloading the Slave with Master data.
 
9:51 PM
That is a painful option in this case.
 
How many databases are you replicating from the Master ?
 
22
I am going to start slave again.
As I mentioned, we are running a lot of triggers and procedures. They seem to be running on one thread. So I suspect things are getting backed up.
 
Please run this query on the Master: SELECT IFNULL(B.engine,'Total') "Storage Engine",
CONCAT(LPAD(REPLACE(FORMAT(B.DSize/POWER(1024,pw),3),',',''),17,' '),' ',
SUBSTR(' KMGTP',pw+1,1),'B') "Data Size", CONCAT(LPAD(REPLACE(
FORMAT(B.ISize/POWER(1024,pw),3),',',''),17,' '),' ',
SUBSTR(' KMGTP',pw+1,1),'B') "Index Size", CONCAT(LPAD(REPLACE(
FORMAT(B.TSize/POWER(1024,pw),3),',',''),17,' '),' ',
SUBSTR(' KMGTP',pw+1,1),'B') "Table Size" FROM
(SELECT engine,SUM(data_length) DSize,SUM(index_length) ISize,
Please post output of that query
 
+----------------+----------------------+----------------------+----------------------+
| Storage Engine | Data Size | Index Size | Table Size |
+----------------+----------------------+----------------------+----------------------+
| MyISAM | 0.012 GB | 0.009 GB | 0.021 GB |
| InnoDB | 2352.374 GB | 1050.290 GB | 3402.664 GB |
| Total | 2352.386 GB | 1050.299 GB | 3402.685 GB |
Also I should add that I have seen the second system user pop in and out just lately.
show processlist;
+-------+-----------------+-------------------+-------+---------+--------+----------------------------------------+------------------+
| Id | User | Host | db | Command | Time | State | Info |
+-------+-----------------+-------------------+-------+---------+--------+----------------------------------------+------------------+
| 1 | event_scheduler | localhost | NULL | Daemon | 287 | Waiting for next activation | NULL |
 
???
2TB of data OUCH
 
10:00 PM
yeah
We do it with zfs
 
Do you have binary logging enable on the the Slave ?
 
Is the Slave also a Master ?
 
Not yet, but will be in the future
 
Please run SHOW BINARY LOGS on the Slave. How many are there ?
 
10:03 PM
too many. 1893 rows
 
Is each binary log about 1G ?
 
Average is about 200M
what are thinking about the binlogs?
 
You many have to disable binary logging on the Slave.
I had this problem on Friday with a client whose Slave I would reload and setup replication using a bash script. Replication would fall behind 1 sec every 2 seconds with the slave's binary logging enabled. When I disabled binary logging on that slave, the replication lag vanishes in like 2 min.
 
I'm willing to try it. The slave server isn't even done being built yet. And I should be able to turn it on later anyway, right?
 
yup
You could just do the following:
STOP SLAVE;
Goto OS
vi /etc/my.cnf
comment out the master-bin
#log-bin=master-bin
log back into MySQL
SET GLOBAL innodb_fast_shutdown = 0;
log out of mysql
service mysql restart
 
10:19 PM
OK
 
BTW do you have skip-slave-start in /etc/my.cnf on the Slave ?
 
OK good
Once mysql is restarted, run START SLAVE;
 
I can't do this right now, but I think I can try it tomorrow. Thanks for spending so much time on this.
 
your welcome. ping me in this chat room when you have done so
 
10:23 PM
I will
 
BTW you should zap all the binary logs on the Slave
I recapped our conversation in my answer
 

  last day (28 days later) »