业务过程中出现数据库锁异常
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
2019-03-04 11:17:12.636 [XNIO-2 task-31] ERROR c.g.u.b.impl.UserInfoServiceImpl-[UserServiceInfoImpl.login] [登陆异常]:org.springframework.dao.CannotAcquireLockException:
### Error updating database. Cause: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Lock wait timeout exceeded; try restarting transaction
### The error may involve defaultParameterMap
### The error occurred while setting parameters
### SQL: update user set login_time = ? where user_name = ? or phone = ?
### Cause: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException:
; SQL []; Lock wait timeout exceeded; try restarting transaction; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException:
2019-03-04 11:17:12.637 [XNIO-2 task-31] ERROR c.g.u.facade.UserServiceFacade-[UserService.login]-[系统异常原因:null]
java.lang.NullPointerException: null
at com.gjmctp.usercenter.facade.UserServiceFacade.login(UserServiceFacade.java:113)
at sun.reflect.GeneratedMethodAccessor760.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2019-03-04 11:17:17.155 [XNIO-2 task-3] DEBUG c.g.u.mapper.UserMapper.findByParams-==> Preparing: select id, customer_no, name, gender,area_no, phone, cert_no, cert_type,login_num,login_time, user_name, password, mail, forum_id, operator, status, make_date, make_time, modify_date, modify_time, standby,uuid,ed_state,photo_md5,invited from user WHERE phone = ? and ed_state = ?
2019-03-04 11:17:17.155 [XNIO-2 task-3] DEBUG c.g.u.mapper.UserMapper.findByParams-==> Parameters: 13918910424(String), E(String)
2019-03-04 11:17:17.156 [XNIO-2 task-3] DEBUG c.g.u.mapper.UserMapper.findByParams-<== Total: 1
错误中,update 语句执行失败 ,但是 select语句可以成功
查询锁发现很锁等待的线程
1
2
3
4
5
6
7
8
9
10
select * from information_schema.innodb_trx;
select * from information_schema.INNODB_LOCKS;
select * from information_schema.INNODB_LOCK_WAITS;
select trx_mysql_thread_id from information_schema.innodb_trx it
JOIN information_schema.INNODB_LOCK_WAITS ilw
on ilw.blocking_trx_id = it.trx_id;
mysql中kill掉以后,仍然会出现新的锁等待超时。
发现数据库所在机器磁盘满
由于我们的数据库开启了binlog日志备份,数据占满了磁盘空间,导致无法更新
深入了解mysql的机制
磁盘空间满了之后MySQL会怎样? 官方说法:
MySQL本身并不会做任何操作,如官方文档说说,只会每分钟check一次是否有空闲空间,并且10分钟写一次错误日志。
- select不会受影响
- insert,update等更改数据或表结构等操作,经过不通的测试发现,当磁盘满了之后,并不是第一个insert就卡住,而是会在n个之后出现卡住的情况。
分析发现卡住现象和磁盘刷新有关
操作会被卡住,这主要是由于执行showslavestatus需要获得LOCK_active_mi锁,然后锁上mi->data_lock,但是由于磁盘满了无法将io_thread中的数据写入到relaylog中,导致io_thread持有mi->data_lock锁,这就导致了死锁。
解决办法
提高监控系统检测频率,预防再次发生;
及时删除不用的文件,释放空间;
若有线程因磁盘满的问题被阻塞了,可先杀掉,等到下一分钟重新检测时它可能又可以正常工作了;
可能因磁盘满导致某些线程被阻塞,引发其他线程也被阻塞,可把导致阻塞的线程杀掉,其他被阻塞的线程也就能继续工作了。