Did I break Mysql? Cannot login

Before posting something, READ the changelog, WATCH the videos, howto and provide following:
Your install is: Bare metal, ESXi, what CPU model, RAM, HD, what EVE version you have, output of the uname -a and any other info that might help us faster.

Moderator: mike

Post Reply
2600
Posts: 36
Joined: Mon Mar 20, 2017 7:45 pm

Did I break Mysql? Cannot login

Post by 2600 » Thu Aug 31, 2017 2:37 am

Basics are that I am on 2.0.3-77 on a bare-metal server that has been working just fine. I updated 1 process using apt-get and I found I couldn't login. I checked apache2, and that was running, and it seems from journalctl -xe, that the issue is mysql service will not restart now. I will paste the output below, but as of now, mysql is exiting out due to an "invalid argument" and is denied.

So, I don't know if this is a bug, if I have to/or how to roll back, or, if I have to reinstall.

Code: Select all

root@eve-ng:~# systemctl status apache2
● apache2.service - LSB: Apache2 web server
   Loaded: loaded (/etc/init.d/apache2; bad; vendor preset: enabled)
  Drop-In: /lib/systemd/system/apache2.service.d
           └─apache2-systemd.conf
   Active: active (running) since Wed 2017-08-30 22:06:18 EDT; 7min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 3675 ExecStart=/etc/init.d/apache2 start (code=exited, status=0/SUCCESS)
    Tasks: 7
   Memory: 31.7M
      CPU: 348ms
   CGroup: /system.slice/apache2.service
           ├─3968 /usr/sbin/apache2 -k start
           ├─4038 /usr/sbin/apache2 -k start
           ├─4039 /usr/sbin/apache2 -k start
           ├─4041 /usr/sbin/apache2 -k start
           ├─4042 /usr/sbin/apache2 -k start
           ├─4043 /usr/sbin/apache2 -k start
           └─5824 /usr/sbin/apache2 -k start

Aug 30 22:06:09 eve-ng systemd[1]: Starting LSB: Apache2 web server...
Aug 30 22:06:10 eve-ng apache2[3675]:  * Starting Apache httpd web server apache2
Aug 30 22:06:18 eve-ng apache2[3675]:  *
Aug 30 22:06:18 eve-ng systemd[1]: Started LSB: Apache2 web server.
root@eve-ng:~# systemctl restart apache2
root@eve-ng:~# journalctl -xe
--
-- The result is failed.
Aug 30 22:14:50 eve-ng systemd[1]: mysql.service: Unit entered failed state.
Aug 30 22:14:50 eve-ng systemd[1]: mysql.service: Failed with result 'exit-code'.
Aug 30 22:14:50 eve-ng systemd[1]: mysql.service: Service hold-off time over, scheduling restart.
Aug 30 22:14:50 eve-ng systemd[1]: Stopped MySQL Community Server.
-- Subject: Unit mysql.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mysql.service has finished shutting down.
Aug 30 22:14:50 eve-ng systemd[1]: Starting MySQL Community Server...
-- Subject: Unit mysql.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mysql.service has begun starting up.
Aug 30 22:14:50 eve-ng audit[8309]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/pro
Aug 30 22:14:50 eve-ng audit[8309]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys
Aug 30 22:14:50 eve-ng audit[8309]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/pro
Aug 30 22:14:50 eve-ng kernel: audit: type=1400 audit(1504145690.653:69): apparmor="DENIED" operation="open" pro
Aug 30 22:14:50 eve-ng kernel: audit: type=1400 audit(1504145690.653:70): apparmor="DENIED" operation="open" pro
Aug 30 22:14:50 eve-ng kernel: audit: type=1400 audit(1504145690.653:71): apparmor="DENIED" operation="open" pro
Aug 30 22:14:51 eve-ng systemd[1]: mysql.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
lines 2487-2510/2510 (END)
--
-- The result is failed.
Aug 30 22:14:50 eve-ng systemd[1]: mysql.service: Unit entered failed state.
Aug 30 22:14:50 eve-ng systemd[1]: mysql.service: Failed with result 'exit-code'.
Aug 30 22:14:50 eve-ng systemd[1]: mysql.service: Service hold-off time over, scheduling restart.
Aug 30 22:14:50 eve-ng systemd[1]: Stopped MySQL Community Server.
-- Subject: Unit mysql.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mysql.service has finished shutting down.
Aug 30 22:14:50 eve-ng systemd[1]: Starting MySQL Community Server...
-- Subject: Unit mysql.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mysql.service has begun starting up.
Aug 30 22:14:50 eve-ng audit[8309]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc
Aug 30 22:14:50 eve-ng audit[8309]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/
Aug 30 22:14:50 eve-ng audit[8309]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc
Aug 30 22:14:50 eve-ng kernel: audit: type=1400 audit(1504145690.653:69): apparmor="DENIED" operation="open" prof
Aug 30 22:14:50 eve-ng kernel: audit: type=1400 audit(1504145690.653:70): apparmor="DENIED" operation="open" prof
Aug 30 22:14:50 eve-ng kernel: audit: type=1400 audit(1504145690.653:71): apparmor="DENIED" operation="open" prof
Aug 30 22:14:51 eve-ng systemd[1]: mysql.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
lines 2487-2510/2510 (END)
--
-- The result is failed.
Aug 30 22:14:50 eve-ng systemd[1]: mysql.service: Unit entered failed state.
Aug 30 22:14:50 eve-ng systemd[1]: mysql.service: Failed with result 'exit-code'.
Aug 30 22:14:50 eve-ng systemd[1]: mysql.service: Service hold-off time over, scheduling restart.
Aug 30 22:14:50 eve-ng systemd[1]: Stopped MySQL Community Server.
-- Subject: Unit mysql.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mysql.service has finished shutting down.
Aug 30 22:14:50 eve-ng systemd[1]: Starting MySQL Community Server...
-- Subject: Unit mysql.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mysql.service has begun starting up.
Aug 30 22:14:50 eve-ng audit[8309]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/8309/status" pid=8309 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=111 ouid=111
Aug 30 22:14:50 eve-ng audit[8309]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=8309 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=111 oui
Aug 30 22:14:50 eve-ng audit[8309]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/8309/status" pid=8309 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=111 ouid=111
Aug 30 22:14:50 eve-ng kernel: audit: type=1400 audit(1504145690.653:69): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/8309/status" pid=8309 comm="mysqld" requested_mask="r" den
Aug 30 22:14:50 eve-ng kernel: audit: type=1400 audit(1504145690.653:70): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=8309 comm="mysqld" requested_mask
Aug 30 22:14:50 eve-ng kernel: audit: type=1400 audit(1504145690.653:71): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/8309/status" pid=8309 comm="mysqld" requested_mask="r" den
Aug 30 22:14:51 eve-ng systemd[1]: mysql.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
~
~
~
lines 2487-2510/2510 (END)
--
-- The result is failed.
Aug 30 22:14:50 eve-ng systemd[1]: mysql.service: Unit entered failed state.
Aug 30 22:14:50 eve-ng systemd[1]: mysql.service: Failed with result 'exit-code'.
Aug 30 22:14:50 eve-ng systemd[1]: mysql.service: Service hold-off time over, scheduling restart.
Aug 30 22:14:50 eve-ng systemd[1]: Stopped MySQL Community Server.
-- Subject: Unit mysql.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mysql.service has finished shutting down.
Aug 30 22:14:50 eve-ng systemd[1]: Starting MySQL Community Server...
-- Subject: Unit mysql.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mysql.service has begun starting up.
Aug 30 22:14:50 eve-ng audit[8309]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/8309/status" pid=8309 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=111 ouid=111
Aug 30 22:14:50 eve-ng audit[8309]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=8309 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=111 ouid=0
Aug 30 22:14:50 eve-ng audit[8309]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/8309/status" pid=8309 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=111 ouid=111
Aug 30 22:14:50 eve-ng kernel: audit: type=1400 audit(1504145690.653:69): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/8309/status" pid=8309 comm="mysqld" requested_mask="r" denied_ma
Aug 30 22:14:50 eve-ng kernel: audit: type=1400 audit(1504145690.653:70): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=8309 comm="mysqld" requested_mask="r" d
Aug 30 22:14:50 eve-ng kernel: audit: type=1400 audit(1504145690.653:71): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/8309/status" pid=8309 comm="mysqld" requested_mask="r" denied_ma
Aug 30 22:14:51 eve-ng systemd[1]: mysql.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
~
~
~
lines 2487-2510/2510 (END)
--
-- The result is failed.
Aug 30 22:14:50 eve-ng systemd[1]: mysql.service: Unit entered failed state.
Aug 30 22:14:50 eve-ng systemd[1]: mysql.service: Failed with result 'exit-code'.
Aug 30 22:14:50 eve-ng systemd[1]: mysql.service: Service hold-off time over, scheduling restart.
Aug 30 22:14:50 eve-ng systemd[1]: Stopped MySQL Community Server.
-- Subject: Unit mysql.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mysql.service has finished shutting down.
Aug 30 22:14:50 eve-ng systemd[1]: Starting MySQL Community Server...
-- Subject: Unit mysql.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mysql.service has begun starting up.
Aug 30 22:14:50 eve-ng audit[8309]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/8309/status" pid=8309 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=111 ouid=111
Aug 30 22:14:50 eve-ng audit[8309]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=8309 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=111 ouid=0
Aug 30 22:14:50 eve-ng audit[8309]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/8309/status" pid=8309 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=111 ouid=111
Aug 30 22:14:50 eve-ng kernel: audit: type=1400 audit(1504145690.653:69): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/8309/status" pid=8309 comm="mysqld" requested_mask="r" denied_mas
Aug 30 22:14:50 eve-ng kernel: audit: type=1400 audit(1504145690.653:70): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=8309 comm="mysqld" requested_mask="r" de
Aug 30 22:14:50 eve-ng kernel: audit: type=1400 audit(1504145690.653:71): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/8309/status" pid=8309 comm="mysqld" requested_mask="r" denied_mas
Aug 30 22:14:51 eve-ng systemd[1]: mysql.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
~
~
lines 2487-2510/2510 (END)
--
-- The result is failed.
Aug 30 22:14:50 eve-ng systemd[1]: mysql.service: Unit entered failed state.
Aug 30 22:14:50 eve-ng systemd[1]: mysql.service: Failed with result 'exit-code'.
Aug 30 22:14:50 eve-ng systemd[1]: mysql.service: Service hold-off time over, scheduling restart.
Aug 30 22:14:50 eve-ng systemd[1]: Stopped MySQL Community Server.
-- Subject: Unit mysql.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mysql.service has finished shutting down.
Aug 30 22:14:50 eve-ng systemd[1]: Starting MySQL Community Server...
-- Subject: Unit mysql.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mysql.service has begun starting up.
Aug 30 22:14:50 eve-ng audit[8309]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/8309/status" pid=8309 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=111 ouid=111
Aug 30 22:14:50 eve-ng audit[8309]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=8309 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=111 ouid=0
Aug 30 22:14:50 eve-ng audit[8309]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/8309/status" pid=8309 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=111 ouid=111
Aug 30 22:14:50 eve-ng kernel: audit: type=1400 audit(1504145690.653:69): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/8309/status" pid=8309 comm="mysqld" requested_mask="r" denied_mas
Aug 30 22:14:50 eve-ng kernel: audit: type=1400 audit(1504145690.653:70): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=8309 comm="mysqld" requested_mask="r" de
Aug 30 22:14:50 eve-ng kernel: audit: type=1400 audit(1504145690.653:71): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/8309/status" pid=8309 comm="mysqld" requested_mask="r" denied_mas
Aug 30 22:14:51 eve-ng systemd[1]: mysql.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
~
~
~
lines 2487-2510/2510 (END)
--
-- The result is failed.
Aug 30 22:14:50 eve-ng systemd[1]: mysql.service: Unit entered failed state.
Aug 30 22:14:50 eve-ng systemd[1]: mysql.service: Failed with result 'exit-code'.
Aug 30 22:14:50 eve-ng systemd[1]: mysql.service: Service hold-off time over, scheduling restart.
Aug 30 22:14:50 eve-ng systemd[1]: Stopped MySQL Community Server.
-- Subject: Unit mysql.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mysql.service has finished shutting down.
Aug 30 22:14:50 eve-ng systemd[1]: Starting MySQL Community Server...
-- Subject: Unit mysql.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mysql.service has begun starting up.
Aug 30 22:14:50 eve-ng audit[8309]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/8309/status" pid=8309 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=111 ouid=111
Aug 30 22:14:50 eve-ng audit[8309]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=8309 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=111 ouid=0
Aug 30 22:14:50 eve-ng audit[8309]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/8309/status" pid=8309 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=111 ouid=111
Aug 30 22:14:50 eve-ng kernel: audit: type=1400 audit(1504145690.653:69): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/8309/status" pid=8309 comm="mysqld" requested_mask="r" denied_mas
Aug 30 22:14:50 eve-ng kernel: audit: type=1400 audit(1504145690.653:70): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=8309 comm="mysqld" requested_mask="r" de
Aug 30 22:14:50 eve-ng kernel: audit: type=1400 audit(1504145690.653:71): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/8309/status" pid=8309 comm="mysqld" requested_mask="r" denied_mas
Aug 30 22:14:51 eve-ng systemd[1]: mysql.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
~

One last thing, I checked ownership of the database directory, below, and it looks ok.

Code: Select all

root@eve-ng:~# ls -lt /var/lib/mysql
total 176168
-rw-r----- 1 mysql mysql 79691776 Aug 30 21:56 ibdata1
-rw-r----- 1 mysql mysql 50331648 Aug 30 21:53 ib_logfile0
-rw-r----- 1 mysql mysql 50331648 Aug 30 21:53 ib_logfile1
-rw-r----- 1 mysql mysql      803 Aug 30 21:53 ib_buffer_pool
drwxr-x--- 2 mysql mysql     4096 Jul 23 21:18 guacdb
-rw-r--r-- 1 mysql mysql        6 Jul 21 14:52 mysql_upgrade_info
drwxr-x--- 2 mysql mysql     4096 Jul 21 14:52 mysql
drwxr-x--- 2 mysql mysql     4096 Jul 21 14:52 performance_schema
-rw-r--r-- 1 mysql mysql        0 Jul 21 14:52 debian-5.7.flag
drwxr-x--- 2 mysql mysql     4096 Feb  9  2017 eve_ng_db
drwxr-x--- 2 mysql mysql    12288 Feb  9  2017 sys
-rw-r----- 1 mysql mysql       56 Feb  9  2017 auto.cnf
Top
2600
Posts: 36
Joined: Mon Mar 20, 2017 7:45 pm

Re: Did I break Mysql? Cannot login

Post by 2600 » Thu Aug 31, 2017 11:56 am

Here are the other syslogs. Any ideas?

Code: Select all

root@eve-ng:~# vi /var/log/syslog
Aug 31 06:25:09 eve-ng systemd[1]: Failed to start MySQL Community Server.
Aug 31 06:25:09 eve-ng systemd[1]: mysql.service: Unit entered failed state.
Aug 31 06:25:09 eve-ng systemd[1]: mysql.service: Failed with result 'exit-code'.
Aug 31 06:25:09 eve-ng systemd[1]: mysql.service: Service hold-off time over, scheduling restart.
Aug 31 06:25:09 eve-ng systemd[1]: Stopped MySQL Community Server.
Aug 31 06:25:09 eve-ng systemd[1]: Starting MySQL Community Server...
Aug 31 06:25:09 eve-ng kernel: [28789.458107] audit: type=1400 audit(1504175109.375:2847): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/16993/status" pid=16993 comm="mysqld" requested
_mask="r" denied_mask="r" fsuid=111 ouid=111
Aug 31 06:25:09 eve-ng kernel: [28789.458151] audit: type=1400 audit(1504175109.375:2848): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=16993 comm="mysqld" re
quested_mask="r" denied_mask="r" fsuid=111 ouid=0
Aug 31 06:25:09 eve-ng kernel: [28789.458217] audit: type=1400 audit(1504175109.375:2849): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/16993/status" pid=16993 comm="mysqld" requested
_mask="r" denied_mask="r" fsuid=111 ouid=111
Aug 31 06:25:09 eve-ng systemd[1]: mysql.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Top
2600
Posts: 36
Joined: Mon Mar 20, 2017 7:45 pm

Re: Did I break Mysql? Cannot login

Post by 2600 » Thu Aug 31, 2017 12:47 pm

When I look in the /var/log/mysql/error.log, I get this:

Code: Select all

InnoDB: End of page dump
2017-08-31T10:25:09.728127Z 0 [Note] InnoDB: Uncompressed page, stored checksum in field1 4183208079, calculated checksums for field1: crc32 4183208079/1261802258, innodb 2664128050, none 3735928559, stored check
sum in field2 759701967, calculated checksums for field2: crc32 4183208079/1261802258, innodb 3452705975, none 3735928559,  page LSN 0 167597959, low 4 bytes of LSN at page end 167461745, page number (if stored t
o page already) 49, space id (if created with >= MySQL-4.1.1 and stored already) 0
InnoDB: Page may be a system page
2017-08-31T10:25:09.728144Z 0 [ERROR] [FATAL] InnoDB: The page in the doublewrite buffer is corrupt. Cannot continue operation. You can try to recover the database with innodb_force_recovery=6
2017-08-31 06:25:09 0x7f4a7987a740  InnoDB: Assertion failure in thread 139957843240768 in file ut0ut.cc line 916
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
10:25:09 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.

key_buffer_size=16777216
read_buffer_size=131072
max_used_connections=0
max_threads=151
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 76382 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x3b)[0xe8950b]
/usr/sbin/mysqld(handle_fatal_signal+0x489)[0x785f09]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f4a783d2390]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x38)[0x7f4a7778b428]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7f4a7778d02a]
/usr/sbin/mysqld[0x75c0d2]
/usr/sbin/mysqld(_ZN2ib5fatalD1Ev+0x145)[0x11149a5]
/usr/sbin/mysqld(_Z17buf_dblwr_processv+0x1152)[0x115de32]
/usr/sbin/mysqld(_Z35recv_recovery_from_checkpoint_startm+0x2536)[0xfe56c6]
/usr/sbin/mysqld(_Z34innobase_start_or_create_for_mysqlv+0x38cb)[0x10bc34b]
/usr/sbin/mysqld[0xf7e179]
/usr/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x51)[0x7d3e51]
/usr/sbin/mysqld[0xc6c4c5]
/usr/sbin/mysqld(_Z11plugin_initPiPPci+0x608)[0xc73cd8]
/usr/sbin/mysqld[0x77e27a]
/usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x816)[0x77f9c6]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7f4a77776830]
/usr/sbin/mysqld(_start+0x29)[0x7762b9]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
2017-08-31T10:25:39.883884Z 0 [Warning] Changed limits: max_open_files: 1024 (requested 5000)
2017-08-31T10:25:39.883961Z 0 [Warning] Changed limits: table_open_cache: 431 (requested 2000)
2017-08-31T10:25:40.042146Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2017-08-31T10:25:40.043209Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.19-0ubuntu0.16.04.1) starting as process 22367 ...
2017-08-31T10:25:40.047602Z 0 [Note] InnoDB: PUNCH HOLE support available
2017-08-31T10:25:40.047642Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2017-08-31T10:25:40.047649Z 0 [Note] InnoDB: Uses event mutexes
2017-08-31T10:25:40.047655Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2017-08-31T10:25:40.047661Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.8
2017-08-31T10:25:40.047667Z 0 [Note] InnoDB: Using Linux native AIO
2017-08-31T10:25:40.047944Z 0 [Note] InnoDB: Number of pools: 1
:$                                                                                                                                                                                                
InnoDB: End of page dump
2017-08-31T12:42:03.838273Z 0 [Note] InnoDB: Uncompressed page, stored checksum in field1 4183208079, calculated checksums for field1: crc32 4183208079/1261802258, innodb 2664128050, none 3735928559, stored check
sum in field2 759701967, calculated checksums for field2: crc32 4183208079/1261802258, innodb 3452705975, none 3735928559,  page LSN 0 167597959, low 4 bytes of LSN at page end 167461745, page number (if stored t
o page already) 49, space id (if created with >= MySQL-4.1.1 and stored already) 0
InnoDB: Page may be a system page
2017-08-31T12:42:03.838290Z 0 [ERROR] [FATAL] InnoDB: The page in the doublewrite buffer is corrupt. Cannot continue operation. You can try to recover the database with innodb_force_recovery=6
2017-08-31 08:42:03 0x7fa027585740  InnoDB: Assertion failure in thread 140325831595840 in file ut0ut.cc line 916
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
12:42:03 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.

key_buffer_size=16777216
read_buffer_size=131072
max_used_connections=0
max_threads=151
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 76382 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x3b)[0xe8950b]
/usr/sbin/mysqld(handle_fatal_signal+0x489)[0x785f09]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7fa0260dd390]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x38)[0x7fa025496428]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7fa02549802a]
/usr/sbin/mysqld[0x75c0d2]
/usr/sbin/mysqld(_ZN2ib5fatalD1Ev+0x145)[0x11149a5]
/usr/sbin/mysqld(_Z17buf_dblwr_processv+0x1152)[0x115de32]
/usr/sbin/mysqld(_Z35recv_recovery_from_checkpoint_startm+0x2536)[0xfe56c6]
/usr/sbin/mysqld(_Z34innobase_start_or_create_for_mysqlv+0x38cb)[0x10bc34b]
innodb_force_recovery = 4
/usr/sbin/mysqld[0xf7e179]
/usr/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x51)[0x7d3e51]
/usr/sbin/mysqld[0xc6c4c5]
/usr/sbin/mysqld(_Z11plugin_initPiPPci+0x608)[0xc73cd8]
/usr/sbin/mysqld[0x77e27a]
/usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x816)[0x77f9c6]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7fa025481830]
/usr/sbin/mysqld(_start+0x29)[0x7762b9]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Top
Chris929
Posts: 83
Joined: Tue Jun 27, 2017 8:51 am

Re: Did I break Mysql? Cannot login

Post by Chris929 » Thu Aug 31, 2017 1:52 pm

Restore your Backup (that you hopefully made as always before upgrading) and it should be fine ;)
Top
2600
Posts: 36
Joined: Mon Mar 20, 2017 7:45 pm

Re: Did I break Mysql? Cannot login

Post by 2600 » Thu Aug 31, 2017 2:45 pm

Oh yeah, that's a great idea, but I did the upgrade long before the corruption. No, no backup.

At this point I have backups of files (below), which I presume I can import into a new ESXi VM.

Code: Select all

-rw-r--r--  1 root     root     33118042520 Aug 31 09:33 addons.tar.gz
-rw-r--r--  1 root     root      8111138507 Aug 31 08:41 tmp.tar.gz
-rw-r--r--  1 root     root        25430969 Aug 31 07:57 labs.tar.gz
Top
2600
Posts: 36
Joined: Mon Mar 20, 2017 7:45 pm

Re: Did I break Mysql? Cannot login

Post by 2600 » Tue Sep 05, 2017 4:10 pm

So, one thing I do know is that apparmor is part of the problem. From my logs, you can see "DENIED" from apparmor.

Further, when I kill apparmor, see below.

I see here there is a bug fix, but it doesn't work for this issue still.: https://bugs.launchpad.net/ubuntu/+sour ... ug/1610765

Code: Select all

Sep 05 12:01:31 eve-ng systemd[1]: Failed to start MySQL Community Server.
-- Subject: Unit mysql.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit mysql.service has failed.
-- 
-- The result is failed.
Sep 05 12:01:31 eve-ng systemd[1]: mysql.service: Unit entered failed state.
Sep 05 12:01:31 eve-ng systemd[1]: mysql.service: Failed with result 'exit-code'.
Sep 05 12:01:31 eve-ng systemd[1]: mysql.service: Service hold-off time over, scheduling restart.
Sep 05 12:01:31 eve-ng systemd[1]: Stopped MySQL Community Server.
-- Subject: Unit mysql.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit mysql.service has finished shutting down.
Sep 05 12:01:31 eve-ng systemd[1]: Starting MySQL Community Server...
-- Subject: Unit mysql.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit mysql.service has begun starting up.
Sep 05 12:01:31 eve-ng audit[21027]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/21027/status" pid=21027 comm="mysqld" requested_mask="r" denied_m
Sep 05 12:01:31 eve-ng audit[21027]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=21027 comm="mysqld" requested_mask="r" d
Sep 05 12:01:31 eve-ng audit[21027]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/21027/status" pid=21027 comm="mysqld" requested_mask="r" denied_m
Sep 05 12:01:31 eve-ng kernel: audit: type=1400 audit(1504627291.382:47326): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/21027/status" pid=21027 comm=
Sep 05 12:01:31 eve-ng kernel: audit: type=1400 audit(1504627291.382:47327): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=2102
Sep 05 12:01:31 eve-ng kernel: audit: type=1400 audit(1504627291.382:47328): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/21027/status" pid=21027 comm=
Sep 05 12:01:31 eve-ng systemd[1]: mysql.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
root@eve-ng:~# systemctl stop apparmor.service
root@eve-ng:~# systemctl start mysql.service  
Job for mysql.service failed because the control process exited with error code. See "systemctl status mysql.service" and "journalctl -xe" for details.
root@eve-ng:~# journalctl -xe
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- Documentation: http://www.freedesktop.org/wiki/Software/systemd/multiseat
-- 
-- A session with the ID 432 has been terminated.
Sep 05 12:07:12 eve-ng systemd[1]: Stopping LSB: AppArmor initialization...
-- Subject: Unit apparmor.service has begun shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit apparmor.service has begun shutting down.
Sep 05 12:07:12 eve-ng apparmor[23233]:  * Clearing AppArmor profiles cache
Sep 05 12:07:12 eve-ng apparmor[23233]:    ...done.
Sep 05 12:07:12 eve-ng apparmor[23233]: All profile caches have been cleared, but no profiles have been unloaded.
Sep 05 12:07:12 eve-ng apparmor[23233]: Unloading profiles will leave already running processes permanently
Sep 05 12:07:12 eve-ng apparmor[23233]: unconfined, which can lead to unexpected situations.
Sep 05 12:07:12 eve-ng apparmor[23233]: To set a process to complain mode, use the command line tool
Sep 05 12:07:12 eve-ng apparmor[23233]: 'aa-complain'. To really tear down all profiles, run the init script
Sep 05 12:07:12 eve-ng apparmor[23233]: with the 'teardown' option."
Sep 05 12:07:12 eve-ng systemd[1]: Stopped LSB: AppArmor initialization.
-- Subject: Unit apparmor.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit apparmor.service has finished shutting down.
Sep 05 12:07:20 eve-ng systemd[1]: Failed to start MySQL Community Server.
Top
2600
Posts: 36
Joined: Mon Mar 20, 2017 7:45 pm

Re: Did I break Mysql? Cannot login

Post by 2600 » Tue Sep 05, 2017 4:24 pm

My question is: is there a conflict between apparmor and SELinux ?
Top
ecze
Posts: 533
Joined: Wed Mar 15, 2017 1:54 pm

Re: Did I break Mysql? Cannot login

Post by ecze » Tue Sep 05, 2017 7:23 pm

Don't ask. me how you break your mysql server... But to recover , you need to uninstall mysql-server and purge it

then reinstall it automatically with apt-get install eve-ng

E.
Top
2600
Posts: 36
Joined: Mon Mar 20, 2017 7:45 pm

Re: Did I break Mysql? Cannot login

Post by 2600 » Tue Sep 05, 2017 9:09 pm

Don't ask. me how you break your mysql server... But to recover , you need to uninstall mysql-server and purge it
then reinstall it automatically with apt-get install eve-ng
Exactly !!! All my searching took me to this.. Thank you!! :oops: :lol: :idea:

Just to document for posterity what Ecze has said:

Code: Select all

apt-get purge mysql-server mysql-client mysql-common mysql-server-core-5.7 mysql-client-core-5.7 
apt-get autoremove
rm -rf /etc/mysql /var/lib/mysql
apt-get install eve-ng
Voia la!

That worked like a dream and I won't forget.

Code: Select all

root@eve-ng:~# systemctl status mysql.service
● mysql.service - MySQL Community Server
   Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2017-09-05 15:58:00 EDT; 1h 3min ago
 Main PID: 17386 (mysqld)
    Tasks: 28
   Memory: 48.1M
      CPU: 7.891s
   CGroup: /system.slice/mysql.service
           └─17386 /usr/sbin/mysqld

Sep 05 15:57:59 eve-ng systemd[1]: Starting MySQL Community Server...
Sep 05 15:58:00 eve-ng systemd[1]: Started MySQL Community Server.
:D :lol:
Top
gem
Posts: 1
Joined: Mon Jun 04, 2018 3:20 pm

Re: Did I break Mysql? Cannot login

Post by gem » Wed Jun 06, 2018 5:04 pm

Had the same problem with broken mysql.
I followed the same procedure but did not quite work.
There errors i got where:

Code: Select all

Checking MySQL... done
Creating database and users... failed
dpkg: error processing archive /var/cache/apt/archives/eve-ng-guacamole_2.0.3-69_amd64.deb (--unpack):
 subprocess new pre-installation script returned error exit status 1

Errors were encountered while processing:
 /var/cache/apt/archives/eve-ng-guacamole_2.0.3-69_amd64.deb
W: http://us.archive.ubuntu.com/ubuntu/pool/main/a/apr/libapr1_1.5.2-3_amd64.deb: Automatically disabled Acquire::http::Pipeline-Depth due to incorrect response from server/proxy. (man 5 apt.conf)
E: Sub-process /usr/bin/dpkg returned an error code (1)
Instead, I used:

Code: Select all

apt-get install eve-ng eve-ng-guacamole
and it worked perfectly.

Note:
Labs and configs are not lost after this process.
Top
Post Reply

Return to “EVE-NG Help-Desk”