Yeah, but I seem to be running into a catch-22.
I can start mysqld when I set innodb_force_recovery=1 but in that mode I can't do any commands (Operation not allowed when innodb_force_recovery
0 )
And then also getting a secondary issue that when I run commands, it appears that mysqld itself crashes when innodb hits the error)
-----
InnoDB: Page may be an index page where index id is 2129 2022-07-10T12:45:03.529567Z 0 [ERROR] [MY-011937] [InnoDB] [FATAL] Apparent corruption of an index page [page id: space=754, page number=4] to be written to data file. We intentionally crash the server to prevent corrupt data from ending up in data files. 2022-07-10T12:45:03.529614Z 0 [ERROR] [MY-013183] [InnoDB] Assertion failure: buf0dblwr.cc:1031:ib::fatal triggered thread 139965364348672 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 12:45:03 UTC - mysqld got signal 6 ; Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware. Thread pointer: 0x7f4c44363410 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 7f4c39d279b0 thread_stack 0x100000 /usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x41) [0x559acf4f9ad1] /usr/sbin/mysqld(print_fatal_signal(int)+0x2fb) [0x559ace39a7db] /usr/sbin/mysqld(my_server_abort()+0x76) [0x559ace39a926] /usr/sbin/mysqld(my_abort()+0xe) [0x559acf4f3ace] /usr/sbin/mysqld(ut_dbg_assertion_failed(char const*, char const*, unsigned long)+0x349) [0x559acf76f7b9] /usr/sbin/mysqld(ib::fatal::~fatal()+0xd5) [0x559acf771c75] /usr/sbin/mysqld(Double_write::croak(buf_block_t const*)+0x25e) [0x559acf7e551e] /usr/sbin/mysqld(Double_write::prepare(buf_page_t const*, void**, unsigned int*)+0x15d) [0x559acf7e575d] /usr/sbin/mysqld(Double_write::enqueue(buf_flush_t, buf_page_t*, file::Block const*)+0x97) [0x559acf7ecfc7] /usr/sbin/mysqld(dblwr::write(buf_flush_t, buf_page_t*, bool)+0x1fb) [0x559acf7ed80b] /usr/sbin/mysqld(buf_flush_page(buf_pool_t*, buf_page_t*, buf_flush_t, bool)+0x277) [0x559acf7f7a67] /usr/sbin/mysqld(+0x2591657) [0x559acf7fa657] /usr/sbin/mysqld(buf_flush_do_batch(buf_pool_t*, buf_flush_t, unsigned long, unsigned long, unsigned long*)+0x653) [0x559acf7fb583] /usr/sbin/mysqld(buf_flush_lists(unsigned long, unsigned long, unsigned long*)+0xc7) [0x559acf7fc907] /usr/sbin/mysqld(+0x2595ee4) [0x559acf7feee4] /usr/sbin/mysqld(std::thread::_State_impl<std::thread::_Invoker<std::tuple<Detached_thread, void (*)()> > >::_M_run()+0xbc) [0x559acf60308c] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xd6de4) [0x7f4c5f824de4] /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f4c602c1609] /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7f4c5f511133]
Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0): is an invalid pointer Connection ID (thread ID): 0 Status: NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash.
------
Is there a 'trick' around the catch-22 (still googling but so far haven't turned up anything).
Steve
On 2022-07-10 04:06, Adam Armstrong via observium wrote:
Seems like it’s just the indexes that are corrupted?
Try: https://blog.ghost3k.net/articles/mysql/169/fixing-mysql-innodb-index-corrup...
Adam.
Sent from my iPhone
On 10 Jul 2022, at 01:15, Steve Costaras via observium observium@observium.org wrote:
Had observium running on a system for a bit under ubuntu 20.04. Had a power issue where it seemed to screw the pooch on the server. don't see filesystem errors that I can detect, however do find that the mysql instance doesn't want to start back up.
When trying to do a repair on the database, get the following:
(edited my.cnf with innodb_force_recovery so as to get mysqld to run)
mysqlcheck -u root -p --auto-repair --all-databases
---cut
observium.slas OK observium.snmp_errors OK observium.status OK observium.storage To be repaired, cause follows: Server issued Warning : InnoDB: The B-tree of index PRIMARY is corrupted. Warning : InnoDB: Index index_unique is marked as corrupted Warning : InnoDB: Index device_id is marked as corrupted error : Corrupt observium.syslog OK observium.syslog_alerts OK
---cut
Only the observium.storage table seems to be messed up (or only one throwing errors at least at this point). Now this is thankfully not the main deployment this is in my test lab (reason why it didn't have good power backup). So I /could/ just re-do the entire deployment, that being said, I would like, if possible to be able to recover this just to save about a years worth of data in the lab, and to see how to do this in case it actually happened on the production system.
anyone have suggestions?
If not, second question would be has anyone tested the install scripts with ubuntu 22.04 yet? If I have to do it from scratch may as well use the newer version.
Thanks
steve
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium