Yeah, but I seem to be running into a
catch-22.
I can start mysqld when I set
innodb_force_recovery=1 but in that mode I can't do any
commands (Operation not allowed when innodb_force_recovery >
0 )
And then also getting a secondary issue that when I run commands, it appears that mysqld itself crashes when innodb hits the error)
-----
InnoDB: Page
may be an index page where index id is 2129
2022-07-10T12:45:03.529567Z 0 [ERROR] [MY-011937] [InnoDB]
[FATAL] Apparent corruption of an index page [page id:
space=754, page number=4] to be written to data file. We
intentionally crash the server to prevent corrupt data from
ending up in data files.
2022-07-10T12:45:03.529614Z 0 [ERROR] [MY-013183] [InnoDB]
Assertion failure: buf0dblwr.cc:1031:ib::fatal triggered
thread 139965364348672
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes,
even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB:
http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
12:45:03 UTC - mysqld got signal 6 ;
Most likely, you have hit a bug, but this error can also be
caused by malfunctioning hardware.
Thread pointer: 0x7f4c44363410
Attempting backtrace. You can use the following information to
find out
where mysqld died. If you see no messages after this,
something went
terribly wrong...
stack_bottom = 7f4c39d279b0 thread_stack 0x100000
/usr/sbin/mysqld(my_print_stacktrace(unsigned char const*,
unsigned long)+0x41) [0x559acf4f9ad1]
/usr/sbin/mysqld(print_fatal_signal(int)+0x2fb)
[0x559ace39a7db]
/usr/sbin/mysqld(my_server_abort()+0x76) [0x559ace39a926]
/usr/sbin/mysqld(my_abort()+0xe) [0x559acf4f3ace]
/usr/sbin/mysqld(ut_dbg_assertion_failed(char const*, char
const*, unsigned long)+0x349) [0x559acf76f7b9]
/usr/sbin/mysqld(ib::fatal::~fatal()+0xd5) [0x559acf771c75]
/usr/sbin/mysqld(Double_write::croak(buf_block_t
const*)+0x25e) [0x559acf7e551e]
/usr/sbin/mysqld(Double_write::prepare(buf_page_t const*,
void**, unsigned int*)+0x15d) [0x559acf7e575d]
/usr/sbin/mysqld(Double_write::enqueue(buf_flush_t,
buf_page_t*, file::Block const*)+0x97) [0x559acf7ecfc7]
/usr/sbin/mysqld(dblwr::write(buf_flush_t, buf_page_t*,
bool)+0x1fb) [0x559acf7ed80b]
/usr/sbin/mysqld(buf_flush_page(buf_pool_t*, buf_page_t*,
buf_flush_t, bool)+0x277) [0x559acf7f7a67]
/usr/sbin/mysqld(+0x2591657) [0x559acf7fa657]
/usr/sbin/mysqld(buf_flush_do_batch(buf_pool_t*, buf_flush_t,
unsigned long, unsigned long, unsigned long*)+0x653)
[0x559acf7fb583]
/usr/sbin/mysqld(buf_flush_lists(unsigned long, unsigned long,
unsigned long*)+0xc7) [0x559acf7fc907]
/usr/sbin/mysqld(+0x2595ee4) [0x559acf7feee4]
/usr/sbin/mysqld(std::thread::_State_impl<std::thread::_Invoker<std::tuple<Detached_thread,
void (*)()> > >::_M_run()+0xbc) [0x559acf60308c]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xd6de4)
[0x7f4c5f824de4]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8609)
[0x7f4c602c1609]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7f4c5f511133]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): is an invalid pointer
Connection ID (thread ID): 0
Status: NOT_KILLED
The manual page at
http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the
crash.
------
Is there a 'trick' around the catch-22 (still googling but so far haven't turned up anything).
Steve
Seems like it’s just the indexes that are corrupted?
Adam.
Sent from my iPhone
On 10 Jul 2022, at 01:15, Steve Costaras via observium <observium@observium.org> wrote:
Had observium running on a system for a bit under ubuntu 20.04. Had a power issue where it seemed to screw the pooch on the server. don't see filesystem errors that I can detect, however do find that the mysql instance doesn't want to start back up.
When trying to do a repair on the database, get the following:
(edited my.cnf with innodb_force_recovery so as to get mysqld to run)
mysqlcheck -u root -p --auto-repair --all-databases
---cut
observium.slas OK
observium.snmp_errors OK
observium.status OK
observium.storage To be repaired, cause follows:
Server issued Warning : InnoDB: The B-tree of index PRIMARY is corrupted.
Warning : InnoDB: Index index_unique is marked as corrupted
Warning : InnoDB: Index device_id is marked as corrupted
error : Corrupt
observium.syslog OK
observium.syslog_alerts OK
---cut
Only the observium.storage table seems to be messed up (or only one throwing errors at least at this point). Now this is thankfully not the main deployment this is in my test lab (reason why it didn't have good power backup). So I /could/ just re-do the entire deployment, that being said, I would like, if possible to be able to recover this just to save about a years worth of data in the lab, and to see how to do this in case it actually happened on the production system.
anyone have suggestions?
If not, second question would be has anyone tested the install scripts with ubuntu 22.04 yet? If I have to do it from scratch may as well use the newer version.
Thanks
steve
_______________________________________________
observium mailing list
observium@observium.org
http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
-- ---------------------------------------------- Steve Costaras Apprentice codger stevecs@chaven.com https://join.skype.com/invite/r5vO5SB8c17s - "Wealth consists not in having great possessions, but in having few wants." (Epictetus) ----------------------------------------------