What checks are we supposed to perform before bouncing the NuoDB database instance?
NuoDB is designed to keep running even with single failures, so bouncing a database as a problem resolution procedure is considered a last resort for critical problems.
Checks to perform before bouncing a database:
Check that all nodes (brokers) are up and connected;
- If a broker is listed as UNREACHABLE, then fix that problem first, if at all possible;
Check that there is no hung process (ps -ef | grep nuo);
Check that any inconsistencies in the domain state are addressed;
- Any processes that are listed but don’t exist;
Check for any reported errors which would stop the database from starting correctly;
- For example, are there any errors reported by SMs regarding the archives?
Once the database is down, check the archive for consistency (nuochk and use --repair in case there are issues);
- This can be done on an offline copy of the archive if the database needs to be brought up immediately;