How do I run neptune-core in a way that is optimized for debugging?

Development
1

(This post supersedes GitHub Issue #349.)

Here is a recipe for enabling all available debugging tools. You might want to pick and choose depending on the nature of the specific bug you are hunting.

  1. Catch stack traces.
  • Set the environment variable RUST_BACKTRACE, either run export RUST_BACKTRACE=1 before invoking neptune-core or prepend the neptune-core with RUST_BACKTRACE=1 .
  1. Choose informative log level.
  • In principle there are four log levels, ERROR, WARN, INFO, and DEBUG, and by default everything up to and including INFO is enabled. Phrased differently, DEBUG messages are disabled by default.
  • To enable DEBUG messages, run export RUST_LOG="debug" or prepend the neptune-core command with RUST_LOG="debug" .
  1. Trace locks.
  • Lock tracing is another (non-standard) set of log messages. It requires nightly.
  • Enable nightly: rustup default nightly, or alternatively add +nightly immediately after cargo in commands that use cargo.
  • Set log level to "debug,neptune_cash=trace" with an export command or by prepending.
  • Build and run with --features log-slow-write-lock,log-slow-read-lock,log-lock_events (immediately before the double dash (--) indicating the start of the command line arguments).
  1. Use tokio-console.
  • tokio-console is an htop-like tool to monitor which threads are doing what.
  • Install tokio-console: cargo install --locked tokio-console
  • To instruct neptune-core to support tokio-console, add --tokio-console as an extra CLI argument.
  • Note that tokio-console uses port 6669 for communication, so that port had better not be in use by other processes.
  1. Pipe output into a log file.
  • tee can be used to put the console output into a file while preserving the console output.
  • sed can be used to strip colors, making the file more reader-friendly.
  • Use both: pipe the neptune-core output through | tee >(sed -r "s/\x1B\[([0-9]{1,3}(;[0-9]{1,3})*)?[mGK]//g" > neptune-core.log)
  • This suffix will overwrite the log file neptune-core.log, so be sure to back it up if one exists already and you might need it.
  1. All together.
  • (Don’t forget to cherry pick!)
  • (Don’t forget to cargo install dependencies.)
  • (Don’t forget to replace [other CLI arguments] with the CLI arguments you would otherwise run neptune-core with, such as --max-num-peers 5 or --guess.)
  • RUST_BACKTRACE=1 RUST_LOG="debug,neptune_cash=trace" cargo +nightly run neptune-core --features log-slow-write-lock,log-slow-read-lock,log-lock_events -- --tokio-console [other CLI arguments] | tee >(sed -r "s/\x1B\[([0-9]{1,3}(;[0-9]{1,3})*)?[mGK]//g" > neptune-core.log)