(This post supersedes GitHub Issue #349.)
Here is a recipe for enabling all available debugging tools. You might want to pick and choose depending on the nature of the specific bug you are hunting.
- Catch stack traces.
- Set the environment variable
RUST_BACKTRACE
, either runexport RUST_BACKTRACE=1
before invokingneptune-core
or prepend theneptune-core
withRUST_BACKTRACE=1
.
- Choose informative log level.
- In principle there are four log levels,
ERROR
,WARN
,INFO
, andDEBUG
, and by default everything up to and includingINFO
is enabled. Phrased differently,DEBUG
messages are disabled by default. - To enable
DEBUG
messages, runexport RUST_LOG="debug"
or prepend theneptune-core
command withRUST_LOG="debug"
.
- Trace locks.
- Lock tracing is another (non-standard) set of log messages. It requires nightly.
- Enable nightly:
rustup default nightly
, or alternatively add+nightly
immediately aftercargo
in commands that usecargo
. - Set log level to
"debug,neptune_cash=trace"
with anexport
command or by prepending. - Build and run with
--features log-slow-write-lock,log-slow-read-lock,log-lock_events
(immediately before the double dash (--
) indicating the start of the command line arguments).
- Use
tokio-console
.
tokio-console
is anhtop
-like tool to monitor which threads are doing what.- Install
tokio-console
:cargo install --locked tokio-console
- To instruct
neptune-core
to supporttokio-console
, add--tokio-console
as an extra CLI argument. - Note that
tokio-console
uses port 6669 for communication, so that port had better not be in use by other processes.
- Pipe output into a log file.
tee
can be used to put the console output into a file while preserving the console output.sed
can be used to strip colors, making the file more reader-friendly.- Use both: pipe the
neptune-core
output through| tee >(sed -r "s/\x1B\[([0-9]{1,3}(;[0-9]{1,3})*)?[mGK]//g" > neptune-core.log)
- This suffix will overwrite the log file
neptune-core.log
, so be sure to back it up if one exists already and you might need it.
- All together.
- (Don’t forget to cherry pick!)
- (Don’t forget to
cargo install
dependencies.) - (Don’t forget to replace
[other CLI arguments]
with the CLI arguments you would otherwise runneptune-core
with, such as--max-num-peers 5
or--guess
.) RUST_BACKTRACE=1 RUST_LOG="debug,neptune_cash=trace" cargo +nightly run neptune-core --features log-slow-write-lock,log-slow-read-lock,log-lock_events -- --tokio-console [other CLI arguments] | tee >(sed -r "s/\x1B\[([0-9]{1,3}(;[0-9]{1,3})*)?[mGK]//g" > neptune-core.log)