There are a number of tools to make debugging easier in DyNet. In addition, we welcome any questions or issues, and will be able to respond most effectively if you follow the guidelines below.
It is possible to create visualizations of the computation graph by calling the
print_graphviz() function, which can be helpful to debug. When this functionality is used in Python, it is necessary to add the command line argument
--dynet-viz. In Python, there is also a
print_text_graphviz() function which will be less pretty than the
print_graphviz() function, but doesn’t require the command line flag.
In general, DyNet performs symbolic execution. This means that you first create the computation graph, then the computation will actually be performed when you request a value using functions such as
value(). However, if an error occurs during calculation, this can be hard to debug because the error doesn’t occur immediately where the offending graph node is created. To make debugging simpler, you can use immediate computing mode in dynet. In this mode, every computation gets executed immediately, just like imperative programming, so that you can find exactly where goes wrong.
In C++, you can switch to the immediate computing mode by calling ComputationGraph::set_immediate_compute as follows:
ComputationGraph cg; cg.set_immediate_compute(true);
Further, dynet can automatically check validity of your model, i.e., detecting Inf/NaN, if it is in immediate computing mode. To activate checking validity, you can add the following code after switching to immediate computing mode.
In Python, these values can be set by using optional arguments to the
renew_cg() function as follows:
dy.renew_cg(immediate_compute = True, check_validity = True)
By default, DyNet is built with all optimization enabled.
You can build DyNet without optimizations by adding
-DCMAKE_BUILD_TYPE=Debug to the cmake command
cd dynet mkdir build cd build cmake .. -DCMAKE_BUILD_TYPE=Debug make -j8 # replace 8 properly
Note: pass other cmake options based on your environment.
Build with ASan¶
If you’re on Linux or macOS, you can build DyNet with
(aka ASan). ASan is a memory error detector for C/C++. It’s useful for debugging
bugs or crashes caused by memory errors such as use-after-free, heap buffer overflow,
stack buffer overflow. By running ASan-enabled tests or programs, ASan finds memory
errors at runtime. To enable ASan, add
to the cmake command:
cd dynet mkdir build-asan cd build-asan cmake .. -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_FLAGS="-fsanitize=address" make -j8
Please see the official wiki for the details.
CAUTION: Please do not install ASan enabled libraries or programs under root partition. You might have a bad time.
Debugging Threading Issues¶
Build with TSan¶
If you’re on Linux or macOS, you can build DyNet with ThreadSanitizer (aka TSan). TSan is a data race error detector for C/C++. It finds data races at runtime just like ASan. Please see the official wiki for more details.
By running TSan-enabled tests or programs, TSan finds data races at runtime.
To enable TSan, add
to the cmake command:
cd dynet mkdir build-tsan cd build-tsan cmake .. -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_FLAGS="-fsanitize=thread"
CAUTION: Please do not install TSan enabled libraries or programs under root partition. You might have a bad time.
Asking Questions/Reporting Bugs¶
Feel free to contact the dynet-users group or file an issue on github with any questions or problems.
(If you subscribe to
dynet-users and want to receive email make sure to select “all email” when you sign up.)
When you have an issue, including the following information in your report will greatly help us debug:
- What is the error? Copy and paste the error message.
- What is your environment? Are you running on CPU or GPU? What OS? If the problem seems to be related to a specific library (CUDA, Eigen), what version of that library are you using?
- If possible, it will be really really helpful if you can provide a minimal code example that will cause the problem to occur. This way the developers will be able to reproduce the problem in their own environment.
If you have a build problem and want to debug, please run
make clean make VERBOSE=1 &> make.log
then examine the commands in the
make.log file to see if anything
looks fishy. If you would like help, send this
make.log file via the
“Issues” tab on GitHub, or to the dynet-users mailing list.