There are significant structural changes to the CRM system with version 5.0 (hence the 5.0 moniker). At the same time we took the opportunity to actually remove many of the old script functions that we have stated as being deprecated for many releases. The functionality in these script functions has not gone, it can just be called much more directly.
I suggest reading this article https://forums.xtuple.com/t/xtuple-erp-5-0-0-beta-and-other-components-are-now-available/6420 and looking at the updatetodb package that is attached to the article. This package runs through the scripts and checks for problems.
Not really answering your question, however I believe the problem you are seeing is likely caused by a call to a script function that no longer exists. The package will suggest how to fix those problems.
Getting back to the actual question. Adding a debugger; statement inside the initMenu script and switching on Script Debugging in Preferences works. The initMenu script will stop it's execution at that point and allow you to step through the script and identify the problem.
Setup is a problem area in that the setup screens are modal and block everything. The only thing I've found to work are lots of QMessageBox calls to popup debug text, or when the debugger call is hit and the code stops, to actually close the setup screen, which then allows you to step through the script.