Collecting and Analyzing Crash Data from a Solaris Environment.

Document ID : KB000053838
Last Modified Date : 14/02/2018
Show Technical Document Details

Description:

When a SiteMinder component crashes and generates a corefile, what is the best approach to collect the necessary data for the Support team to analyze?

Solution:

While the corefile itself is useful for analysis, the associated symbols within linked libraries are also necessary. There is a script, "pkg_app.sh", that will automatically gather the relevant linked libraries and executables of the crashed process, create a tarfile with these and the actual corefile, and also create files for later use with the 'dbx' debugger. The steps to use this are as follows:

(This example assumes a Policy Server crash, just substitute the actual crashing process name)

Step 1) Create the situation where the Policy Server process (smpolicysrv) crashes, and generates a corefile.

Step 2) Run "pkgapp -c <corefile name> -p <full path to process binary including the name of he executable>". This creates a tarfile which includes the corefile, all relevant linked libraries, relevant executables of the crashed process, and generated files for use with the dbx debugger.

Step 3) Upload the resulting tarfile to Support for analysis, along with any additional requested log files (ideally all generated logs surrounding the crash).

(These last steps may be performed by Support after uploading.)

Step 4) run 'opencore' (an auto-generated script within the tarfile). This should only be run on Solaris (the Solaris version doesn't need to match, but should ideally be the same or later OS version). Running this should place you in the 'dbx' debugger.

Step 5) Once in 'dbx', the following commands will generally provide the necessary debugging information for a crash analysis:

> 'where -l' (lowercase ell; shows backtrace of current (crashed) thread, plus libraries associated with the crash)
> 'lwps' (lists all threads)
> 'proc -map' (shows address ranges of different libraries, may be used to determine which library actually failed)

To obtain the actual 'pkg_app.sh' script, please see the following Sun Support link (Note: you must be logged in with your Sun ID to see this link), and download "SUN-GDD_pkgapp_solaris_sparc.tar":

http://www.sun.com/bigadmin/jsp/descFile.jsp?url=descAll/sun_gdd__directory.

More information about the Gathering Debug Data tool can be found here on Sun's website:

http://www.sun.com/service/gdd/index.xml.