Understanding NetMaster Databases

Document ID : KB000055433
Last Modified Date : 14/02/2018
Show Technical Document Details

NetMaster Databases (NDBs) allow NCL programmers to easily define, create, update, and search a database based on a collection of user-defined fields. NDBs support information storage and retrieval through the CA NetMaster and SOLVE functions. For example, a SHOW NDB=ALL OCS command typically shows up to five NDBs, depending on the product:

Figure 1

The RAMDB (Automation Database File) and RSDB (Network Model File) are stable NDBs that rarely require attention; however, they should be backed up periodically to secure changes. For example, changes to resource definitions and Automation Services processes are stored in the RAMDB.

The Alert History File ($ALERTH), File Transfer Event Database (EVENTDB), and IPLOG Event History File (IPLOG) are used to store data in a cumulative fashion for searching, which is why all have customer-defined parameters such as the number of days to keep data. These files can become full if they are too small or there is an unusual peak in activity. In CA NetMaster r11 and CA NetMaster r11.5, enhancements were made to remove the need for reorganization and to reduce the chances of a file becoming full without warning.

RID Reuse

Each record in an NDB is assigned a unique Record ID (RID). Prior to CA NetMaster r11, the RID incremented and as records were deleted, unusable free space accumulated. To reclaim the space left by the deleted records, a periodic reorganization was needed.

In CA NetMaster Version 7, automatic reorganization and resize for the IPLOG and EVENTDB was introduced to reduce the requirement for maintenance. In CA NetMaster r11, the RID reuse feature was introduced and made active on the IPLOG and EVENTDB NDBs, and in CA NetMaster r11.5, RID reuse was actived on the $ALERTH NDB. Now, all dynamic data NDBs used by CA NetMaster and SOLVE do not require reorganization to reclaim free space.

RID reuse accumulates ranges of unused RIDs from deleted records during a KEYSTATS run, which is performed automatically during the daily old record deletion process. By reusing RIDS, free space in the VSAM cluster is also reused.

Looking at the output from the SHOW NDB command shown previously, the RID reuse status is shown at the end of the line for each NDB and you can see that it is A (Active) on each of the $ALERTH, EVENTDB, and IPLOG files. We do not recommend reorganizing NDBs when RID reuse is active.

The following is the log output from an EVENTDB daily purge showing RID reuse as being effective. It is easy to find the messages in the log by looking at the purge or delete time for the file in the Customizer Parameter group (/PARMS) and then locating that time in the log using the T hh.mm command.

Figure 2

You can see how quickly the KEYSTATS run completed - in the same second that it starts!

Alerts for VSAM Monitoring

In CA NetMaster r11.5, you can create alerts for VSAM files allocated to a CA NetMaster or SOLVE region. This includes the NDBs because they use the same VSAM API. Event Distribution Services (EDS) has for some time created events for file open, close, allocate, unallocated, and so on. From CA NetMaster r11.5, the following new events, triggered by VSAM return and feedback codes, were introduced:

  • $$SYS.FILE.EOV for End Of Volume (new data or index extent)
  • $$SYS.FILE.CA.SPLIT for a VSAM control area split (not used by the VSAM monitor)
  • $$SYS.FILE.FULL for a file full condition
  • $$SYS.FILE.SHORTAGE for a string or buffer shortage
  • $$SYS.FILE.ERROR for a VSAM error condition

These events are trapped by the VSAM monitor and alerts are raised if required. Log files and files opened for input are excluded from alerts related to file size because log files wrap and input files cannot be extended by the region's activity. The following shows the parameter panel for the VSAM monitor:

Figure 3

As you can see, Alerts for String / Buffer shortage or file extensions are optional. If you do not want the IPLOG or EVENTDB to be automatically resized, the file extension severity count should be cleared; otherwise, resize is attempted when the Extents count is exceeded. The improvements in CA NetMaster r11.5 are as follows:

  • VSAM file size monitoring is based on the number of file extents as they occur, rather than a time-based check on data and index use percentages. The latter can change when a new extent is allocated and the time-based check may not detect a file growing close to maximum size until it is too late.
  • There is no automatic reorganization of files because RID reuse takes care of free space from deletions.

Respond to an Alert for File Size or File Full

The following panel shows an alert for file size:

Figure 4

The Alert Text contains data for up to 20 extensions, enabling the file growth rate to be predicted. The alert for file full is similar and the Recommended Action contains instructions on recovery. When recovering a full or near-full NDB to a larger file, it is important to disconnect the old and new files completely from the region during the REPRO of old file records into a new file (it should be empty) so that there are no problems with control record corruption. Unfortunately, the event data is not recorded during this time; however, the automatic resize of IPLOG and EVENTDB does cache data during the resize, though loss of data is not guaranteed.

Manage NDBs at Installation and Region Setup

The majority of calls to Technical Support about NDBs are about installation. When you create a new region for a product upgrade, ensure that you do the following:

  • Use the JCL created during installation to allocate new NDBs.
  • Review the size of files such as IPLOG from previous use and modify the JCL if required to increase the file size.
  • Do not attempt to allocate an NDB from another region. A migration utility is provided to preserve RAMDB data across product version upgrades.

Fix a Corrupted NDB

Occasionally, an NDB may become corrupted by an NDB full or region cancelled event. To recover the data, you must use the NDB ALTER OPT=BLDX command to rebuild the indexes.

Note : The following steps are shown for the IPLOG file and apply to all NDBs.

To fix a corrupted NDB

  1. Create a work file.

    Note : The size must relate to the number of records in the NDB.

    Sample JCL is:
    //DEFWRK   EXEC  PGM=IDCAMS//SYSPRINT DD  SYSOUT=*                          //SYSIN    DD  *                                  DELETE (hlq.TEST.NDBWORK) CL                     DEFINE CLUSTER (NAME(hlq.TEST.NDBWORK)      -                    MGMTCLAS(DEFAULT)           -                    STORCLAS(NMDPOOL)           -                    INDEXED                     -                    RECORDS(1000000 100000)     -                    SPEED                       -                    SHAREOPTIONS(2 3)           -                   )                            -           DATA    (NAME(hlq.TEST.NDBWORK.D)    -                    CONTROLINTERVALSIZE(4096)   -                    RECORDSIZE(1017 4089)       -                    FREESPACE(0 0)              -                    KEYS(4 0)                   -                   )                            -           INDEX   (NAME(hlq.TEST.NDBWORK.I)    -                    CONTROLINTERVALSIZE(1024))      /*

  2. Enter the following command from OCS to allocate the work file to the region:
    ALLOCATE DD=NDBWORK DSN=hlq.TEST.NDBWORK DISP=SHR
  3. Issue the following command to open the work file as a VSAM file.
    UDBCTL OPEN=NDBWORK STRNO=3 BUFNI=5 BUFND=6
  4. Issue the following command to remove the NDB bad-locked status:
    NDB PURGE IPLOG
  5. Issue the following command to stop the NDB to prevent access:
    NBD STOP IPLOG IMM
  6. Issue the NDB ALTER command to check the indexes:
    NDB ALTER IPLOG OPT=CHKX DB WORK=NDBWORK DETAIL=YES SORT=100000
    Important! Minimizing I/O on the work file by specifying the maximum value for sort memory can make a big difference to execution time. A CHKX on a file of 676,000 records takes 1.5 minutes with SORT=100000 and 7 minutes without.

  7. Issue the following command to close and free the work file:
    UDBCTL CLOSE=NDBWORKUNALLOC DD=NDBWORK
  8. If the CHKX shows an error, rebuild the indexes by redefining the work file to empty it, and then use the BLDX option of NDB ALTER:
    NDB ALTER IPLOG OPT=BLDX DB WORK=NDBWORK DETAIL=YES SORT=100000
    Important! Minimizing I/O on the work file by specifying the maximum value for sort memory can make a big difference to execution time. A BLDX on a file of 676,000 records takes 2.1 minutes with SORT=100000 and 7.6 minutes without.

  9. Reallocate the NDB by actioning the IPFILES - TCP/IP File Specifications Customizer Parameter Group.

    If the NDB ALTER BLDX action fails, and needs to be rerun, the work file must be unallocated and redefined.