ILM partition failed halfway through

Document ID : KB000098739
Last Modified Date : 07/06/2018
Show Technical Document Details
Issue:
The ILM partition failed halfway through and is in a partial state. Guidance is needed on how to proceed with the rest of the statements so that the ILM partition finishes. 
Cause:
There are several factors that can cause the ILM partition to be disrupted. Some example lines from the PWP:

U00005323 WARNING: ==ILM-REDO-MODE== Due to a possible Server crash some ILM statements could not be processed. They will be executed now.

U00005322 There is a statement showing status 'R' (unsuccessfully restarted) in the ISTMT table. This has to be manually checked and solved.
Resolution:
The following steps should be done to run the rest of the ILM partition:

- Find the remaining statements that need to be ran (will be in the ISTMT table under column ISTMT_TEXT)

- The statements can be run outside the Automation Engine but then all index on the tables are invalid - as long as the core is not started with handling invalid index on Oracle this cannot be done online. Thus, the boundaries for the remaining partition need to be set to some values higher than today - so no data needs to be moved and all indexes will stay valid. This will require DBA assistance. Update the number to split at for each remaining statement by using the values from the table 'IDS' for each type, and add to this value some numbers to make sure no data is moved.

- Update the IDS to the new numbers - otherwise the WP's will still be writing old numbers to the old partition for new started tasks.  

-Recycle the WP's to pickup the new counter's (this will require no downtime as the WP's can be restarted one by one)

- Once the statements have been ran outside the engine then the ISTMT.ISTMT_STATE needs to be set to 'C' (Completed) for all rows and ILM can be activated afterwards. 
 
Additional Information:
If there are further questions on how to proceed, please contact support and send in the PWP log and a screenshot of the contents of the ISTMT table. 

The PWP is the WP in System Overview with Type 'P'.