Multiple instances of BOXI reports spawned after daylight saving time change

Document ID : KB000049617
Last Modified Date : 14/02/2018
Show Technical Document Details

Description:

Problem:

Multiple instances of CA BI reports spawned after Daylight Saving Time change

  • Scheduled reports that are supposed to run once a day, run multiple times
  • Duplicate instances are being created
  • Instead of one instance failing/succeeding at the intended start time, ~650 instances fail per minute per recurring report

The issue is related only to Daily recurring schedules.

Environment:

  • CA BI r3 through CA BI 3.2

For CA BI version cross reference, please see Knowledge Base Article TEC567554

Cause:

This is a known issue with CA BI.

CA BI r3, CA BI 3.1 SP3 & below have a defect that causes daily recurrences to be scheduled continuously on each day after the transition from Standard Time to Daylight Saving Time. This continuous scheduling will start from the time the recurrence becomes eligible to run and lasts for an hour. All instances produced for the recurrence during this hour will fail. The continuous scheduling will consume a large amount of CPU resources and will cause the Central Management Server (CMS) to become unresponsive. Applying this fix prior to the Daylight Saving Time change will prevent the problem from occurring.

Affected CA Products:

CA Service Desk Manager, CA Service Desk on Demand, CA Service Catalog

CA Client Automation, CA Server Automation , CA Configuration Automation

CA Virtual Performance Manager, CA IT Asset Manager, CA Asset Portfolio Manager

CA Software Compliance Manager, CA IT Client Manager , CA IT Inventory Manager

CA Automation Suite for Data Centers, CA Autosys, CA Siteminder, CA Identity Manager

Access Control, CA Clarity on Premise, CA Governance, Risk & Compliance Manager

CA eco Governance, CA Clarity on Demand, CA Spectrum IM, CA Service Operations Insight

CA Storage Resource Manager, CA Software Change Manager, CA DLP

Solution:

Resolution:

We strongly suggest installing CA BI 3.2 SP4 or above. CA BI updates can be downloaded from the following location ftp://ftp.ca.com/CAproducts/CABI

Please check your product compatibility prior to upgrading your CA BI installation. The issue is resolved in CA BI 3.2 SP4 and CA BI 3.2 SP5/.

If the DST issue has already occurred in the affected region, apply the product-specific workaround below to repair the schedules.

Workaround:

If the DST (Daylight Saving Time) issue has already occurred in the affected region, follow the steps outlined below (this issue is related only to Daily recurring schedules).

  • Step 1
    Download the script (EDST_Tool_v1.3.zip), available at ftp://ftp.ca.com/CAproducts/CABI/CABI-3.x/Fixes/TEC566080/

    The functionality of this script is summarized below:

    • Runs a query to display all DAILY Recurring jobs
    • Provides an option to choose the jobs and reschedule them by 1 hour
    • Provides an option to select the same jobs and change them back by 1 hour
    • Provides an option to modify the schedule times (as the schedule itself is being modified, this would keep the issue of duplicate instances from occurring)
    • Upon extraction of the zip file, there should be a "EDST Tool v1.3.hta" file in the extracted folder
    • Login to CMC and stop the Job servers
    • Double-click the "EDST Tool v1.3.hta" file which would launch the "Enterprise Instance Rescheduler" window

      Figure 1

    • Specify the CMS name in the "System Name:" field
    • Specify the Administrator password in the "Password:" field
    • Click the "Log On" button
    • If an error "object required" appears (Windows environments), perform the below steps:

      • Exit out of the "Enterprise Instance Rescheduler" window
      • Register the "EnterpriseFramework.dll" under win32_x86 using the below command:

      regsvr32 /s "<CABI Install Folder>\BusinessObjects Enterprise 12.0\win32_x86\EnterpriseFramework.dll"

    • Re-launch the tool by double-clicking the "EDST Tool v1.3.hta" file under the extracted folder

  • Click the "List All Recurring Instances" button
  • Click the "Select All" button
  • Make a note of what it says in "Schedule Selected Recurring Instances" (should be 1 hour earlier)
  • Click the "Reschedule Selected Recurring Instances" button
  • Choose all instances again (via the "Select All" button) and change "Schedule Selected Recurring Instances" to 1 hour later
  • Click the "Reschedule Selected Recurring Instances" button
  • Restart the Job servers in CMC
  • The issue should not occur again
  • Step 2
    As a result of the DST change, there can be several failed instances. To delete the failed instances, perform the below steps:

    • Be sure to backup the CMS database and FileStore prior to executing the steps mentioned below

      Download the script (deleteFailedInstancesV3.zip) located at ftp://ftp.ca.com/CAproducts/CABI/CABI-3.x/Fixes/TEC566080/

      The functionality of this script is summarized below:

      • Queries for all instances that have a SI_SCHEDULE_STATUS = 3 (3 representing failed)

    • Loops through each of the found instances and deletes them (in batches of 1000 objects which can be configured)

  • Outputs to the browser the instance names and ids of what is being deleted

  • Upon extraction of the zip file, there should be a "deleteFailedInstancesV3.jsp" file in the extracted folder
  • Copy the "deleteFailedInstancesV3.jsp" file to the <Tomcat folder>/webapps/InfoViewApp folder.

    • Ex. Windows
      <CABI Install Directory>\Tomcat55\webapps\InfoViewApp

  • Ex. Unix
    <CABI Install Directory>/bobje/tomcat/webapps/InfoViewApp

  • Edit the "deleteFailedInstancesV3.jsp" using a text editor
  • At the beginning of this file, there would be entries for userName, password, CMSName and batchSize

    String userName = "Administrator";
    String password = "<Password>";
    String CMSName = "<CMSHostName>:<CMSPort>";
    int batchSize = 1000;

    Update these settings appropriate for your system.

  • Save the changes
  • Execute the script by navigating to the below URL in a browser: http://<servername>:8080/InfoViewApp/deleteFailedInstancesV3.jsp

    After the DST change, if the schedules fail with "Object could not be scheduled within the specified time interval", execute the steps outlined below only after the steps in Step 1 above have been performed. The cause for this issue is primarily because the timezone information for Java is not up to date.

  • Download the Java Time Zone update tool from the below location

    http://www.oracle.com/technetwork/java/javase/downloads/index.html

  • Upon extraction of the zip file, there should be a tzupdater.jar in the extracted folder
  • Prior to running the tool, ensure that:

    • There are no running instances of JDK/JRE software to be operated upon

  • User account under which the tool is executed has sufficient privileges to modify the JDK_HOME/jre/lib or JRE_HOME/lib directory

  • Run the tzupdater.jar using the java command from the JDK within the CABI installation folder

    • Windows
      "<CABI Install Directory>\javasdk\bin\java" -jar tzupdater.jar -v -u

  • Unix
    <CABI Install Directory>/bobje/jdk/bin/java -jar tzupdater.jar -v -u


  • Output of running the above command on windows should be similar to below:

    java.home: <CABI Install Directory>\javasdk\jre
    java.vendor: Sun Microsystems Inc.
    java.version: 1.5.0_17
    JRE time zone data version: tzdata2008i
    Embedded time zone data version: tzdata2011b
    Extracting files... done.
    Renaming directories... done.
    Validating the new time zone data... done.
    Time zone data update is complete

In the above example, the Java timezone information file was tzdata2008i and has been updated to tzdata2011b

Prevention:

To prevent this problem from occurring in an environment where the DST issue has not occurred, update to CABI 3.2 SP4 or CABI 3.2 SP5 (assuming that the embedding product has certified these releases).

* For SAP guidance with this issue, please see SAP Note 1568718.