Workload Automation AE was configured in the following manner:
Network issues occurred such that the primary scheduler lost connectivity with shadow database and the other two schedulers.
The shadow and tie-breaker schedulers lost connectivity with the primary database and the primary scheduler.
The primary scheduler continued processing events in single database and single scheduler mode.
The shadow scheduler rolled into single database mode.
Since the primary scheduler could not communicate with the shadow database the primary scheduler's time stamp in the ujo_ha_process table became stale.
The shadow scheduler assumed the primary scheduler was having issues and assumed the role of primary scheduler.
Both schedulers generated the DB_ROLLOVER alarm.
The primary scheduler generated EP_HIGH_AVAIL while the shadow generated EP_ROLLOVER.
During a period of time both schedulers were actively processing events in their own separate databases.
As part of the procedure to get AE back into the high availability mode the databases must be synced.
But as a result one database will lose what it had processed.
Can the two AE databases be merged instead?
There is no supported procedure to manually merge two out of sync AE databases.
There would be conflicts if attempts were made, such as duplicate or mismatched EOIDs, run_nums, statuses, etc...
The Workload Automation AE product ships with the autobcp utility that should be used to sync two AE databases.
Users provide a source and target database.
The target database is truncated and then re-populated with a copy of the data from the source database.
For more details about autobcp see the product documentation here:
The primary and shadow schedulers event_demon logs can be referred to for an account of what occurred leading up to and including the problem time frame.