Several jobs failed up to 45 times after the predecessor jobs were put ON_ICE.
Why did a job run when it's predecessor job was put ON_ICE?
When a job goes ON_ICE, the job is effectively taken out of the job stream as if the job did not exist. So if a job has a SUCCESS dependent on an ON_ICE job, the condition behaves as if the ON_ICE job’s condition was never there. If the only other condition is a NOT RUNNING condition on a job that is not running, then the job will execute. This is expected behavior.