I am trying to use CMN_SCH_JOBS.ID field to filter on s chronological jobs in my custom code.
What I see is that it does not return the jobs in the time order they are run, I see gaps in IDs, I see the last running jobs using older IDs. What is the reason for this behavior.
The behavior you are facing is expected.
The ID in CMN_SCH_JOBS will not be chronological and contiguous, as it depends on sequences cache, which is different for each BG service running it. It's expected to have gaps, or older ID used when a newer is already used for previous job instances.
If you have 2 BG services, each will use their own app cache of a specific amount IDs, and they will not overlap. Say BG 1 will have 500001 to 5000050, and BG2 5000050 to 5000100 and they will assign them as needed, depending on the service that's getting the request. If a BG is restarted, the unused IDs will be dropped. So the IDs are really not meant to be used in chronological order.
Suggested way to apply your needs:
We would suggest you use your custom code to create your own chronological id to use. This would be the best possible solution as sometimes the PPM ids may not behave as expected or they may be change in functionality.
If you cannot create an ID in your custom code, then we would suggest you use the start date or a similar date field to get the results formatted as you need.