If you are running with VM:Tape, things will be much easier. VM:Tape has a documented interface to DFSMS/RMS running on z/VM. RMSMASTR and DFSMS/RMS are required to be installed and running.
VM:Tape can also be used with other "tape appliances" (3rd Party VTLs), although not necessarily using a published interface.
If you run VM:Tape, you will need to configure VM:Backup to run with VM:Tape and define new VM:Backup TAPEPOOL records with the DENSITY and MEDIA attributes that match your VTL volumes.
You would then modify your VM:Backup Job Templates to use the new "tape" Tape Pools instead of your existing "DASDTape” pools. Then in theory you are done.
If you don't run VM:Tape, that complicates things. VM:Backup does not have any native interfaces other than to use VM:Tape.
VM:Backup does have a Tape User Exit (VMBEXIT5), although, it would take substantial coding to make it work with the IBM VTL and DFSMS/RMS.