When we checked QoS on doc ops, it also says it could go above 100%:-
vmware probe monitoring vcenter and esx servers and estsummary unified dashboard, HostMemoryUsage metric shows above 100%
why it can go above 100% and whats the calculation formula used for this.
The VMware probe does not use vCenter for its metric gathering, so comparing it's numbers to the probe reported values isn't 1-to-1.
The probe uses the Managed Object Browser, which is a separate API designed specifically for other applications to grab metrics from the VMware instance. The probe grabs metrics straight from the MOB, so the reported value is exactly what the VMware server reported, so it's not an issue with the probe.
It is entirely possible that a VM uses more than 100% of its allocated memory at times, VMware allows for that in certain situations, but the OS and vCenter will report something different, because they're measuring the value differently.