Connecting to SCVMM remotely via install on workstation cannot run console windows to VMs (locally on scvmm it works)

We recently installed a SCVMM 2012 machine and connected it to a few HyperV2012 hosts.

From the SCVMM console locally on the machine it is installed on, I can connect to the Console of the VMs on the remote HyperV machines.  However for some reason when I run it remotely from my workstation, I can connect to SCVMM just fine, but connecting to the console of the VMs gives me the error:

Virtual Machine Manager lost the connection to the virtual machine for one of the following reasons:
Another connection was established to the console of this manchine.
The virtual machine has been shut down or put into the saved state.
The user credentials do not have the necessary privilege to connect.
0x0003, 0x0000

The firewall is not detecting any dropped packets (I enabled logging) and I am logging in with the same user as I do locally to the SCVMM server, so it is not permissions.

Any advice would be extremely helpful.  Thanks,

Nicholas