Today I tested OracleVM (OVM) templates on their own distribution of Oracle Enterprise Linux (OEL) 5 with seeded VNC Viewer Free Edition 4.1.2 for X. All went fine, but the VNC connection to virtual machine was not painless as I expected.

[root@oram ~]# rpm -qa "vnc|xen"
kernel-xen-2.6.18-128.el5
xen-3.0.3-80.el5
kmod-gfs-xen-0.1.31-3.el5
kmod-cmirror-xen-0.1.21-10.el5
vnc-4.1.2-14.el5
kmod-gnbd-xen-0.1.5-2.0.1.el5
vnc-server-4.1.2-14.el5

On a brand new OEL5 system with virtualization support, I have created a new virtual machine with a fresh OEL4 (plain OS), and set appropriate memory for the virtual domain using the Xen management user interface commands. I also checked if the VNC port was allocated with the command virsh dumpxml:

[root@oram ~]# xm create /OVS/running_pool/OVM_EL4U6_X86_PVM_4GB/vm.cfg
Using config file "/OVS/running_pool/OVM_EL4U6_X86_PVM_4GB/vm.cfg".
Started domain OVM_EL4U6_X86_PVM_4GB
[root@oram ~]# xm mem-set 1 1024
[root@oram ~]# xm list
Name                     ID Mem(MiB) VCPUs State   Time(s)
Domain-0                 0      997     2 r-----   284.5
OVM_EL4U6_X86_PVM_4GB    1     1023     1 -b----    73.6
[root@oram ~]# virsh dumpxml OVM_EL4U6_X86_PVM_4GB |grep port
<graphics type='vnc' port='5900' listen='0.0.0.0'/>
[root@oram ~]# netstat -l |grep 5900
tcp        0      0 *:5900               *:*                  LISTEN

Now I expected that VNC Viewer would simply connect to the console of VM, but it displayed only the very first frame and then died after a few moments with a Connection reset by peer (104) error:

[root@oram ~]# vncviewer localhost:0
VNC Viewer Free Edition 4.1.2 for X - built Jan 21 2009 14:35:26
Copyright (C) 2002-2005 RealVNC Ltd.
See http://www.realvnc.com for information on VNC.
Fri Aug 14 15:59:24 2009
CConn:       connected to host localhost port 5900
CConnection: Server supports RFB protocol version 3.8
CConnection: Using RFB protocol version 3.8
TXImage:     Using default colormap and visual, PseudoColor, depth 8.
Fri Aug 14 15:59:25 2009
CConn:       Using pixel format depth 6 (8bpp) rgb222
CConn:       Using ZRLE encoding
Fri Aug 14 15:59:29 2009
CConn:       Throughput 20156 kbit/s - changing to hextile encoding
CConn:       Throughput 20156 kbit/s - changing to full colour
CConn:       Using pixel format depth 8 (8bpp) colour-map
CConn:       Using hextile encoding
main:        read: Connection reset by peer (104)

As I was able to see the VM console for a while,  my first guess was that it would be related to the virtual system itself, so  I decided to create another virtual domain to see if the problem is replicable. And yes, it was! (Well, in addition to the 104 error I got another one: Broken pipe (32), which was very likely related to domain auto-restarts, and easily traceable by xm list command showing the current VM domain ID changes.)

And the solution for Connection reset by peer (104)? I noticed that VNC Viewer’s default setting is auto-encoding selection, which means first use ZRLE and then switch to hextile if possible—that creates a problem here. When I set any encoding (fixed to either ZRLE or hextile), it started to work well:

[root@oram ~]# vncviewer localhost:0 PreferredEncoding=Hextile
VNC Viewer Free Edition 4.1.2 for X - built Jan 21 2009 14:35:26
Copyright (C) 2002-2005 RealVNC Ltd.
See http://www.realvnc.com for information on VNC.
Fri Aug 14 18:10:39 2009
CConn:       connected to host localhost port 5900
CConnection: Server supports RFB protocol version 3.8
CConnection: Using RFB protocol version 3.8
TXImage:     Using default colormap and visual, PseudoColor, depth 8.
Fri Aug 14 18:10:40 2009
CConn:       Using pixel format depth 6 (8bpp) rgb222
CConn:       Using hextile encoding

Do you have any related or similar issues to share?