TLS for VNC

I ported the patches from Mark McLoughlin for TLS support to realvnc version 4.1.1 and modified them so, that they are working under Unix, Windows and Java. As I introduced an additional byte before the TLS Handshake, I use other security types.

Important: RealVNC 4.1.1 (and therefore the TLS for VNC patches) can be forced to use any security type by the client. This patch solves the problem.

The patches implement the following encryption/authentication methods for VNC:

• None no encryption and authentication
• VncAuth no encryption, standard VNC authentication
• Plain no encryptions, asks username and password. In the server, no verification code for the username/password pair have been written, for testings purposes a disabled verificator exists, which accept username/password combinations, where the first character are equal. In the client, this security type is not activated (addSecType(secTypeVncPlain); is commented out), as it is insecure to transmit a password without any encryption. Additionally the number of supported security types in the viewer would have to be raised.
• TLSNone no authentication, data is encrypted using GNUTLS. The identity of the server is not verified.
• TLSVnc standard VNC authentication with encryption like TLSNone.
• TLSPlain authentication like Plain with encryption like TLSNone. It is present in the clients, but no password verficator is available.
• X509None no authentication, data is encrypted using GNUTLS. The identity of the server is verified using X509 certificates. The key and certificated are specified using the x509cert and x509key command line parameter in the server. The unix and windows viewer accept the command line parameter x509ca and x509crl to specify optionally the certificate of the CA to verify against. If a check of the server certificate fails, a cryptic error message is shown and the user is asked, if he wants to abort. To generate X509 certificates, use openssl or a frontend like TinyCA.
• X509Vnc standard VNC authentication with encryption like X509None.
• X509Plain authentication like Plain with encryption like X509None. It is present in the clients, but no password verficator is available.

Some limitations:

• X509None/Plain/Vnc show no useful data, if it asks the user, if he wants to abort.
• More checks for the server certificate are needed (eg. is it expired)
• The Java Viewer does not verify the server certificate
• Certificates with more than 1024 Bit are not supported by the SUN JRE.
• The Java Viewer runs only on a Sun JRE 1.4.2 or later.
• TLS in the Windows server can only be enabled using the Registry or command line parameter SecurityTypes.
• The patches use 7 new Security Types.
• Windows viewer delays in some situations screen updates, until a key is pressed. As a fix add

  SendMessage(handle,WM_PAINT,0,0);

  after the call of InvalidateRect in DesktopWindow::invalidateDesktopRect (win/vncviewer/DesktopWindow.cxx).

All source files (and patches) are contained in tlsvnc.tgz. Additionally you need realvnc free edition 4.1.1 (4.1 for Java) from RealVNC.

To compile the unix version, unpack the unix sources, apply the patches vnc-tls-unix.patch, vnc-tls-build.patch and vnc-tls-sectype.patch. Then copy all cxx and h files out of tlsvnc.tgz to the common directory (common/rdr and common/rfb, depending on the directory tlsvnc.tgz). Then run aclocal and autoconf. To enable the TLS support, pass the parameter --enable-gnutls to configure.

For the java viewer, unpack the java sources, copy all java files out of vnctls.tgz to common/rfb and apply the patch vnc-tls-java.patch. Then build the java viewer.

For the windows version, GNUTLS must be compiled first. I managed to get all need features of GNUTLS under MS Visual C++ 6 working, but other functions (like network IO) will not work:

• Unpack gnutls-win32-msvc.tgz.
• Unpack libgpg-error-1.0.tar.gz in the same directory.
• Unpack libgcrypt-1.2.1.tar.gz in the same directory.
• Unpack gnutls-1.2.4.tar.bz2 in the same directory.
• Open the work space and compile the gnutls.lib project (gnutls - Win32 Release)

• Unpack the windows sources
• Copy the directory gnutls/gnutls out of the GNUTLS build tree into the win directory.
• Coppy gnutls.lib to win/gnutls
• Then copy all cxx and h files out of tlsvnc.tgz to the common directory (common/rdr and common/rfb, depending on the directory tlsvnc.tgz).
• Apply vnc-tls-win.patch and vnc-tls-sectypes.patch. vnc-tls-winbuild.patch should contain all changes to the project files:
  • Add the definitions HAVE_GNUTLS to each subproject
  • Add the directory gnutls/gnutls out of the GNUTLS build tree to the include path to each subproject
  • Add gnutls.lib as library for all executables.
• Open the project and start the build.

For gnutls 1.4.0, libgcrypt 1.2.2 and libgpg-error-1.3, use gnutls-win32-1.4.0.tgz as build files.