A number of websites on a single Apachee/2.2.16 (Debian) server were being migrated from named based virtual hosting to IP based virtual hosting and SSL. When the SSL sites were tested, most were OK, but four gave the error:
SSL received a record that exceeded the maximum permissible length. (Error code: ssl_error_rx_record_too_long)
at the browser, and this:
Invalid method in request \x16\x03\x01
at the server error log. Several discussions (such as at Linux Questions, at Ubuntu Forums, and at Server Fault) suggest the server is misconfigured and serving plain HTTP, without SSL, perhaps because the mod_ssl module is not loaded. However, that seemed unlikely for this case; the server was already serving SSL for one site, and the other sites that were migrated at the same time were OK.
After fruitless detailed checking of the Apache configuration files, it was noticed that the DNS was still serving the old IP address for the affected sites. The migration required each of the sites to have its own IP address, so additional IP addresses had been configured in
/etc/network/interfaces, and the DNS zone files had been modified accordingly. A typo in one of the zone files had made it invalid, so bind had refused to load it and kept using the old zone file. Therefore, attempts to reach the migrated sites were contacting port 443 on the server's base IP address, which was configured with a different SSL certificate.
Correcting the DNS did not immediately solve the problem, because the testing client had cached the DNS entries locally. Restarting the client was the easiest way to clear the cache, and the sites worked correctly afterwards.
The error messages "rx_record_too_long" and "Invalid method in request" are not very helpful, they suggest that the fault lies with the client. It seems these messages can appear in different circumstances, including when a server is not using SSL, and when a server is using a valid certificate for a different host.