I'm trying to automate detection of broken links and I was startled to see that HTTP HEAD method is not always reliable. For example the URL: http://www.youtube.com/mit -- the URL is accessible with GET method but trying to contact the URL with HEAD method ends up with receiving 404 error (not found).
Note: The example can be easily checked online via: http://web-sniffer.net/ (I'm not affiliated with the webpage in any way)
So how reliable is the HEAD method in the end?
As a reasonable solution I see just don't use HEAD and always use GET method always but it is wastes bandwidth.