Talking to web servers with netcat on MacOS

Apple stopped shipping telnet some MacOS version ago, leaving netcat (nc) as an option to tinker with network protocols in shell. If you're unfamiliar with netcat, it's nifty versatile tool that can be used for various debugging purposes.

Let's say you want to send typical GET request to some webserver:

$ nc -c google.com 80
GET / HTTP/1.1
Host: google.com

HTTP/1.1 302 Found
<response headers continue>

Note the "-c" switch. It ensures that "enter" gets converted to carriage return+line feed combo, as required by HTTP to be parsed. Also note that request ends with two newlines: this informs web server that we're done articulating the request.

Of course you don't have to retype HTTP requests every time, feel free to preformat them in advance, parametrize and reuse:

$ printf "GET / HTTP/1.1\r\nHost: google.com\r\n\r\n" | nc google.com 80
HTTP/1.1 302 Found
<response headers continue>

This time "-c" is not needed as printf covers proper formatting of newlines for us. With a simple Bash loop we can send HEAD to a list of hosts:

$ for website in google.com yahoo.com amazon.com; do printf "HEAD / HTTP/1.1\r\nHost: ${website}\r\n\r\n" | nc "${website}" 80 | grep HTTP; done
HTTP/1.1 302 Found
HTTP/1.1 301 Moved Permanently
HTTP/1.1 301 Moved Permanently

Curious for more? Read HTTP RFCs, play with some websites, think about the server responses. Be aware: there are hosts not respecting HTTP rules for multiple reasons (eg. performance, security, misconfiguration).