In the beginning

Similar to some Mail User Agents (MUAs) and NNTP clients, the specification for HTTP since at least version 0.9 have included User-Agent headers that still exist in modern HTTP standards.

This header lets the server know what software made the request. But it also provides a reminder that the software existed to act on behalf of the user.

Abuse

Sadly, server-side software started to abuse the User-Agent header. Based on the value, a web-server would respond with different output depending on what it thought the other side could handle. By accommodating a broken client and making presumptions about how it would behave, this User-Agent sniffing led to a fractured web.