Browsing: content credentials

For years, websites included information about what kind of crawlers were not allowed on their site with a robots.txt file.…