GPTBot – OpenAI’s Web Crawler Use robots.txt to configure OpenAI GPTBot access to your website. openai ai web-crawler web 2023 Aug. 10