User-agent: * Crawl-Delay: 20 Disallow: /gd-archive/ Disallow: /gd-archive-item/ Disallow: /gd-details/ Disallow: /location/ Disallow: /gd-login/