User-agent: * Disallow : /api/web_request_api/src/uploads/ Disallow : *.pdf Disallow : *.xls Disallow : *.doc