User-agent: * Allow: / Allow: /?hl= Allow: /*?hl= #block filters (forever) Disallow: /*?* #block API Disallow: /api/* #block crawler User-agent: AhrefsBot User-agent: SemrushBot User-agent: BLEXBot User-agent: dotbot User-agent: MJ12bot User-agent: Barkrowler User-agent: Optimizer Disallow: /