html.best-practice.robots-denied.robots-denied

profile photo of semgrepsemgrep
Author
225
Download Count*

This page denies crawlers from indexing the page. Remove the robots 'meta' tag.

Run Locally

Run in CI

Defintion

rules:
  - id: robots-denied
    metadata:
      category: best-practice
      technology:
        - html
      license: Commons Clause License Condition v1.0[LGPL-2.1-only]
    message: This page denies crawlers from indexing the page. Remove the robots
      'meta' tag.
    severity: INFO
    languages:
      - generic
    pattern-either:
      - pattern: <meta name="robots" ...>
      - pattern: <META NAME="ROBOTS" ...>

Examples

robots-denied.html

<!DOCTYPE html>
<html lang="en-us">
  <head>
    <meta charset="utf-8" />
    <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1" />

    <title>Elbrus Elevation | Angler&#39;s Paradise</title>
    <meta name="viewport" content="width=device-width,minimum-scale=1" />
    <meta name="generator" content="Hugo 0.74.3" />

    <!-- ruleid: robots-denied -->
    <meta name="robots" content="noindex, nofollow" />

    <meta
      name="description"
      content="Elbrus Elevation"
    />

    <link rel="stylesheet" href="/site.css" />

    <link rel="shortcut icon" href="/images/favicon.png" type="image/x-icon" />

    <meta property="og:title" content="Angler&#39;s Paradise" />
    <meta
      property="og:description"
      content="Enjoy the hospitality of the Avar, Nogai, and Kumyk people as you fish the untapped potential of waters around the Caspian Sea."
    />
    <meta property="og:type" content="article" />
    <meta
      property="og:url"
      content="http://dev.elbruselevation.com.s3-website-us-west-1.amazonaws.com/tours/multi-day_adventures_anglers_paradise.html"
    />
    <meta
      property="article:published_time"
      content="2019-12-01T19:10:36+00:00"
    />
    <meta
      property="article:modified_time"
      content="2019-12-01T19:10:36+00:00"
    />
    <meta itemprop="name" content="Angler&#39;s Paradise" />
    <meta
      itemprop="description"
      content="Enjoy the hospitality of the Avar, Nogai, and Kumyk people as you fish the untapped potential of waters around the Caspian Sea."
    />
    <meta itemprop="datePublished" content="2019-12-01T19:10:36+00:00" />
    <meta itemprop="dateModified" content="2019-12-01T19:10:36+00:00" />
    <meta itemprop="wordCount" content="444" />

    <meta itemprop="keywords" content="fishing," />
    <meta name="twitter:card" content="summary" />
    <meta name="twitter:title" content="Angler&#39;s Paradise" />
    <meta
      name="twitter:description"
      content="Enjoy the hospitality of the Avar, Nogai, and Kumyk people as you fish the untapped potential of waters around the Caspian Sea."
    />
  </head>
</html>