html.best-practice.robots-denied.robots-denied
semgrep
Author
225
Download Count*
License
This page denies crawlers from indexing the page. Remove the robots 'meta' tag.
Run Locally
Run in CI
Defintion
rules:
- id: robots-denied
metadata:
category: best-practice
technology:
- html
license: Commons Clause License Condition v1.0[LGPL-2.1-only]
message: This page denies crawlers from indexing the page. Remove the robots
'meta' tag.
severity: INFO
languages:
- generic
pattern-either:
- pattern: <meta name="robots" ...>
- pattern: <META NAME="ROBOTS" ...>
Examples
robots-denied.html
<!DOCTYPE html>
<html lang="en-us">
<head>
<meta charset="utf-8" />
<meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1" />
<title>Elbrus Elevation | Angler's Paradise</title>
<meta name="viewport" content="width=device-width,minimum-scale=1" />
<meta name="generator" content="Hugo 0.74.3" />
<!-- ruleid: robots-denied -->
<meta name="robots" content="noindex, nofollow" />
<meta
name="description"
content="Elbrus Elevation"
/>
<link rel="stylesheet" href="/site.css" />
<link rel="shortcut icon" href="/images/favicon.png" type="image/x-icon" />
<meta property="og:title" content="Angler's Paradise" />
<meta
property="og:description"
content="Enjoy the hospitality of the Avar, Nogai, and Kumyk people as you fish the untapped potential of waters around the Caspian Sea."
/>
<meta property="og:type" content="article" />
<meta
property="og:url"
content="http://dev.elbruselevation.com.s3-website-us-west-1.amazonaws.com/tours/multi-day_adventures_anglers_paradise.html"
/>
<meta
property="article:published_time"
content="2019-12-01T19:10:36+00:00"
/>
<meta
property="article:modified_time"
content="2019-12-01T19:10:36+00:00"
/>
<meta itemprop="name" content="Angler's Paradise" />
<meta
itemprop="description"
content="Enjoy the hospitality of the Avar, Nogai, and Kumyk people as you fish the untapped potential of waters around the Caspian Sea."
/>
<meta itemprop="datePublished" content="2019-12-01T19:10:36+00:00" />
<meta itemprop="dateModified" content="2019-12-01T19:10:36+00:00" />
<meta itemprop="wordCount" content="444" />
<meta itemprop="keywords" content="fishing," />
<meta name="twitter:card" content="summary" />
<meta name="twitter:title" content="Angler's Paradise" />
<meta
name="twitter:description"
content="Enjoy the hospitality of the Avar, Nogai, and Kumyk people as you fish the untapped potential of waters around the Caspian Sea."
/>
</head>
</html>
Short Link: https://sg.run/J94Q