Search Engine Optimization (SEO) Crawl Terms


If you or an out-sourced SEO “agency” ever crawled your website for an SEO audit, the following terms will eventually come up. Get to know them and how they relate to your website. Its crucial as all of them affect SEO rankings.

Title Tag: Title tags—technically called title elements define the title of a document. Title tags are often used on search engine results pages (SERPs) to display preview snippets for a given page, and are important both for SEO and social sharing. Keep your titles under 55 characters, and you can expect at least 95% of your titles to display properly. Keep in mind that search engines may choose to display a different title than what you provide in your HTML. Titles in search results may be rewritten to match your brand, the user query, or other considerations.

Meta description: Meta descriptions are HTML attributes that provide concise explanations of the contents of web pages. Meta descriptions are commonly used on search engine result pages (SERPs) to display preview snippets for a given page. Optimal Length for Search Engines ~ Roughly 155 Characters. These short paragraphs is your opportunity to advertise content to searchers and to let them know exactly whether the given page contains the information they’re looking for. Tailor the Meta description to each webpage instead of a generic description.

HTTP Status Code: HyperText Transfer Protocol (or HTTP) response status codes are returned whenever search engines or website visitors make a request to a web server. These three-digit codes indicate the response and status of HTTP requests.

Referrer: The HTTP referrer is an HTTP header field that identifies the address of the webpage that linked to the resource being requested. By checking the referrer, the new webpage can see where the request originated.

Link Count: High link count is good. Make sure some of the links go outside your website to other resources.

Content-Type Header: text/html; charset=UTF-8 is good! Choose UTF-8 for all content and consider converting any content in legacy encodings to UTF-8. A Unicode-based encoding such as UTF-8 can support many languages and can accommodate pages and forms in any mixture of those languages, etc.

4XX (Client Error): If you get one of these errors, the description says that this is either bad or a broken link on your page. Find the URLs that cause the error.

5XX (Server Error): This group of HTTP status codes indicates that the server is aware that it is on error or is incapable of performing the request. The server response usually includes an explanation of the error situation and if it is a temporary or permanent condition.

Title Missing or Empty: This is a no-brainer. You need to have Titles for each webpage on the website.

URLs with Duplicate Page Content: Duplicate page content is bad. Never have duplicate page content or Google will penalize!

Duplicate Page Title: Same with Duplicate Page Titles. No duplicate page titles Google will penalize.

URLs with Duplicate Title Tags: No duplicate URLs or Title Tags. Google will penalize.

Long URL: Keep them short. Never long URLs.

Overly-Dynamic URL: A URL that results from the search of a database-driven Web site or the URL of a Web site that runs a script. In contrast to static URLs, in which the contents of the Web page do not change unless the changes are coded into the HTML, dynamic URLs are generated from specific queries to a site’s database.

301 (Permanent Redirect): Redirection is the process of forwarding one URL to a different URL.

302 (Temporary Redirect): Redirection is the process of forwarding one URL to a different URL.

Meta Refresh: Generally it’s not good for SEO, as in it doesn’t really pass link equity. So, if the crawl test says you don’t have any, then that’s a good thing.

Title Element Too Short: Don’t have them too short (just like the rapper!).

Title Element Too Long: Don’t have them too long <insert funny joke here>.

Too Many On-Page Links: Keep them low as to not look like spam.

Missing Meta Description Tag: Basic thing. Need them on every page. The description should optimally be between 150-160 characters.

Search Engine blocked by robots.txt: All pages are open (indexed) to Google, Yahoo, Bing, etc., to search your webpages. You MAY want certain webpages blocked due and not crawled by search engines, but that’s up to you and the content you choose to block.

Blocked by meta-robots: Some might blocked and some might not be blocked. Sometimes a meta robots block comes from your CMS or something in your header. Always check on the pages that show as blocked to see if there’s any noindex sort of code in those pages.

Meta Robots Tag: The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website. In most cases, meta robots with parameters “noindex, follow” should be employed as a way to to restrict crawling or indexation.

Blocking Google: If all are No then its good! You do not want to block Google as its good for content to be seen.

Blocking Yahoo: If all are No then its good! You do not want to block Yahoo as its good for content to be seen.

Blocking Bing: If all are No then its good! You do not want to block Bling as its good for content to be seen.

Author: Online Marketing Juice

Digital + Social Media Marketing. I hold a B.A. in Economics & Marketing from the University of California. SEC and FINRA Social Media compliance best practices: HubSpot Academy Social Media Certified. "Open to new clients" Disclaimer: This is my personal blog. The views expressed on these pages are mine alone and not those of my employer.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: