• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar

EntrepreneurMuslimMoms

Online Business ideas for moms

  • Home
  • About Us
  • Categories
    • Blogging Tips
    • Blogging
    • Business Ideas for Moms
    • women entrepreneurs
  • Contact Us
You are here: Home / Blogging / A Guide On Your WordPress Robots.txt File

A Guide On Your WordPress Robots.txt File

by Salmat Leave a Comment

Robots Robots.txt file is used by site owners to stop the search engines from crawling certain part of your blog or website like the admin dashboard, admin folder  and the rest. A Robots.txt always reside at the root of all domains. For instance your domain.com/Robot.txt
If the  search engine bots visits a  blog or website and they have limited resources to crawl the blog or website,  they may  stop crawling, and this Robots.txtcould  affect your blog or website indexing. Also,  there are many parts of your blog or website, that you may not like the search engine to crawl and when you use  robots.txt, you are directing search engine crawlers not to crawl such certain area of your blog or website. Note that this action would  speed up the crawling process of your blog or website and  also help the  deep crawling of your inner pages.

To check your domain robots.txt file  go to www.domain.com/robots.txt.  In most blogging platforms you may see a blank robots.txt file but you can also check your domain Robots.txt file using Google webmaster tools.
To do this Google webmaster tool
Go to site configuration
Click on Crawler Access
To  avoid duplicate content the structure should be like this;
User-agent:  *
Disallow: /wp-
Disallow:  /trackback/
The robot.txt file is only going to stop the search engines from crawling your blogs or websites but not the indexing parts. It will only prevent  aspects like your  feeds, trackbacks, admin folder as well as your  pages and comments from being crawled. If you have some pages or posts you do not want the search engines to index, you can make use of the Yoast SEO plugin to add the Noindex tags to the pages or posts you do not want the search engines to index on your blog or website.
Hope this help? Do share!

 

60

SHARES
facebook Share on Facebook
Twitter Tweet
Follow Follow us
custom Share
custom Share
custom Share
custom Share
custom Share

Filed Under: Blogging Tagged With: google crawling rate, google index, Robots.txt

Reader Interactions

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Primary Sidebar

Hello! Welcome to my Blog!.

deesbadmus image

I’m Modupesalmat Badmus (Entremum), am a mom, a side hustler and a full time Muslimah momblogger. I love helping  women create,grow businesses they have passion for via Blogging|Online Business Ideas|Affiliate marketing|Pinterest Stategies and Online Course Creation Tips that i share on this blog.

You can find me on:

Pinterest|Facebook|Twitter|Instagram

Instagram
Facebook
Facebook
fb-share-icon
Twitter
Visit Us
Follow Me
Tweet
Pinterest
Pinterest
fb-share-icon
LinkedIn
LinkedIn
Share
RSS
Follow by Email

Can’t Find What You Are Searching For? Try Google Search

Make your start 100% hassle free!
Get new posts by email

RSS Get New Updates In Your Inbox!

  • Monetizing Your New Blog
  • Differences Between Blogger And WordPress
  • How To Create A Blog With WordPress
  • Blogging Mistakes You Should Not Make

Copyright © 2021 · https://www.entrepreneurmuslimmoms.com/

This website uses cookies to improve your experience. We'll assume you're ok with this, but you can opt-out if you wish. Cookie settingsACCEPT
Privacy & Cookies Policy

Privacy Overview

This website uses cookies to improve your experience while you navigate through the website. Out of these cookies, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may have an effect on your browsing experience.
Necessary
Always Enabled

Necessary cookies are absolutely essential for the website to function properly. This category only includes cookies that ensures basic functionalities and security features of the website. These cookies do not store any personal information.

Non-necessary

Any cookies that may not be particularly necessary for the website to function and is used specifically to collect user personal data via analytics, ads, other embedded contents are termed as non-necessary cookies. It is mandatory to procure user consent prior to running these cookies on your website.