FOLLOW

Update Robots.txt in Layout & Design Tool

You can now easily allow or disallow user agents in Global Settings

Robots.txt can be updated from our Layout & Design tool. You can now easily allow or disallow user agents within Layout & Design tool's Global Settings as well.

What Is a Robots.txt File?

A robots.txt file tells search engine crawlers which pages or files the crawler can or can't request from your site. This is mainly used to avoid overloading your site with requests. It's not a mechanism for keeping a web page off of Google. To keep a web page off of Google, you should use noindex directives or protect your page with a password.

What Is Robots.txt Used For?

Robots.txt is primarily used to manage crawler traffic to your site, and usually to keep a page off of Google, depending on the file type.

If you have any questions about this feature, email support@rebelmouse.com .

layout & design tool google seo google crawler

Might be interesting for you

Shortcodes Dashboard: Dynamic Media in Just a Few Clicks

Where
Websites Are Built

The Fastest Sites in the World Run on RebelMouse

Let’s Chat

new!

RebelMouse Performance Monitoring

Real-Time Core Web Vitals

Get Started

DISCOVER MORE

Our Core Features

Our platform is a complete digital publishing toolbox that's built for modern-day content creators, and includes game-changing features such as our:

Lightning-Fast Page Speeds

Beautiful User Experiences

Organic Reach and Search Wins

Diversified Monetization Methods

Engaged User Communities

Secure, 99.99% Uptimes

Why RebelMouse?

Unprecedented Scale

RebelMouse sites reach more than 120M people a month, with an always-modern solution that combines cutting-edge technology with decades of media savvy. And due to our massive scale, 1 in 3 Americans have visited a website powered by RebelMouse.

120M+ Users

550M+ Pageviews

17+ Avg. Minutes per User

6+ Avg. Pages per User

Today's Top Websites Use RebelMouse

Thanks to the tremendous scale of our network, we are able to analyze a wealth of traffic data that informs our strategies and allows us to be a true strategic partner instead of just a vendor.

What Clients Say

We’re here to help you weigh and understand every tech and strategic decision that affects your digital presence. Spend less time managing everything yourself, and more time focused on creating the quality content your users deserve.

It’s like we’ve been riding horses to commute all our lives and someone just showed us a Tesla.

Brendan Farley

Vice President, Product Management
Strada Education Network

Thanks to RebelMouse, United doesn’t have to sacrifice great content for speed. You can have both!

Ryan Bujeker

Director, Social Media Strategy & Digital Engagement
United Airlines

We're in a much better position to make it through the current crisis in part due to RebelMouse and their platform. We've not only saved on overhead costs, we've more easily optimized our traffic with their smart measuring and SEO tools, as well as their social integrations.

Jill Braff

President/Chief Operating Officer
Brit + Co

THANK YOU for all the good, fast work you put into making our public rollout such a big success. I know we threw a lot at you very quickly, but you were calm under pressure and pulled it all off perfectly. Very glad to be working with you all.

Tim Grieve

Executive Editor
Protocol

So inspired and impressed by these results from RebelMouse. Check them out and ask Andrea Breanna how they got there! Just goes to show that spending millions doesn't necessarily get you to the right destination. It's spending no more than you need on the right technology with the right platform that does! 🌐💙🏎️

Andrey Lipattsev

Partner Development Manager
Google

It is not often I jump out of my chair and scream in a positive way, but I just did! This is the happiest I have seen my Managing Editor in 2021! Thank you.

Nick Smith

CEO
Investing News Network

Working with RebelMouse is an actual treat. From the CEO down, every single member of the team has partnered with us to meet our needs. Everyone willingly (and with endless patience!) shares their expertise and insights to ensure we optimize search and create a great audience experience. As we have implemented the various search tactics, positive results are almost instantaneous. Training our content creators has been simple. The system is incredibly user friendly. As a co-worker commented, "RebelMouse doesn't just sell you something, they partner with you to deliver exactly what you want." I couldn't agree more!

Emma Schofield

Director of Change Management
Investing News Network

We continue to be huge fans of RebelMouse and envision a very long-term relationship with the platform. The service has been impeccable.

Douglas Fischer

Executive Director
Environmental Health News

I am a HUGE RebelMouse fan, and basically tell everyone I meet who's looking to overhaul their website to get in touch right away.

Susan Hassler

Editor Emeritus
IEEE Spectrum

Every member of the RebelMouse team, but especially our project team, are superstars! Our site launch on the platform met all our expectations and we love the technology. We are positive about the future of our site and look forward to improving our approaches and processes with RebelMouse's guidance!

Terrence O'Hanlon

Founder and Chairman
Reliabilityweb

RebelMouse's Layout & Design Tool is one of the most comprehensive and flexible tools I have ever used of any CMS. Their product team is also extremely helpful and constantly supports to make sure your properties are up to date and use the best industry standards, especially in terms of page speed.

Chuck Lapointe

CEO
Narcity Media

The best thing about RebelMouse is how much they care about their customers. When working with them, I always get a sense that they go above and beyond for their customers. When working with the RebelMouse team, I very much consider them to be a partner, rather than just a vendor. Upon migrating to their technology, we have seen tremendous improvements in many key areas, particularly technical SEO. They have out-of-the-box functionality that is tremendously helpful to customers. At the same time, their system is flexible enough to allow us to adapt it to any new business needs that we have.

David Nguyen

Senior Director of Digital Marketing
Investing News Network

RebelMouse has shepherded us through the modernization phase, and has been keen, practical, and nimble throughout the entire journey. They have helped modernize our website through an infinite scroll setup that allows us more advertising real estate while creating more time on site to capture reader engagement. Plus, the RebelMouse team has helped educate our own team on the various aspects of programmatic advertising and worked to best implement them. Premier Guitar joining forces with RebelMouse in 2020 was one of the best choices we could have made!

Luke Viertel

Digital Strategist
Premier Guitar

Case Studies

A Team Built Like No Other

RebelMouse employs a unique, diverse, and decentralized team that consists of 70+ digital traffic experts across more than 25 different countries. We have no central office, and we cover every time zone to ensure that we’re able to deliver amazing results and enterprise-grade support around the clock.

Our team is well-versed in all things product, content, traffic, and revenue, and we strategically deploy ourselves to help with each element across all of our clients. We thrive on solving the complex.

Let's Chat

OUR TECHNOLOGY

OUR CREATIVE AGENCY

FOR CONTENT CREATORS

FOR DEVELOPERS

INSIGHTS

ABOUT US