Forum
Prevent Perch resources from getting crawled
Hi! I am trying to get some info on the best practices for preventing spiders from crawling the Perch resources directory. If I put an .htaccess block on the entire resources directory will it cause problems for the Perch engine itself?
We've been contacted by a person who is upset that their address is showing up in Google via some shady "People Search" site called Yasni. It seems that the Yasni bot crawled a PDF newsletter we had briefly posted on our site.
What really slays me about this is when I saw this newsletter on the site, I told the content editor that it had personal information like birthdays and addresses in it and should be pulled down. The posting was killed, but the uploaded PDF remained in the resources directory.
I don't know if Yasni crawled the link, which was up for a day at most, or if they crawled the directory itself to get to the PDF, but I want to make sure I do what I can to prevent malicious bots from crawling our back-end.
We're running Perch 2.3.3.
Firstly you need to upgrade Perch. You are very out of date. In the latest version you will have Assets Management so you can view and delete assets to make sure that if you need to remove a sensitive file, you can.
If you block the directory then Assets cannot be viewed via the web. You could use a robots.txt to block ethical engines but the solution is to not upload things you do not want on the web.
Well, that's where the canker gurgles for any CMS, right? The main problem is human. Have to keep reminding content editors that anything they post is like putting up a 30-foot billboard along every highway on the planet.
I read somewhere that putting a robots.txt file in a directory might keep the honest spiders out but has the unfortunate effect of drawing more attention from the black-hats.
Will try it anyway and will schedule the upgrade, as per your suggestion. Thanks!