Mar 31, 2019 I worked with @leanaddict a couple months ago to identify that we've been having some pretty severe google crawling issues, which has led to a TON of lost traffic/potential. Looks like this has been going on for a year now... i could really use some help if any of you are familiar with server admin stuff-- The issue: We're receiving 403 server errors when google tries to crawl the site -- robots.txt is good, meta tags are good, etc etc. Based on everything I’ve been looking into and reading, it seems like it may be a communication issue on the server, which i know very little about. https://aw-snap.info/file-viewer/?p...ioneighty.com/&ref_sel=none&ua_sel=gbot2&fs=0 ^confirms a 403 crawler error Tried working with my server admin @Boolo but he's s--- the bed.. PM me if you can help. @BestMembers
Mar 31, 2019 Those errors are shady. You sure this site has the programming right? 403 is a forbidden error - have you checked your robots.txt file to ensure you have not blocked crawler access? You could also try fetching the page as Googlebot to see what response you get.
Mar 31, 2019 ...leads to a link that says the old "fetch as google" tool is no longer available and has been replaced by the "URL Inspection" tool, but that tool is absolute s---. if you search google for "sectioneighty", then our root domain doesn't even show in the results. robots.txt: Code: Sitemap: https://forum.sectioneighty.com/sitemap.php User-agent: * Disallow: /account/ Disallow: /members/ Disallow: /conversations/ Disallow: /online/ Disallow: /account/ Disallow: /attachments/ Disallow: /goto/ Disallow: /login/ Disallow: /best-posts/ Disallow: /profile-posts/ Disallow: /posts/ Disallow: /misc/style?* Disallow: /misc/quick-navigation-menu?* Disallow: /lost-password/ Disallow: /help/ Disallow: /poll/ Disallow: /likes/ Disallow: /sportsbook/ Disallow: /search/ Disallow: /private.f175/ Disallow: /admin.php Disallow: /search/ Disallow: /express/ Disallow: /off-topic.f12/ Allow: /
Mar 31, 2019 i really don't know much on the server side at all, but i know that boolo at least claimed to have done that a couple times, along with a few other small things but came up empty.
Mar 31, 2019 Strange that goglebot isn't working either. You might need to go on another forum for this one, usually that works most of the time.
Mar 31, 2019 any idea on where's best to go for help? google specific forums haven't been great, aside from a couple people confirming it's a 403 server error, but unclear as to why
Mar 31, 2019 Yeah that’s odd. I don’t know enough these days to go in the system and mess with it and make it worse tbh. I can do some more into depth research this evening for you at least. You don’t think some other forum or forums purposely bugged sxn80 so it doesn’t show up on google? Ik it’s a reach, but there’s past events where forums/subs went to that extent to try and gain more attraction than their competitors.
Mar 31, 2019 Stack exchange might help. Got some good answers from there a few years ago. Gotta keep looking.
Apr 1, 2019 I think there's a redirect on https://forum.sectioneighty.com/robots.txt If that's where it's located? Robots.txt can't be in a sub folder but they can be on a sub domain.
Apr 1, 2019 Nvm - it didn't redirect the second time I tried to access that url Another suggestion since my first one was useless: rename robots.txt to robots1.txt to troubleshoot. That will make crawlers ignore robots.txt completely. If it's still broken, you know it's not robots.txt.
Apr 6, 2019 When did you start noticing this? Surely something has changed for this to happen. Can the web admin/server hoster check logs and see if he can see these errors? Maybe they installed or beefed up the firewall at some stage or changed product for this to start? The robots.txt looks OK and I assume that hasn't changed either which makes things point more to your server host. 403 means: The request was valid, but the server is refusing action. The user might not have the necessary permissions for a resource, or may need an account of some sort. @Slyk
Apr 9, 2019 i started noticing it shortly after gaining possession of the site (once koolo stepped down, in october/november-ish)....i didn't have the tools to personally look into it until that point. i did some research into some historical google/ranking data and it led me to believe that it has been happening for a while. As of this past month, i no longer have a web admin... boolo/saladin went totally silent on me out of nowhere. he's either dead, in jail, or a piece of s---. he was getting paid and vanished without even completing the time that i already paid him for in advance. it's definitely possible that it's something on that end...again, i just don't know much about that to investigate myself. i did change the robots.txt file when looking into this, but i only further opened it -- i'm not sure why some of the additional restrictions were put in place by koolo in the past, so i freed up a handful (i didn't restrict further though).
Apr 9, 2019 what's also interesting, and likely related, is that the ads on the homepage only show sometimes. at least for me... anybody else notice the same thing?
Apr 9, 2019 thats normal, you will never have 100% ads coverage, let me look into the webmaster stuff
Apr 9, 2019 @Slyk its indexing fine what is the problem exactly? fetching is working fine...if you see tons of fetch issue on a certain old date thats normal..it can happen if the server was down