As of 8:35 last night,
theorion.com returned to its newsy, speedy self after the paper purchased a new cloud hosting package from BlueHost, our web hosting company, that added capacity and a sexy new caching function. Gone (I hope forever) are the 30-second home page loads and database connection error screens.
What an awful month!
Just how important the website is to the rest of The Orion's operations became painfully apparent as the website stayed broken for most of October. Because I think it's important for everyone on the staff to know what happened and what was done to cure the problem, here's the month in a nutshell:
• On Oct. 8, BlueHost took down the website for terms of service violations. The website had too many connections, which triggered the shutdown. I removed the Twitter feed at their recommendation, and they agreed to put the site back up.
• Later that day, I paid BlueHost $99 to optimize the site. Not the solution.
• We suspected the tens of thousands of spambot comments that had been building on the site might have been the cause, so Web Editor Matt Nachtigal deleted all comments from the site. Still no improvement. He also added a captcha plugin to stop the bots.
• Our friends at Camayak noticed that The Orion's entire copy-process system wasn't working because of the errors and let us know. They also suggested shutting down Camayak to cut down on editor frustration :-)
• More calls to BlueHost support. No help and no resolution.
• The public relations team -- tasked with doing a better job of using social media -- found it had nothing to promote because stories weren't being posted on the website.
• I dove into the backend of the site one morning and noticed many image files larger than 1 megabyte. I immediately pulled them down, optimized them to be no more than 100k, and put them back up. Editors were reminded to optimize photos.
• Editor Risa Johnson then took the drastic step of pulling down all the site's images and asked section editors to optimize them and reload them through Camayak. That resulted in fewer errors, but the site was still slow and still throwing database connection errors.
• Ad manager Cortnee Uriz expressed concern at a Big 5 management meeting that site difficulties were threatening business and new ads that were set to come online Nov. 1.
• Another call to BlueHost support. They assured me everything that could be done to the database had been done and said the error was with WordPress. When I went on WordPress discussion forums, I found others with the same problem but no solution.
• Last weekend, I Googled "WordPress repair" and requested quotes from three companies to solve the problem. I hired one of them Monday morning.
• Report from quickwebsitefix.com: malware detected (the site had been hacked).
• Follow-up report: The errors were triggered by high use of the database.
• I called BlueHost support, described the problem, and this time the support person suggested that the intermittent nature of the database errors indicated capacity issues. He suggested an upgrade to cloud hosting, which would give theorion.com more resources (capacity) and smarter caching. I paid for two years of cloud hosting.
• The website was migrated from its old server last night. Risa Johnson reported speedy loads later that evening. Problem apparently solved.
I plan to ask quickwebsitefix to clean up the malware and make other repairs. Going forward, I'm hoping the section editors will take the time today and tomorrow to put the optimized images back online and Matt to do what he can to upgrade the WordPress theme and other plugins to head off any further problems.
Thanks, everyone, for your patience. I'm hoping one upside to all these problems is understanding how crucial the website is to The Orion's mission. Be aware that the focus of your work needs to shift back to theorion.com and keeping the site fresh and up to date.
