Over the weekend we implemented a fix to our Crawler which addressed the central issue and greatly improved the crawl throughput. The crawlers have been crawling happily along since then and we're now almost through the queue which means crawls are no longer delayed. We thank you for your patience and once again apologize for any inconvenience.

We're still working on catching up with the crawl queue. We have generally identified what the issues are, and are implementing multiple strategies for improving the throughput of the crawler.

We are currently experiencing a larger than usual crawl queue, which means some crawls will be delayed. We're looking into the situation with priority.