Apple is crawling the web to help your Siri and Spotlight searches
Apple doesn't have to rely solely on outside web providers like Google or Microsoft to fuel your iOS and Mac searches. The company has confirmed the rumored existence of Applebot, a web crawler that collects site information for the sake of Siri and Spotlight queries. It behaves much like Google's crawler, looking for the familiar "robots.txt" file that tells it what results to exclude on a given site; it'll follow typical Google instructions if there isn't any Apple-specific rule set. It's not clear how long Cupertino has been running its bot, or whether there's anything more in the works. However, it's evident that Apple wants its online searches to work no matter what its partnerships look like in the future.
http://www.engadget.com/2015/05/06/apple-web-crawler/
