Apple confirms ‘Applebot’ web crawler is indexing the web

“Rumors that Apple might be creating its own search engine started doing the rounds last fall, when webmasters started seeing web crawler visits from IP addresses starting with 17 – the IP address block owned entirely by Apple,” Ben Lovejoy reports for 9to5Mac. “Apple has now officially confirmed in a support document that it is operating its own web crawler called Applebot.”

Applebot is the web crawler for Apple, used by products including Siri and Spotlight Suggestions. It respects customary robots.txt rules and robots meta tags. It originates in the net block. – Apple Inc. support document

“Is it likely that Apple is developing its own search engine, with plans to take on Google, Bing and Yahoo?” Lovejoy writes. “I think not, for two reasons.”

Read more in the full article here.

MacDailyNews Take: This is likely, as Lovejoy asserts, a tool for Apple to ensure that Siri and Spotlight are delivering the best results possible.

[Thanks to MacDailyNews Reader “Lynn Weiler” for the heads up.]


    1. Now, if only Apple would crawl their own website and improve the site’s incredibly lousy search engine. It remains far easier to search Apple’s website with Google than with Apple’s own website search. (0_o)

    2. SIRI and Spotlight are front ends to a search engine. Making SIRI/Spotlight searches better is a very small step away from competing directly with Google. At the very least, making SIRI/Spotlight searches better eliminates reliance on Google, and that hurts Google ad revenue.

    3. Apple need not compete “head to head” with Google at all. If people ask Siri for what they are looking for and get it promptly with little trash, why would they ever think of searching with Google. Google will be long forgotten as more and more they just ask Siri. For now most do not see this coming as Apple builds the infrastructure behind Siri, but if Google does not recognize the Freeway being constructed around their small town, they are fools…..Can you spell “Bypass”????

  1. They may not be trying to compete directly with Google, but just the fact that they will be bypassing Google in searches with Siri and Spotlight will be a huge blow for Google.

    1. I agree. The point about this approach is that when people use Siri, they don’t really think about which search engine is doing the work for them. If Apple uses it’s own search engine instead of Google, then Google will lose income from all those searches.

      It would be quite a daunting ( but not completely impossible ) task to build a stand alone search engine which could gain anything like the traction that Google has, but by taking this approach Apple doesn’t need to go head to head with Google. Greatly reducing Google’s income would be an appealing outcome for Apple.

      1. I think you are close to the secret here in those opening lines. this think about this.
        1) It helps Apple determine which search engines are being most honest and support accordingly.
        2) It additionally keeps them on their toes by giving Apple a search base option should they push the button depending upon future (sometimes unpredictable) tech/social developments.
        3) But most interestingly what a wonderful option to let those search engines compete to provide all the heavy work while Apples solution provides, initially filler or specialist responses to increasingly give unique information & services in the cloud closely linked (only) to Apples products and future technologies all accessed seamlessly through siri and spotlight which can pick and choose which and when to use them or the general search engines as Apple wishes rather like an intelligent, automatic cloud version of Sherlock.

        So for Apple and its users flexibility, scalability, potentially a superior product and yet no great overheads by re inventing the wheel and it offers a reason for buying Apple products so could be add free if desired.

        as an example of this in action Apple’s patents on user supplied information through maps. This could be exploited within a search framework in various ways and support ‘unique to Apple’ services that could supply for example restaurant, sport, business information more reliably, less subjective and more focused on Apple users than other options. The more of these specific functions Apple’s search facility offers the more like a ‘proper’ search engine it would become and the less reliance on the others, plus the more personal siri can be but at all times Apple could take it at its own pace and additionally prevent the maps type fiasco of an all in one shot solution. It would specifically for business offer that smooth widespread user interface for an IBM Watson back end, Sherlock and Watson beautifully reunited in a way there simply would be no competition for in that sphere.

      2. Unfortunately with that approach the quality of the Siri results (if Apple really makes Siri dependent solely on a home-grown search engine) will simply be compared to that of Google Now and Cortana. This will also necessitate that the browser on the desktop and iPad use the same result set or risk different results when you use Handover.

    2. During Charlie Rose’s interview, he asked who was Apple’s competition. Tim Cook named only Google. I very much expect Apple to start invading Google’s turf. Access to information without being bombarded by ads would be the most exciting future I can imagine!

  2. I believe Apple will go nearly head to head with Google within 3 years. I would wish Apple would generate text pages similar to Google without the intrusion and lame results. But somehow that does not seem Apple’s way.

    1. I don’t think so, not unless Google does something drastic like attempt to block any access by Safari, in which case Google would run afoul of anti-trust laws (there’s not much question about Google being a monopoly in search, especially when people don’t search, they “Google” something).

      However, this could be a gigantic hedge bet by Apple to guard against Google doing something stupid. Apple may have a search engine being developed which could be released quickly if necessary, but would otherwise be used to support Siri and Spotlight searching or even lease out to other search engines (Bing?).

      I think Apple is mindful of the Maps debacle, and even though Maps have greatly improved now (and weren’t as bad at first as the publicity stated), the black eye Apple suffered from the Maps release should not be forgotten. Apple Search could only do as well as Google, but it could do much worse, which would be a disastrous PR move.

  3. I disagree with the author regarding Apple have no interest in ad model. I think ad free search which preserves user privacy ( and produces search results which are not influenced by payment) would be a total bonus feature for Apple products. I would love to banish Google from all my devices permanently.

  4. I would love this. I’ve been suffering with Bing for years just to avoid Google as much as possible. At least using Bing I get a free iTunes gift card as a reward for using Bing Rewards. Lol they had to bribe me with Apple stuff to get me to use it at all.

Reader Feedback

This site uses Akismet to reduce spam. Learn how your comment data is processed.