Hi
This can be done by identifying the IP sets that search engines use, so if the visitor is not in the list, it will be blocked.
This has to be implemented via .htaccess file, as you dont want to replace XML file with a PHP file (that also generates XML) because you'll miss some search results until the newer file is parsed.
Please feel free to contact me.
Best regards,
Baris