In this post Barry Schwartz draws our attention to the recommendation given by Google's John Mueller in a Google Webmaster Help thread: when possible, keep the robots.txt file as simple as possible; in some cases you can remove the robots.txt file completely if it is not needed, but always make sure to keep it under 500KB.