Cache size grows with no bounds for large repos

Created by: rmaz

The dir artifact cache is set to clean up either when 2/3 of the cache size has been written to per buck invocation, or as a cleanup operation at the end of each buck invocation. The latter operation is given a 10 second timeout. When the cache size is set sufficiently large, say 10GB, the first condition is unlikely to be met. When the build targets are of a certain size, every build will generate more files than can be indexed and sorted before 10 seconds have elapsed. This results in the cache always growing, as the delete files operation will never have time to complete its cache indexing and start actually deleting files.

The quick solution here is to increase the diskioexecutor timeout to something greater than 10 seconds and to make this parameter customisable via config settings or runtime flags, but open to better suggestions here? Does it ever make sense to kill the cleanup operation at the end of each invocation, should this always be run under certain conditions, say > x files cached?