Python "FileExists" error when making directory
I have several threads running in parallel from Python on a cluster system. Each python thread outputs to a directory mydir
. Each script, before outputting checks if mydir exists and if not creates it:
if not os.path.isdir(mydir):
os.makedirs(mydir)
but this yields the error:
os.makedirs(self.log_dir)
File "/usr/lib/python2.6/os.py", line 157, in makedirs
mkdir(name,mode)
OSError: [Errno 17] File exists
I suspect it might be due to a race condition, where one job creates the dir before the other gets to it. Is this possible? If so, how can this error be avoided?
I'm not sure it's a race condition so was wondering if other issues in Python can cause this odd error.
Solution 1:
As of Python >=3.2
, os.makedirs()
can take a third optional argument exist_ok
:
os.makedirs(mydir, exist_ok=True)
Solution 2:
Any time code can execute between when you check something and when you act on it, you will have a race condition. One way to avoid this (and the usual way in Python) is to just try and then handle the exception
while True:
mydir = next_dir_name()
try:
os.makedirs(mydir)
break
except OSError, e:
if e.errno != errno.EEXIST:
raise
# time.sleep might help here
pass
If you have a lot of threads trying to make a predictable series of directories this will still raise a lot of exceptions, but you will get there in the end. Better to just have one thread creating the dirs in that case