I am trying to sort a large bam file (a couple of hundred GB) with samtools and get an error:
open: Too many open files
What should I do to sort this file?
Your large file likely doesn’t fit in the memory on your machine so SAMTools tries to distribute the data into a large number of small files to help it do the sorting. This large number of temporary files is likely causing the problem. Here are a few things you can try
- Your OS likely has a hard limit on how many files you can have open at any one time. You can query this with “ulimit -n”. You can ask someone with permission to increase your ulimit and try again.
- If you have sufficient RAM on your machine you can use the “-m” option to “SAMTools sort” to keep more data in RAM and likely reduce the number of temporary files that get created.
I’m sure there are other options that would work, but the above are worth a try.
You can temporarily increase your ulimit value for a session up to some maximum with “ulimit -n ” (the maximum value is “ulimit -Hn”). To increase the hard limit, you’ll need admin permissions and have to edit /etc/security/limits.conf.