I am trying to sort a large bam file (a couple of hundred GB) with samtools and get an error:
open: Too many open files
What should I do to sort this file?
Your large file likely doesn’t fit in the memory on your machine so SAMTools tries to distribute the data into a large number of small files to help it do the sorting. This large number of temporary files is likely causing the problem. Here are a few things you can try
- Your OS likely has a hard limit on how many files you can have open at any one time. You can query this with « ulimit -n ». You can ask someone with permission to increase your ulimit and try again.
- If you have sufficient RAM on your machine you can use the « -m » option to « SAMTools sort » to keep more data in RAM and likely reduce the number of temporary files that get created.
I’m sure there are other options that would work, but the above are worth a try.
You can temporarily increase your ulimit value for a session up to some maximum with « ulimit -n » (the maximum value is « ulimit -Hn »). To increase the hard limit, you’ll need admin permissions and have to edit /etc/security/limits.conf.