Job submission and result retrieval fails and server error logs fill with 'Too many open files' Exceptions

A server 'event' appeared to occur on Monday 13th October 2008. Excerpts from the vamsas.log file for this event are in additional information.

lsof wasn't used on the process to verify the problem - the service was simply restarted. This took some time, due to a particularly high load on the box (around 15). It is unclear if this event was due to side effects from other services (e.g. the cruisecontrol service under the ws-dev1 user which was started around the same time the problem was observed).



****** ADDITIONAL INFORMATION ******

ERROR 2008-10-13 14:09:11,260 TP-Processor38 vamsas.secstrpred.jpred3 Error whil
st handling /fc_gpfs/gjb_lab/www-jpred/results/jp_YtSdQW2
java.io.FileNotFoundException: /fc_gpfs/gjb_lab/www-jpred/results/jp_YtSdQW2/LOG
 (Too many open files)
        at java.io.FileInputStream.open(Native Method)
        at java.io.FileInputStream.<init>(FileInputStream.java:106)
        at java.io.FileReader.<init>(FileReader.java:55)
        at vamsas.WebService.fileMatches(WebService.java:307)
        at vamsas.secstrpred.jpred3.getresult(jpred3.java:379)
        at sun.reflect.GeneratedMethodAccessor116.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:25)
..
WARN 2008-10-13 14:20:26,157 TP-Processor34 vamsas.SungridJob Exception for qsub
  -N clustalw37653 -l h_cpu=01:00:00,h_vmem=1024M -P webservices -e /fc_gpfs/g
jb_lab/ws-dev1/jobs/JalviewWS/MsaWS-Jobs/clustalw37653/STDERR -o /fc_gpfs/gjb_la
b/ws-dev1/jobs/JalviewWS/MsaWS-Jobs/clustalw37653/STDOUT
java.io.IOException: java.io.IOException: Too many open files
        at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
        at java.lang.ProcessImpl.start(ProcessImpl.java:65)
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:451)
        at java.lang.Runtime.exec(Runtime.java:591)
..
WARN 2008-10-13 14:28:02,625 TP-Processor33 vamsas.SungridJob Exception for qsub
  -N clustalw37658 -l h_cpu=01:00:00,h_vmem=1024M -P webservices -e /fc_gpfs/g
jb_lab/ws-dev1/jobs/JalviewWS/MsaWS-Jobs/clustalw37658/STDERR -o /fc_gpfs/gjb_la
b/ws-dev1/jobs/JalviewWS/MsaWS-Jobs/clustalw37658/STDOUT
java.io.IOException: java.io.IOException: Too many open files
        at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
        at java.lang.ProcessImpl.start(ProcessImpl.java:65)
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:451)
        at java.lang.Runtime.exec(Runtime.java:591)
        at java.lang.Runtime.exec(Runtime.java:429)
        at java.lang.Runtime.exec(Runtime.java:326)
..
WARN 2008-10-13 14:29:45,775 TP-Processor47 vamsas.SungridJob Exception for qsub
  -N clustalw37659 -l h_cpu=01:00:00,h_vmem=1024M -P webservices -e /fc_gpfs/g
jb_lab/ws-dev1/jobs/JalviewWS/MsaWS-Jobs/clustalw37659/STDERR -o /fc_gpfs/gjb_la
b/ws-dev1/jobs/JalviewWS/MsaWS-Jobs/clustalw37659/STDOUT
java.io.IOException: java.io.IOException: Too many open files
        at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
        at java.lang.ProcessImpl.start(ProcessImpl.java:65)
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:451)
        at java.lang.Runtime.exec(Runtime.java:591)
        at java.lang.Runtime.exec(Runtime.java:429)
        at java.lang.Runtime.exec(Runtime.java:326)
        at vamsas.SungridJob.Submitjob(SungridJob.java:446)
        at vamsas.SungridJob.Submitjob(SungridJob.java:408)
        at vamsas.SungridJob.Submitjob(SungridJob.java:399)
        at vamsas.msa.ClustalWS.align(ClustalWS.java:142)
WARN 2008-10-13 14:32:47,376 TP-Processor21 vamsas.SungridJob Exception for qsub
  -N muscle37660 -l h_cpu=01:00:00,h_vmem=1024M -P webservices -e /fc_gpfs/gjb
_lab/ws-dev1/jobs/JalviewWS/MsaWS-Jobs/muscle37660/STDERR -o /fc_gpfs/gjb_lab/ws
-dev1/jobs/JalviewWS/MsaWS-Jobs/muscle37660/STDOUT
java.io.IOException: java.io.IOException: Too many open files
        at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
        at java.lang.ProcessImpl.start(ProcessImpl.java:65)
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:451)
        at java.lang.Runtime.exec(Runtime.java:591)
        at java.lang.Runtime.exec(Runtime.java:429)
        at java.lang.Runtime.exec(Runtime.java:326)
        at vamsas.SungridJob.Submitjob(SungridJob.java:446)
        at vamsas.SungridJob.Submitjob(SungridJob.java:408)
        at vamsas.SungridJob.Submitjob(SungridJob.java:399)
        at vamsas.msa.MuscleWS.align(MuscleWS.java:107)
..
WARN 2008-10-13 14:33:59,944 TP-Processor18 vamsas.SungridJob Exception for qsub
  -N muscle37661 -l h_cpu=01:00:00,h_vmem=1024M -P webservices -e /fc_gpfs/gjb
_lab/ws-dev1/jobs/JalviewWS/MsaWS-Jobs/muscle37661/STDERR -o /fc_gpfs/gjb_lab/ws
-dev1/jobs/JalviewWS/MsaWS-Jobs/muscle37661/STDOUT
java.io.IOException: java.io.IOException: Too many open files
        at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
        at java.lang.ProcessImpl.start(ProcessImpl.java:65)
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:451)
        at java.lang.Runtime.exec(Runtime.java:591)
        at java.lang.Runtime.exec(Runtime.java:429)
        at java.lang.Runtime.exec(Runtime.java:326)
        at vamsas.SungridJob.Submitjob(SungridJob.java:446)
        at vamsas.SungridJob.Submitjob(SungridJob.java:408)
        at vamsas.SungridJob.Submitjob(SungridJob.java:399)
        at vamsas.msa.MuscleWS.align(MuscleWS.java:107)