Uploaded image for project: 'Jalview'
  1. Jalview
  2. JAL-365

Job submission and result retrieval fails and server error logs fill with 'Too many open files' Exceptions

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: web service bug
    • Labels:
      None
    • Mantis ID:
      38703

      Description

      A server 'event' appeared to occur on Monday 13th October 2008. Excerpts from the vamsas.log file for this event are in additional information.

      lsof wasn't used on the process to verify the problem - the service was simply restarted. This took some time, due to a particularly high load on the box (around 15). It is unclear if this event was due to side effects from other services (e.g. the cruisecontrol service under the ws-dev1 user which was started around the same time the problem was observed).



      ****** ADDITIONAL INFORMATION ******

      ERROR 2008-10-13 14:09:11,260 TP-Processor38 vamsas.secstrpred.jpred3 Error whil
      st handling /fc_gpfs/gjb_lab/www-jpred/results/jp_YtSdQW2
      java.io.FileNotFoundException: /fc_gpfs/gjb_lab/www-jpred/results/jp_YtSdQW2/LOG
       (Too many open files)
              at java.io.FileInputStream.open(Native Method)
              at java.io.FileInputStream.<init>(FileInputStream.java:106)
              at java.io.FileReader.<init>(FileReader.java:55)
              at vamsas.WebService.fileMatches(WebService.java:307)
              at vamsas.secstrpred.jpred3.getresult(jpred3.java:379)
              at sun.reflect.GeneratedMethodAccessor116.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
      sorImpl.java:25)
      ..
      WARN 2008-10-13 14:20:26,157 TP-Processor34 vamsas.SungridJob Exception for qsub
        -N clustalw37653 -l h_cpu=01:00:00,h_vmem=1024M -P webservices -e /fc_gpfs/g
      jb_lab/ws-dev1/jobs/JalviewWS/MsaWS-Jobs/clustalw37653/STDERR -o /fc_gpfs/gjb_la
      b/ws-dev1/jobs/JalviewWS/MsaWS-Jobs/clustalw37653/STDOUT
      java.io.IOException: java.io.IOException: Too many open files
              at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
              at java.lang.ProcessImpl.start(ProcessImpl.java:65)
              at java.lang.ProcessBuilder.start(ProcessBuilder.java:451)
              at java.lang.Runtime.exec(Runtime.java:591)
      ..
      WARN 2008-10-13 14:28:02,625 TP-Processor33 vamsas.SungridJob Exception for qsub
        -N clustalw37658 -l h_cpu=01:00:00,h_vmem=1024M -P webservices -e /fc_gpfs/g
      jb_lab/ws-dev1/jobs/JalviewWS/MsaWS-Jobs/clustalw37658/STDERR -o /fc_gpfs/gjb_la
      b/ws-dev1/jobs/JalviewWS/MsaWS-Jobs/clustalw37658/STDOUT
      java.io.IOException: java.io.IOException: Too many open files
              at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
              at java.lang.ProcessImpl.start(ProcessImpl.java:65)
              at java.lang.ProcessBuilder.start(ProcessBuilder.java:451)
              at java.lang.Runtime.exec(Runtime.java:591)
              at java.lang.Runtime.exec(Runtime.java:429)
              at java.lang.Runtime.exec(Runtime.java:326)
      ..
      WARN 2008-10-13 14:29:45,775 TP-Processor47 vamsas.SungridJob Exception for qsub
        -N clustalw37659 -l h_cpu=01:00:00,h_vmem=1024M -P webservices -e /fc_gpfs/g
      jb_lab/ws-dev1/jobs/JalviewWS/MsaWS-Jobs/clustalw37659/STDERR -o /fc_gpfs/gjb_la
      b/ws-dev1/jobs/JalviewWS/MsaWS-Jobs/clustalw37659/STDOUT
      java.io.IOException: java.io.IOException: Too many open files
              at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
              at java.lang.ProcessImpl.start(ProcessImpl.java:65)
              at java.lang.ProcessBuilder.start(ProcessBuilder.java:451)
              at java.lang.Runtime.exec(Runtime.java:591)
              at java.lang.Runtime.exec(Runtime.java:429)
              at java.lang.Runtime.exec(Runtime.java:326)
              at vamsas.SungridJob.Submitjob(SungridJob.java:446)
              at vamsas.SungridJob.Submitjob(SungridJob.java:408)
              at vamsas.SungridJob.Submitjob(SungridJob.java:399)
              at vamsas.msa.ClustalWS.align(ClustalWS.java:142)
      WARN 2008-10-13 14:32:47,376 TP-Processor21 vamsas.SungridJob Exception for qsub
        -N muscle37660 -l h_cpu=01:00:00,h_vmem=1024M -P webservices -e /fc_gpfs/gjb
      _lab/ws-dev1/jobs/JalviewWS/MsaWS-Jobs/muscle37660/STDERR -o /fc_gpfs/gjb_lab/ws
      -dev1/jobs/JalviewWS/MsaWS-Jobs/muscle37660/STDOUT
      java.io.IOException: java.io.IOException: Too many open files
              at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
              at java.lang.ProcessImpl.start(ProcessImpl.java:65)
              at java.lang.ProcessBuilder.start(ProcessBuilder.java:451)
              at java.lang.Runtime.exec(Runtime.java:591)
              at java.lang.Runtime.exec(Runtime.java:429)
              at java.lang.Runtime.exec(Runtime.java:326)
              at vamsas.SungridJob.Submitjob(SungridJob.java:446)
              at vamsas.SungridJob.Submitjob(SungridJob.java:408)
              at vamsas.SungridJob.Submitjob(SungridJob.java:399)
              at vamsas.msa.MuscleWS.align(MuscleWS.java:107)
      ..
      WARN 2008-10-13 14:33:59,944 TP-Processor18 vamsas.SungridJob Exception for qsub
        -N muscle37661 -l h_cpu=01:00:00,h_vmem=1024M -P webservices -e /fc_gpfs/gjb
      _lab/ws-dev1/jobs/JalviewWS/MsaWS-Jobs/muscle37661/STDERR -o /fc_gpfs/gjb_lab/ws
      -dev1/jobs/JalviewWS/MsaWS-Jobs/muscle37661/STDOUT
      java.io.IOException: java.io.IOException: Too many open files
              at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
              at java.lang.ProcessImpl.start(ProcessImpl.java:65)
              at java.lang.ProcessBuilder.start(ProcessBuilder.java:451)
              at java.lang.Runtime.exec(Runtime.java:591)
              at java.lang.Runtime.exec(Runtime.java:429)
              at java.lang.Runtime.exec(Runtime.java:326)
              at vamsas.SungridJob.Submitjob(SungridJob.java:446)
              at vamsas.SungridJob.Submitjob(SungridJob.java:408)
              at vamsas.SungridJob.Submitjob(SungridJob.java:399)
              at vamsas.msa.MuscleWS.align(MuscleWS.java:107)


        Attachments

          Activity

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            jprocter Jim Procter
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

              Dates

              Created:
              Updated: