Change FragPipe temp directory on Linux/Unix

[GregStacey]

I'm successfully running the headless version of fragpipe, but I'm encountering a problem when submitting it via scheduler on a cluster. When submitted, the cache directory is read-only (~/.config/FragPipe/fragpipe/) and the run fails. The full error log is below.

I suspect this can be solved on the cluster side, but I'm also curious if there's a way to change the cache path in headless mode. Thanks!

2022-07-08 11:19:36,512 WARN - The output directory is not empty. Some files might be overwritten in: /scratch/st-ljfoster-1/staceyri/isabela_fragpipe/experiments/fullsearch
2022-07-08 11:19:36,580 ERROR - Error saving ui cache
java.nio.file.FileSystemException: /home/staceyri/.config/FragPipe/fragpipe/fragpipe-ui.cache: Read-only file system
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219)
at java.base/java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:478)
at java.base/java.nio.file.Files.newOutputStream(Files.java:219)
at com.dmtavt.fragpipe.api.PropsFile.save(PropsFile.java:98)
at com.dmtavt.fragpipe.Fragpipe.saveCache(Fragpipe.java:261)
at com.dmtavt.fragpipe.Fragpipe.on(Fragpipe.java:675)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.greenrobot.eventbus.EventBus.invokeSubscriber(EventBus.java:517)
at org.greenrobot.eventbus.EventBus.invokeSubscriber(EventBus.java:511)
at org.greenrobot.eventbus.AsyncPoster.run(AsyncPoster.java:46)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
2022-07-08 11:19:36,594 ERROR - Error saving runtime cache
java.nio.file.FileSystemException: /home/staceyri/.config/FragPipe/fragpipe/fragpipe-runtime.cache: Read-only file system
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219)
at java.base/java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:478)
at java.base/java.nio.file.Files.newOutputStream(Files.java:219)
at com.dmtavt.fragpipe.api.PropsFile.save(PropsFile.java:98)
at com.dmtavt.fragpipe.Fragpipe.saveCache(Fragpipe.java:269)
at com.dmtavt.fragpipe.Fragpipe.on(Fragpipe.java:675)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.greenrobot.eventbus.EventBus.invokeSubscriber(EventBus.java:517)
at org.greenrobot.eventbus.EventBus.invokeSubscriber(EventBus.java:511)
at org.greenrobot.eventbus.AsyncPoster.run(AsyncPoster.java:46)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
[SEVERE] Could not dispatch event: class com.dmtavt.fragpipe.messages.MessageSaveCache to subscribing class class com.dmtavt.fragpipe.Fragpipe
java.lang.IllegalArgumentException: File parameter 'destDir is not writable: '/home/staceyri/.config/FragPipe/fragpipe/workflows'
at org.apache.commons.io.FileUtils.requireCanWrite(FileUtils.java:2619)
at org.apache.commons.io.FileUtils.doCopyDirectory(FileUtils.java:1305)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:699)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:630)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:531)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:502)
at com.dmtavt.fragpipe.Fragpipe.saveCache(Fragpipe.java:279)
at com.dmtavt.fragpipe.Fragpipe.on(Fragpipe.java:675)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.greenrobot.eventbus.EventBus.invokeSubscriber(EventBus.java:517)
at org.greenrobot.eventbus.EventBus.invokeSubscriber(EventBus.java:511)
at org.greenrobot.eventbus.AsyncPoster.run(AsyncPoster.java:46)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
2022-07-08 11:19:36,629 ERROR - Error delivering events through the bus
java.lang.IllegalArgumentException: File parameter 'destDir is not writable: '/home/staceyri/.config/FragPipe/fragpipe/workflows'
at org.apache.commons.io.FileUtils.requireCanWrite(FileUtils.java:2619)
at org.apache.commons.io.FileUtils.doCopyDirectory(FileUtils.java:1305)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:699)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:630)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:531)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:502)
at com.dmtavt.fragpipe.Fragpipe.saveCache(Fragpipe.java:279)
at com.dmtavt.fragpipe.Fragpipe.on(Fragpipe.java:675)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.greenrobot.eventbus.EventBus.invokeSubscriber(EventBus.java:517)
at org.greenrobot.eventbus.EventBus.invokeSubscriber(EventBus.java:511)
at org.greenrobot.eventbus.AsyncPoster.run(AsyncPoster.java:46)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
2022-07-08 11:19:36,633 ERROR - java.lang.IllegalArgumentException: File parameter 'destDir is not writable: '/home/staceyri/.config/FragPipe/fragpipe/workflows'
at org.apache.commons.io.FileUtils.requireCanWrite(FileUtils.java:2619)
at org.apache.commons.io.FileUtils.doCopyDirectory(FileUtils.java:1305)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:699)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:630)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:531)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:502)
at com.dmtavt.fragpipe.Fragpipe.saveCache(Fragpipe.java:279)
at com.dmtavt.fragpipe.Fragpipe.on(Fragpipe.java:675)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.greenrobot.eventbus.EventBus.invokeSubscriber(EventBus.java:517)
at org.greenrobot.eventbus.EventBus.invokeSubscriber(EventBus.java:511)
at org.greenrobot.eventbus.AsyncPoster.run(AsyncPoster.java:46)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)

2022-07-08 11:19:37,308 ERROR - Could not save fragger_*.params file to working dir.

[fcyu]

I think so. Can you try the approach in #306 (comment) to see if it works for you?

Best,

Fengchao

[GregStacey]

Thanks for the quick response. Unfortunately I get the following error about the X11 display

(base) [staceyri@login02 isabela_fragpipe]$ JAVA_OPTS=-Djava.io.tmpdir=/scratch/st-ljfoster-1/staceyri/isabela_fragpipe/tmp /scratch/st-ljfoster-1/staceyri/fragpipe/bin/fragpipe

Exception in thread "main" java.awt.HeadlessException:
No X11 DISPLAY variable was set, but this program performed an operation which requires it.
at java.desktop/java.awt.GraphicsEnvironment.checkHeadless(GraphicsEnvironment.java:208)
at java.desktop/java.awt.Window.(Window.java:548)
at java.desktop/java.awt.Frame.(Frame.java:423)
at java.desktop/java.awt.Frame.(Frame.java:388)
at java.desktop/javax.swing.JFrame.(JFrame.java:180)
at com.dmtavt.fragpipe.FragpipeLoader.(FragpipeLoader.java:61)
at com.dmtavt.fragpipe.Fragpipe.main0(Fragpipe.java:358)
at com.dmtavt.fragpipe.FragPipeMain.main(FragPipeMain.java:40)

[fcyu]

You need to add the --headless flag (and other commands to set the workflow and manifest) so that FragPipe will run in headless mode. Without any flags, FragPipe runs in GUI mode.

Best,

Fengchao

[GregStacey]

Thanks again. I see my mistake!

Unfortunately export JAVA_OPTS=-Djava.io.tmpdir=/scratch/st-ljfoster-1/staceyri/isabela_fragpipe/tmp/ doesn't fix the error for me. I can see empty folders being made in the new tmp folder:

$ ls /scratch/st-ljfoster-1/staceyri/isabela_fragpipe/tmp/
af6fdd7f-da18-4dd1-b374-2cb499fafd15  d1d6cf4a-2265-4e16-bbaa-d6af9041dfab  e109884b-958b-468a-b85c-4f73c44d0e59  f06836aa-9c9f-4e8c-bb02-77224a71b88a
$ ls /scratch/st-ljfoster-1/staceyri/isabela_fragpipe/tmp/af6fdd7f-da18-4dd1-b374-2cb499fafd15/
$ 

But the same error persists where fragpipe tries to save files in the /home/ directory:

2022-07-08 15:11:15,601 ERROR - Error saving ui cache
java.nio.file.FileSystemException: /home/staceyri/.config/FragPipe/fragpipe/fragpipe-ui.cache: Read-only file system
...

I heard back from the cluster administrators and the problem is that /home/ doesn't exist on the compute nodes. I tried naively redefining $HOME but that also doesn't solve the issue for me. The submission script is below in case that's helpful. Again I don't see the error when just running /path/to/fragpipe on the head node.

# change temp folder
export JAVA_OPTS=-Djava.io.tmpdir=/scratch/st-ljfoster-1/staceyri/isabela_fragpipe/tmp/
# confirm temp folder is changed
echo $JAVA_OPTS

# change home folder
HOME=/scratch/st-ljfoster-1/staceyri/
echo ~

# run fragpipe
/scratch/st-ljfoster-1/staceyri/fragpipe/bin/fragpipe       --headless       --workflow /scratch/st-ljfoster-1/staceyri/isabela_fragpipe/experiments/fullsearch/fullsearch.workflow       --manifest /scratch/st-ljfoster-1/staceyri/isabela_fragpipe/experiments/fullsearch/fullsearch.fp-manifest       --workdir /scratch/st-ljfoster-1/staceyri/isabela_fragpipe//experiments/fullsearch       --ram 0       --config-msfragger /scratch/st-ljfoster-1/staceyri/fragpipe/tools/MSFragger-3.5/MSFragger-3.5.jar       --config-philosopher /arc/project/st-ljfoster-1/philosopher/v4.2.2/philosopher       --config-python /arc/software/spack/opt/spack/linux-centos7-x86_64/gcc-9.1.0/miniconda3-4.6.14-wsswwucnigbnge5edropjstrirpjx7je/bin/python  

[fcyu]

Thanks for your feedback.

I think I found the cause of the issue. In Linux and Unix, FragPipe always save cache to HOME:

FragPipe/MSFragger-GUI/src/com/github/chhh/utils/CacheUtils.java

Line 96 in add8fbe

final String dir = OsUtils.isUnix() ? XDG_CONFIG_HOME + "/FragPipe" : SYS_TEMP_DIR;

For your case, can you try setting a XDG_CONFIG_HOME=<temp path> environment variable? Details of the code can be found from

FragPipe/MSFragger-GUI/src/com/github/chhh/utils/CacheUtils.java

Line 40 in add8fbe

return System.getenv("XDG_CONFIG_HOME");

Best,

Fengchao

[GregStacey]

Thanks, Fengchao. Adding the following lines to the submission script fixed the issue for me.

XDG_CONFIG_HOME="/path/to/working/directory/"
export XDG_CONFIG_HOME

[fcyu]

Thanks for the positive feedback.

Best,

Fengchao

[nyin01]

Thanks, Fengchao. Adding the following lines to the submission script fixed the issue for me.

XDG_CONFIG_HOME="/path/to/working/directory/"
export XDG_CONFIG_HOME

Hi! I'm running into the same issue but setting XDG_CONFIG_HOME alone didn't solve the problem for me. I'm running it on a docker image. Was setting the HOME the only modification you made? Thanks!

[guoci]

What error log did you get? Since you are running on a docker container, there should be no conflicts with the OS filesystem.

[xeniorn]

@nyin01 under docker or containers in general, the best solution is to let fragpipe write where it wants within the container (/home), and simply mount a writeable directory from the os into the container at runtime

https://docs.docker.com/engine/storage/bind-mounts/