Room for PBS question

Secret

11:26

0

Q: How to extract the values of the resources specified in a PBS directive so that they can be mapped into other environment variables?

This question arises when trying to find a script in my host computing cluster that can allow more convenient management of quantum chemical calculations on Gaussian. The user manual of my host gives the following: Gaussian 09 offers an alternative way of passing to the program values for the...

scripts

terdon

Cool. So, why do you think there will be any such variables? The manual seems clear enough: there are three that you can use.

What's wrong with using the PBS commands as in your example script?

Secret

The issue is that without that mapping to gaussian environment variables, you have to make sure the %Mem, %Maxdisk and %nprocs need to match the values of mem, jobfs and ncpus for every input file you fed into the script.

In the past month doing that manually for 18 calculations is still doable, but later on we are going to do at least 120 calculations, thus it is better to automate this process as that will dramatically cut down the chance of making input errors due to the manual effort needed to proofread 120+ input files for those same 3 lines (where some calculations are more lengthy t…

terdon

My point is that it looks like this gaussian thing you are using only offers three env vars for this. If there were any more, they would have been mentioned in the manual right?

Secret

Those 3 are all we need. The idea is that, when they are mapped with the values we need, we don't need to enter the lines %mem, %nprocs and %Maxdisk for each input file, meaning that the resource requirement to be imposed in a given gaussian calculation can be done on the script.

terdon

@Secret Oh. In that case, I don't understand what you're missing.

Secret

11:35

but the issue is, we need a way to ensure the values match up in both the PBS and the gaussian envrionemt variables so that we only need to specify them once and the script will d the rest for us, but it is not sure what is the correct route to do that since PBS does not give environment variables except for PBS_NCPUS which corresponds to the no. of processors used for the job (which need to be matched with the value in %nprocs, which is stored by the gaussian environment variable GAUSS_PREF

terdon

@Secret What PBS variables? You mean the commands set in the script?

@Zanna what is that command trying to do? And grep has an -l option which makes it print only the name of the file (if that's what you were doing)

Secret

For this part of the script stores the aforementioned 3 resources
#PBS -l ncpus=4
#PBS -l mem=4GB
#PBS -l jobfs=100GB
The environment variable PBS_NCPUS stores the number 4, thus I can write export GAUSS_PREF=$PBS_NCPUS to map the 4 to the GAUSS_PREF, which gaussian will then write %nproc=4 for the job when it is executed

There are however no other environment variables that stores 4GB and 100GB, thus another way is needed to get those values in the script

terdon

@Secret Why don't you just define the variables then? I think I'm missing something. I mean, just run PBS_MEM=4GB and there you have your env variable.

Secret

The issue is that PBS seemed to unable to read environment variables within the PBS directives

e.g. doing the following
#PBS -l mem=$PBS_MEM
or
#PBS -l mem=PBS_MEM
will give an error

terdon

Ah, now we're getting somewhere. OK, first, are those variables defined and exported?

Secret

11:44

yes. those gaussian variables are exported in the bash by simply using export (some gaussian variable)

terdon

@Secret What bash? How are they exported? From what file?

← 17 messages moved from Ask Ubuntu General Room

OK, so where are these variables being set?

And yes, it is quite possible that you can't use env variables here, but let's just try first.

Secret

If I understood correctly, when logging into the computing cluster via ssh, the code is done under the bash shell, so are any scripts wrote there (unless you specify another shell)

The variables have default values when not being set which is stored somewhere in gaussian (I need to check on that one again), but as one wrote the script that I showed as an example, the variables can be set in the folder where you use export (some gaussian variable) (in my case, is the home folder)

terdon

This sort of thing is actually quite complex. Everything depends on i) exactly where, which file, they are being exported from; ii) what command is being used to set/export them; iii) whether or not the PBS interpreter is capable of reading shell variables.

So, I would need to know precisely where and how they are being set. Show me the output of this command:

grep PBS_MEM  ~/.bashrc ~/.profile ~/.bash_profile ~/bash.login ~/.bash_aliases /etc/bash.bashrc /etc/profile /etc/profile.d/* /etc/environment 2> /dev/null

Secret

Uh, it doesn't seemed to do anything when I execute that directly on the mac terminal command line

Here's the stuff currently shown in my command line:

[hy1816@raijin6 ~]$ grep PBS_MEM ~/.bashrc ~/.profile ~/.bash_profile ~/bash.login ~/.bash_aliases /etc/bash.bashrc /etc/profile /etc/profile.d/* /etc/environment 2> /dev/null
[hy1816@raijin6 ~]$

(and yes, the dollar sign is part of the user identifer, thus not typed by me)

terdon

Wait, what? You're using a mac?

Secret

11:55

Yeah, and I recall that the mac terminal is basically the same as linux

terdon

No. No it isn't.

And this makes your question completely and utterly off topic.

This is Ask Ubuntu for crying out loud!

So not even non-Ubuntu Linux is on topic here.

But aren't you doing all this on a cluster?

What is the cluster running? Surely not OSX!

@Secret ?

Secret

I am sorry for the trouble, umm, where should I migrate my question to, is their a linux SE?

and yes, everything is done on the cluster (I am currently looking up for the system specs of raijin (the name of the cluster that I am on)

terdon

And what OS is the cluster running?

(yes there is a Unix & Linux, luckily I'm a mod there too :)

Secret

That's what I am currently looking: So far I see the processors and cores, but no info on its OS yet, give me some moment when I continue to look that up)

terdon

But what OS is the cluster running? Everything we've discussed needs to be run on the cluster. Your laptop is completely irrelevant.

@Secret Run cat /etc/os-release

Secret

12:01

Uh...

[hy1816@raijin6 ~]$ cat /etc/os-release
cat: /etc/os-release: No such file or directory
[hy1816@raijin6 ~]$ cd /
[hy1816@raijin6 /]$ cat /etc/os-release
cat: /etc/os-release: No such file or directory
[hy1816@raijin6 /]$

terdon

OK. How about:

uname -a

Secret

[hy1816@raijin6 /]$ uname -a
Linux raijin6 3.10.0-327.36.3.el6.x86_64 #1 SMP Wed Jan 25 05:38:43 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
[hy1816@raijin6 /]$

Ok, so its a linux system

terdon

Oh wow. A very, very, very old Linux system.

OK, so, what is the output of that grep command on the cluster?

Anything I ask about should always be run on the cluster, forget your macbook.

(in the meantime, I'll migrate the question but I'm afraid you haven't explained the issue very clearly)

Secret

[hy1816@raijin6 /]$ grep PBS_MEM ~/.bashrc ~/.profile ~/.bash_profile ~/bash.login ~/.bash_aliases /etc/bash.bashrc /etc/profile /etc/profile.d/* /etc/environment 2> /dev/null
[hy1816@raijin6 /]$

It still gives nothing

terdon

And what does echo $PBS_MEM print?

I mean:

echo $PBS_MEM

Also tell me what this prints:

echo $SHELL; ps -p $$

Secret

12:07

[hy1816@raijin6 /]$ echo $PBS_MEM

[hy1816@raijin6 /]$
i.e., empty string

[hy1816@raijin6 /]$ echo $SHELL; ps -p $$
/bin/bash
PID TTY TIME CMD
12299 pts/30 00:00:00 bash
[hy1816@raijin6 /]$

which... I think I don't understand what those 4 words mean

terdon

@Secret It's OK, that just shows that yes, you are running bash but no, there is no such variable as PBS_MEM

So of course it's not working, the variable is undefined.

Secret

PBS_NCPUS is also empty, but when it is placed after #PBS -l ncpus=2, echo $PBS_NCPUS gives 2 as expected. Thus at least PBS_NCPUS is a variable (consistent with the PBS Professional User Guide)

Ok, so that means we need another way to extract the values, now I am not so sure other than a rather brute force method of scanning the script itself and stop at the correct lines and get the numbers there and store them in the gaussian environment variable

But I felt brute forcing that is quite risky, as it is quite sensitive to the content of the script itself (including where all the characters are). That's why I am wondering whether there's a neater way to acheive the goal

Btw the following is the script I am currently fixing before I head to this chat room:

#PBS -P g23
#PBS -q normal
#PBS -l walltime=00:10:00
#PBS -l ncpus=2
#PBS -l mem=800MB
#PBS -l jobfs=1GB
#PBS -l software=g09
#PBS -j oe
#PBS -l wd
#PBS -V

export GAUSS_PDEF=${PBS_NCPUS}
export GAUSS_MDEF=${PBS_MEM}
export GAUSS_RDEF=${PBS_JOBFS}

echo $PBS_NCPUS
echo $PBS_MEM
echo $PBS_JOBFS
echo $GAUSS_PDEF
echo $GAUSS_MDEF
echo $GAUSS_RDEF
echo -----

when the script is ran, the following is outputted:

2

/jobfs/local/4241143.r-man2
2

/jobfs/local/4241143.r-man2
-----

Therefore, only NCPUS have stored and mapped the correct value to the corresponding gaussian variable

terdon

12:30

@Secret But what happens if you run echo $PBS_NCPUS from a terminal?

Doesn't that also print 2?

Secret

[hy1816@raijin6 ~]$ echo $PBS_NCPUS

[hy1816@raijin6 ~]$

No, it is empty, suggesting it only exists when the PBS is running

You also cannot really run PBS directives without the qsub command, because # in bash are comments

terdon

MIght be, might be. I am guessing PBS has its own user and that variable is being set there.

@Secret I know.

I've used a similar cluster a few years ago.

But I don't know the details. In fact, this sort of thing would probably be better addressed in a forum dedicated to PBS.

Personally, I would just write a file that has lines like this:

PBS_NCPUS=2
PBS_PMEM=WHATEVER
PBS_FOO=BAR

Then, in your scripts, you have:

. /path/to/that/file

After that, you have defined the variables and can use them as you like to map to their gaussian equivalents.

Secret

Hmm, sounds nice and simple. Should try it now

Btw, thanks for helping me to understand it. I am still quite a beginner in linux stuff thus apologies for my relative lack of knowledge on the data structures

terdon

@Secret That's OK. We've all been there.

Transcript for

Room for PBS question