Ronggang Ping pingrg@ihep.ac.cn
Shijie Wang sjwang@ihep.ac.cn
It's a very primary introduction of how to submit a gpu job to conduct a simple FDC-PWA fitting analysis, taking the example of . There is not any physics principle included.
Fundemental environment to conduct fdc-pwa analysis.
If you don't have an account for gpu jobs or you don't have a personal directory under gpupwa/, you need to send an email to dongly@ihep.ac.cn and apply for them, providing your lxslc account. When your application is approved, you should be able to accesss the directory:
/hpcfs/bes/gpupwa/$username/
Since R.G. Ping has configured everything you need for fdc-pwa under his directory, you only need to access to his environment. Create a file .login in which you should set environmental variables as follows (taking the example of zsh):
$ cd ~ $ vim .login # Write the follows in file .login export fdc="/besfs5/groups/psip/psipgroup/public/pingrg/fdc-pwa3.2dbg/" export PATH=$PATH:/besfs5/groups/psip/psipgroup/public/pingrg/fdc-pwa3.2dbg/bin:$PATH alias py37="/hpcfs/bes/gpupwa/pingrg/miniconda/miniconda3/bin/python3.7"
If you use .tcshrc, you set
$ cd ~ $ vim .login # Write the follows in file .login setenv fdc /besfs5/groups/psip/psipgroup/public/pingrg/fdc-pwa3.2dbg set path=($path /besfs5/groups/psip/psipgroup/public/pingrg/reduce6 /besfs5/groups/psip/psipgroup/public/pingrg/fdc-pwa3.2dbg/bin) alias py37="/hpcfs/bes/gpupwa/pingrg/miniconda/miniconda3/bin/python3.7"
If you use .bashrc, you set
$ cd ~ $ vim .login # Write the follows in file .login export fdc=/besfs5/groups/psip/psipgroup/public/pingrg/reduce6 /besfs5/groups/psip/psipgroup/public/pingrg/fdc-pwa3.2dbg/ export PATH=/besfs5/groups/psip/psipgroup/public/pingrg/reduce6 /besfs5/groups/psip/psipgroup/public/pingrg/fdc-pwa3.2dbg/bin:~/:./:$PATH alias py37="/hpcfs/bes/gpupwa/pingrg/miniconda/miniconda3/bin/python3.7"
The syntax depends on which shell you're using, same as below.
The file .login may not be automatically loaded when you're login, in this case you should source it before you conduct fdc-pwa analysis.
$ source ~/.login
There are also some other useful short cuts, you can set them in .zshrc (if you're using bash, you should set it in .bashrc).
alias gpu="cd /hpcfs/bes/gpupwa/$username/" alias out="ls *.out -lt"
The directory structure for a fdc-pwa analysis is not simple. The most efficient way is to copy one from R.G. Ping's directory. Take the example of :
$ cd /hpcfs/bes/gpupwa/$username/ $ mkdir fdc $ cd fdc $ cp -r /hpcfs/bes/gpupwa/pingrg/fdc/psip2LSpi0 .
Then the structure of your gpupwa directory should be:
$ cd /hpcfs/bes/gpupwa/$username/ $ tree -L 3 . └── fdc └── psip2LSpi0 ├── data ├── model └── process
The data files used for fdc-pwa are text files like:
0.400545423950 -0.083236721706 0.295674054506 0.221143410844
1.589680630633 -0.707874531586 0.439717349094 -0.765726195449
1.695550935671 0.791111253291 -0.735391403597 0.544582784601
0.289712255583 0.213774351302 -0.140117209906 0.033226053360
1.618071715636 0.066904959138 -1.123956999279 -0.312877877152
1.777993019036 -0.280679310487 1.264074209166 0.279651823796
...
The first column is the energy term of 4-momenta, the nexts are px, py, pz term. Precision should be set to 12. Each line stands for a particle, for example the first line stands for , the second line is , the third line is and then the fourth line is again of the next event.
Accurately you can write like this in your scripts:
outfile<<fixed; outfile<<setprecision(12); //pi0 outfile<<gamma1[0]+gamma2[0]<<" "<<gamma1[1]+gamma2[1]<<" "<<gamma1[2]+gamma2[2]<<" "" "<<gamma1[3]+gamma2[3]<<endl; //Lambda outfile<<Lambda[0]<<" "<<Lambda[1]<<" "<<Lambda[2]<<" "<<Lambda[3]<<endl; //Sigma_bar outfile<<Lambdabar[0]+gamma[0]<<" "<<Lambdabar[1]+gamma[1]<<" "<<Lambdabar[2]+gamma[2]<<" "<<Lambdabar[3]+gamma[3]<<endl;
For each channel, we should prepare 3 such files for data, backgrounds in incmc and PHSP signal. Specifically, 4 momenta of PHSP signal events should be obtained from MC truth and you need to cut off the events with combined invariant mass less than 1 GeV.
The file names should be no more than 5 characters, normally
# data | backgrounds | PHSP signal pa.dt pa.bg pa.mc # data | backgrounds | PHSP signal for conjugate channel pb.dt pb.bg pb.mc
All these files should be put in the directory data/.
Notice that all tese events should be able to pass your selection criteria.
To configure your model of pwa fitting.
In directory model/, there is a file named "add_vertices". You should edit it to set your final state particles and resonances. The only part you need to edit is:
pa_list_add:={ % name spin parity c isospin G Strange Baryon Charge Mass Widh {"\Lambda", 1/2, +1, X, 0, X, -1, 1, 0, 1.115683, 0.000}, {"\Sigma", 1/2, +1, X, 1, X, -1, 1, 0, 1.192642, 0.000}, {"\pi", 0, -1, +1, 1, -1, 0, 0, 0, 0.134977, 0.000}, {"\psi", 1, -1, -1, 0, -1, 0, 0, 0, 3.686, 0.000294}, {"\Sigma[1385]", 3/2, +1, X, 1, X, -1, 1, 0, 1.3837, 0.0365}, {"\Sigma[1750]", 1/2, -1, X, 1, X, -1, 1, 0, 1.75, 0.15}, %{"\Lambda[1670]", 1/2, -1, X, 0, X, -1, 1, 0, 1.670, 0.035}, {"\Lambda[1690]", 3/2, -1, X, 0, X, -1, 1, 0, 1.690, 0.090}, %{"\Lambda[1800]", 1/2, -1, X, 0, X, -1, 1, 0, 1.800, 0.300}, %{"\Lambda[2000]", 1/2, -1, X, 0, 0, -1, 1, 0, 1.900, 0.155} };
The first three lines are final state particles. The fourth line is your mother particle. Then the next lines are your resonances. Be careful to check the physics properties when you're editing this part, including spin, parity, quantum numbers and so on.
Notice that for particles with isospin 1, the anti-particles will be generated automatically, so you only need to write the ones with charge 0 or 1.
Another file you need to edit is "model.def". The most important is to change the "model_home" in this file to your own directory:
model_home:='"/hpcfs/bes/gpupwa/$username/fdc/psip2LSpi0/model/"; model_name:='"SM and Some Phenomenology Lagrangian"; model_author:='"R.G. Ping"; model_time:='"Aug 17th, 2022";
Actually you can also change the lines of name, author and time.
After editing files above, excute gmodel and lamodel in this directory to generate your model. If there is no problem, you should see three suceeess lines in gmodel output. The out put can also be checked with command tail out1, tail out2, tail out3.
$ gmodel
*----------------------------------------------------------
* Start to build the model
* ...... Successed in first step
* ...... Successed in second step
* ...... Successed in third step
* The model was built successfully if three successed appears
* It means something WRONG if you found ***** in the above
*----------------------------------------------------------
* Start to generate the latex file of the model
* ......The latex file model.tex was generated
* Please latex it
*----------------------------------------------------------
$ lamodel
lamodel can generate a file "model.ps", in which you can check your particel table, resonances and amplitudes.
To configure your fdc-pwa process.
In directory process/, there is a file named "process.def". You should edit it to set your initial and final states and some other properties.
// Set model_home to your own model directory model_home:='"/hpcfs/bes/gpupwa/$username/fdc/psip2LSpi0/model/"; process_name:='"psip --->pi Sigma_bar Lambda, tree"; process_author:='"R.G. Ping"; process_time:='"Aug. 16th, 2022"; // Lines above can also be edited namel:='(ef efb pr1 pr2b pr3)$ inpl:='(1 1 -1 -1 -1)$ // ef & efb stands for electron and positron // pr1, pr2b and pr3 can be found in model.ps // in the table inpl, 1 stands for initial states, -1 stands for final states (ec 3.686) %center mass energy (mc_num 38106) (da_num 5492) // ec stands for energy of centre of mass // mc_num is PHSP events number // da_num is data events number // !!! if you have more than one set of data, write the largest number
Above are the parts you need to edit in "process.def".
After editing "process.def", you can run command diag in directory process/ to generate Feynmann diagrams of your resonance channels. Then run amp and kine to calculate amplitudes and kinematics.
$ diag
There are 1 diagram pages
$ amp
$ kine
You can also use command tail out1, tail out2, tail out3 to check if they're successful. The generated diagrams are in file /process/diagram/diag.ps
Every time you configure your process, directory /process/fort will be replaced. Be sure to save back-ups.
When model and process are all done, move to directory process/fort/. Now you need to edit file "init_fit.f", change te ecm to the right one.
ec = 3.686d0
Then edit file "par.inp", change the numbers of events to the right ones. (ndt and nmc are generated automatically from process.def)
parameter (ndt=5492,nmc=38106,nbg=210)
You can check "reson.inp" to see your particle list. You can also edit it if you need to.
Finally you should edit "myf2.py".
py37 -m numpy.f2py -c af1.f setp12.f read.f smin.f lorentz.f amps2.f genppp.f abc.f parameter.f coupling.f amp12.f amp11.f amp10.f amp9.f amp8.f amp7.f amp6.f amp5.f amp4.f amp3.f amp2.f amp1.f ams.f fff.f init_fit.f -m FDC0 py37 -m numpy.f2py -c af1.f setp12.f read.f smin.f lorentz.f amps2.f genppp.f abc.f parameter.f coupling.f amp12.f amp11.f amp10.f amp9.f amp8.f amp7.f amp6.f amp5.f amp4.f amp3.f amp2.f amp1.f ams.f fff.f init_fit.f -m FDC1 # if you have more than one set of data, write 2 lines of it # Name them with FDC0, FDC1... # !!! you should check how many amp files you should include # !!! Be sure you have the right ampxx.f writen in these lines # !!! check it in file "makefile", you'll see (for example) objs4 = kint.o af1.o setp12.o read.o \ smin.o func.o lorentz.o amps2.o genppp.o \ abc.o parameter.o coupling.o amp12.o amp11.o \ amp10.o amp9.o amp8.o amp7.o amp6.o \ amp5.o amp4.o amp3.o amp2.o amp1.o \ ams.o fff.o # !!! I have 12 amp files up to amp12.o, so I should write to # !!! amp12.f in above lines
After all these edited, you need to excute myf2.py (it's a shell script).
$ myf2.py # if you can't run it directly, try $ source myf2.py
After compile done, you shoud see .so files in your process/fort/ directly consistent with what you write in "myf2.py".
ls *.so
FDC0.cpython-37m-x86_64-linux-gnu.so
FDC1.cpython-37m-x86_64-linux-gnu.so
Prepare your fitting script and submit your job.
You need to check and edit a file "test.py" in process/fort/. Ther are a few parts you need to edit:
os.system("cat test.py") EvtPDL = Script() EvtPDL.readPdtTable('reson.inp') EvtPDL.readParaList('fpara.inp') # !!! you need to make sure particles below are in the order of your # !!! data file EvtPDL.setFinalState(['pi0','Lambda0','anti-Sigma0']) # !!! Make sure these are the data files you prepared in advance EvtPDL.setEvtFileMC(['pa.mc','pb.mc']) EvtPDL.setEvtFileDT(['pa.dt','pb.dt']) EvtPDL.setEvtFileBG(['pa.bg','pa.bg']) # !!! Set the Backgrounds weight, the factor mutiplied to numbers of # !!! events EvtPDL.setBgWeight([0.522581,0.385057]); EvtPDL.setGPUrule(1) # 1 GPU for 1 set of data
# Below are the fitting part for i in range(1): myfit = SMLfit(EvtPDL) myfit.scan(1) myfit.exec('hesse') myfit.exec('migrad') if not myfit.exec('valid'): myfit.exec('hesse') myfit.migrad() myfit.writeParaList('myfit.list',myfit.exec('np_values'),myfit.exec('np_errors')) myfit.write_totMCamps('totAmps.npz',1) if myfit.exec('valid'): numpy.save('mycov',myfit.exec('np_covariance')) print('FVAL= ',myfit.exec('fval')) print('values ',myfit.exec('np_values')) print('errors ',myfit.exec('np_errors')) # !!! You need to make sure that these file names are right myfit.writeEvtMassB3(['pa.mc','pb.mc'],'massmc') myfit.writeEvtMassB3(['pa.dt','pb.dt'],'massdt') myfit.writeEvtMassB3(['pa.bg','pb.bg'],'massbg') myfit.writeModeEvtAmps('mmEvt.npz') # !!! Depend on how many sets you have print('======= Signal yields for each mode set 0=========') myfit.calSta(0) print('======= Signal yields for each mode set 1=========') myfit.calSta(1)
After editing "test.py", you should copy the data files you prepared in to process/fort/.
# for example. The same for pa.mc, pa.bg, pb.dt, pb.mc, pb.bg $ cp ../../data/pa.dt .
To submit gpu jobs, you need to edit "sub.gpu" .
// Make sure it's pwanormal #SBATCH --qos=pwanormal // Cumtomize your job name #SBATCH --job-name=psip2LSpi0 // It's the cpu cores you apply for your job #SBATCH --ntasks=3 // memories applied for each cpu #SBATCH --mem-per-cpu=30000 // gpu number you apply #SBATCH --gres=gpu:v100:2
After everything set, use sbatch to submit your gpu job and use sacct to check your jobs.
$ sbatch sub.gpu
$ sacct
JobID JobName Partition Account AllocCPUS State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
1545701 psip2LSpi0 gpu gpupwa 3 COMPLETED 0:0
1545701.bat+ batch gpupwa 3 COMPLETED 0:0
1545701.0 hostname gpupwa 3 COMPLETED 0:0
The out put file like "sub.gpu-1545701.out" and "myfit.list" contains fitting output. You can adjust your analysis according to them.
The fitted data is saved in .npz files. You can use script "figs.py" to plot your figures, but you need to check and modify it before you plot your figures.
When the fit is finished, the GPU process will return a out file in the format like "sub.gpu-1545894.out", here "sub.gpu" is the GPU script you have submitted, and followed by your job number "1545894". In the end of this file, you can access the signal yields below the line:
"======= Signal yields for each mode set 0=========", for the subset 0 data set.
In the line "Signal yields for mode[ i, j ]= xxx +/- xxx", you get the number of signal yields for mode [i,j]. If , the number denotes the interference between the Feynman diagram i+1 and j+1.