CSCS¶

This page is a stub! You're welcome to update and augment it!

This page provides information for PizDaint at CSCS where ICON-HAMMOZ has been ported.

Table of contents
CSCS

Contact persons¶

Sylvaine Ferrachat (general help, jobscript toolkit, other scripts)

Anne Kubin (input files)

CSCS help desk (any general technical issue)

General, machine-related information¶

Please refer to its companion CSCS' ECHAM-HAMMOZ page.

Environment settings¶

Nothing specific: icon configuration integrates all proper environment settings (it will purge your currently loaded modules and will reload the necessary ones).

How to download, configure, build, run icon-hammoz on daint¶

#-- Preliminary step: change dir to your #SCRATCH
#   !! Important!! For some reasons, the daint env for icon will not be very stable if you install icon in your $HOME.
#   It is therefore recommended to do everything on $SCRATCH
#   The associated caveat is that $SCRATCH is regularly sweeped away. Be careful and back up your work!
#   Update September 2021: Attached is a runscript-example that allows to install the model code in /project/[your project name]/[your username] while doing the simultions in #SCRATCH

cd $SCRATCH

#-- Clone the icon-hammoz on your account:

git clone git@git.iac.ethz.ch:hammoz/icon-hammoz.git [optional: name of the directory to create to store this local repo.]
cd icon-hammoz # or to the custom directory you asked to create
./scripts/deploy.sh  # this loads the actual code of HAMMOZ and ICON which are embedded as submodules in icon-hammoz-distrib

#-- Configure
#   Warning! Only gcc currently works with this version of icon-hammoz on daint, for running CPUs

./configure --with-fortran=gcc --with-hammoz --enable-jsbach 

#-- Compile
#   !! Important! Do not use a simple make command here

./build_command

#-- Prepare runscript for running icon with HAM

./make_runscripts atm_amip_hammoz_marc

#-- Adjust your runscript
#   Update September 2021: See the attached runscript-example on how to adjust the runscript; look for the parts enclosed by "CSCS specific" 

cd run
# --> then edit run/exp.atm_amip_hammoz_marc.run, in particular, watch the #SBATCH header for the number of nodes, watch also the `end_date` variable etc.
#     IMPORTANT NOTE September 2021: use only 1 node as more nodes do not substantially speed up the simulation

#   Update September 2021: create run-directory; adjust experiment name if necessary
mkdir -p ${SCRATCH}/experiments/atm_amip_hammoz_marc

#-- Launch your run

sbatch --time=[some time info here in format HH:MM:SS] exp.atm_amip_hammoz_marc.run
#   Update September 2021: the time can also be set in the runscript (see attached example)

#-- Inspect your output

cd ${SCRATCH}/experiments/atm_amip_hammoz_marc

#-- Inspect your job log
#-- > edit LOG.exp.atm_amip_hammoz_marc.run.[0-9]*.o

Alternate customizations¶

if you want to a custom experiment name (as opposed to producing a sole 'atm_amip_hammoz_marc' experiment output), there's several ways, but the easiest and cleanest is to simply copy the experiment template run/exp.atm_amip_hammoz_marc e.g. run/exp.my_icon-ham_exp. Then, you create the final runscript with:

cd [root of your icon-hammoz distrib, e.g. '..' if you were in run/]
./make_runscripts my_icon-ham_exp

if you're tired of amending the number of nodes in your final runscript each time you prepare an experiment, there's again several ways. Instead of running make_runscript you may use ./config/make_target_runscript, which takes many runtime customizations. However, it is not very well documented / user-friendly. An easy alternative way is to change a bit the way the SBATCH header is formed for daint on cpu. The below recipe is to be done once for all:
- edit the file config/create_target_header and search for the function set_run_target_daint_cpu
- inside this function, there's the following lines:
```
  cat >> $output_script << EOF
# =====================================
# daint batch job parameters
#-----------------------------------------------------------------------------
#SBATCH --constraint=gpu
#SBATCH --partition=normal
#SBATCH --account=s903
#SBATCH --job-name=$job_name
#SBATCH --output=$output_folder/$job_log_name.o
#SBATCH --error=$output_folder/$job_log_name.o
#SBATCH --nodes=$use_nodes
#$SBATCH_ntasks_per_node
#$s_cpu_time
#========================================
```
- Add a '#' or delete the line #SBATCH --nodes=$use_nodes

--> Next time you launch your job, specify directly the number of nodes to use with:

sbatch --nodes=[some number as you wish] --time=[HH:MM:SS as you wish] exp.[your exp].run

Important recommendation: If you start doing some modifications in files that are meant to be versioned, it is strongly recommended that you create your own branch to do whatever you want and version that (this will prevent any disappointment relative to the fact that you're in a non-permanent disk space - see above remark on top of the page relative to doing everything on @$SCRATCH@ -):
```
git checkout -b [your new branch name, e.g. hammoz/myname_someintention]
# hack, hack, hack...
git status # to see your modified files
git add [the files you want to commit]
git commit
```

Files (1)