# Using `famewoks` remotely via the ESRF JupyterHub service `jupyter-slurm.esrf.fr`

It is possible to use `famewoks` via your ESRF account at
`jupyter-slurm.esrf.fr` portal.

## 0) Basics (*only for those who are not familiar with the Jupyter ecosystem, otherwise skip this*)

Here some basics concepts about the [Jupyter](https://jupyter.org/) ecosystem.
This is not an exhaustive guide, just a bird eye vue. Internet or any LLM could
guide you more, if you want to go deeper in the topic.

In the following we consider using only
[JupyterLab](https://jupyterlab.readthedocs.io/en/latest/), which is a
web-based application that permits you running python code (and more) from your
browser. You have a full development environment straight in your browser,
without installing anything. We do not consider here the classic Jupyter
notebook.

[JupyterHub](https://jupyterhub.readthedocs.io/en/stable/) permits running
multiple instances of JupyterLab for multiple users.

[https://jupyter-slurm.esrf.fr](https://gitlab.esrf.fr/silx/training/jupyter-slurm)
is an ESRF service which starts a JupyterHub instance on the ESRF cluster via
the [SLURM](https://slurm.schedmd.com/documentation.html) cluster management
and job scheduling system.

Once you have started your server with a JupyterLab instance, you can open a
Jupyter notebook. It is important to keep in mind that each notebooks runs with
a **python kernel** behind, that is, the Python environment where all libraries
and software that you need in your notebook are installed. There are some
standard Python environments installed on `jupyter-slurm` but they do not have
`famewoks` installed. For this reason, you have to install a custom python
environment first.

## 1) Install the `famewoks` Python environemnt in `jupyter-slurm.esrf.fr`

The following procedure has to be performed only once:

- Login into [https://jupyter-slurm.esrf.fr](https://jupyter-slurm.esrf.fr)
- Go to the **Advanced tab**
- Choose **Jupyter environment: Custom** with the following parameters:
  - Environment name: `famewoks262`
  - Path: `/data/bm16/inhouse/local/miniforge/envs/famewoks262/bin`
  - Click on *Add custom environment*

Once this procedure is completed. You can start your server choosing the
`famewoks262` environment directly from the **Jupyter environment** combo box in
the *Simple* tab. You should have a page that resembles the following one:

![jupyter-slurm main starting page](./figs/jupyter-slurm_main_start.png)

**NOTES**

- *Partition/Number of CPUs/Total memory/etc.*: can be left at default values
- *For French CRG staff*: use the `jupyter-fcrg` Partition
- *Job duration*: this is the time your server will be alive, choose it according your needs
- *Launch JupyterLab*: make sure this is ticked

### 2) Access `/data/visitor` from JupyterLab

When you start a JupyterLab server, it will start by default in your home
directory (`/home/esrf/your_username`). In the files navigation panel on the
left you will see only your home directory, not other directories mounted on
different disks. To circumvent this limitation and access to any other mount
points of the ESRF network, you have to **create symbolic links** to them.

The procedure is rather simple:

- Open a terminal from the JupyterLab interface. Click on the "Terminal" icon,
  or *File -> New -> Terminal*.
- In the terminal, create a symbolic link to your experiment data at
  `/data/visitor/your_proposal`, called *MYDATA*:

```bash
$ ln -s /data/visitor/your_proposal MYDATA
```

You will then see `MYDATA` directory in the file browser panel on the left. By
clicking on it you will access your experiment's data.

 **NOTES**
 
  - the sampe procedure works also for accessing any other mount point available in the
ESRF network (e.g. `/data/inhouse` or `/tmp_14_days` workspaces).

### 3) Open a notebook and perform your data reduction/analysis

- From the file browser panel, go to `MYDATA/your_beamline/your_session/SCRIPTS`
- Open the notebook file you have started on the beamline or create your own.
- Pay attention that the kernel name on the top right is `Python (famewoks)`
- Enjoy!

**NOTES**

- If you do not have a notebook already available in `SCRIPTS`, open a terminal
  from the JupoyterLab interface, go to your directory and download the *Data
  reduction notebook* from
  [https://gitlab.esrf.fr/F-CRG/fames/famewoks](https://gitlab.esrf.fr/F-CRG/fames/famewoks)
  or any other notebook available somewhere else via a lin (e.g. a shared
  cloud). For example:

```bash
cd; cd MYDATA/your_beamline/your_session/SCRIPTS
wget https://gitlab.esrf.fr/F-CRG/fames/famewoks/-/raw/master/notebooks/data_reduction.ipynb
```

### 4) Download the processed data to your local computer

`famewoks` saves its output data as Athena project files (`.prj`) that can be
easily opened in [Larix, the Larch GUI](https://xraypy.github.io/xraylarch/) to
be further processed. Those files are located, by default, in the
`PROCESSED_DATA/famewoks` directory. To download any file frome JupyterLab, you
can do it from the file browser panel. Right-click on the file and choose
"Download", as shown below.

![jupyter-slurm download data](./figs/jupyter-slurm_download_data.png)

This operation is time consuming when you have many files to download. The
easiest is to make an archive and download it. Here the procedure to follow:

- Open a Terminal in JupyterLab
- Go to the `PROCESSED_DATA/famewoks` directory
- Zip the files:

```bash
zip -r myarchive.zip *.prj
```

- Download the Zip file

## Frequently asked questions (FAQ)

### What to do if I have a `PermissionError`?

It may happen that you are trying to read/write data from an experiment as
unauthorized user. In this case you will have a `PermissionError: [Errno 13]
Permission denied: ...` in the output trace. This can be solved by asking to the
main proposer of the experiment (or the local contact) to grant access to your
user account via the [https://data.esrf.fr/](https://data.esrf.fr/) portal.

### I do not see the figures

If the figures do not appear, try commenting the following lines:

```python
import plotly.io as pio
pio.renderers.default = "iframe"
```

### Something is weird, I get strange errors

- First check that you have not done any mistake in your code
- Restart the kernel process (**Kernel -> Restart Kernel)
- Re-execute the notebook from the first cell