Differences between revisions 89 and 90
⇤ ← Revision 89 as of 2023-12-04 13:13:43 →
Size: 13970
Editor: GötzWaschk
Comment:
← Revision 90 as of 2024-04-02 08:45:54 → ⇥
Size: 14089
Editor: GötzWaschk
Comment:
Deletions are marked like this. Additions are marked like this.
Line 18: Line 18:
 * 2024-04-02
   * Apptainer updated to [[https://github.com/apptainer/apptainer/releases/tag/v1.3.0|release 1.3.0]]

Contents

  1. Overview
  2. News
  3. Invocation
  4. Availability
  5. Batch system support
  6. Filesystems shared between Host and Container
  7. Docker image support
  8. CUDA support
  9. Caveats

Overview

Singularity or Apptainer allow instantiating an alternative operating system environment on our Linux systems.

In other words:

  • If you need to develop software in a Scientific Linux 7 environment on your Ubuntu 16.04 desktop and then execute it on a Scientific Linux 6 system, Singularity makes that possible.
  • If you need to run software from Ubuntu 16.04 (like an up to date PDF viewer) or other software requiring recent versions of some shared libraries (like the Atom editor) on the ancient SL6, Singularity makes that possible.
  • If you still have software for the discontinued SL5 around and need to run, build, develop or debug it on a supported system, Singularity makes that possible.

And all this without significant overhead. Sounds like fiction? It's fact. Right now. Some caveats do apply though.

Credit: Singularity was and is being developed at Berkeley Lab. Please see https://apptainer.org/docs-legacy for upstream information.

Apptainer is the new name of Singularity, developed under the umbrella of Linux Foundation. For documentation, see https://apptainer.org/docs .

News

  • 2024-04-02
  • 2023-12-04
  • 2023-11-25
  • 2023-11-14
  • 2023-10-20
  • 2023-09-19
  • 2023-08-14
  • 2023-06-23
  • 2023-05-24
  • 2023-05-11
  • 2023-05-02
  • 2023-04-13
  • 2023-03-02
  • 2023-02-20
  • 2023-01-26
  • 2022-12-22
  • 2022-11-14
  • 2022-10-27
  • 2022-10-13

    • Singularity is replaced by Apptainer 1.1.2, everywhere except on Ubuntu desktops

  • 2022-05-12
  • 2022-03-29
  • 2022-02-21
  • 2021-12-09
  • 2021-11-22
  • 2021-11-15
  • 2021-10-06
  • 2021-09-02
    • SL7 and pax images updated to Scientific Linux 7.9
  • 2021-07-01
  • 2021-06-11
  • 2021-06-03
  • 2021-04-27
  • 2021-04-12

    • AlmaLinux 8.3: initial image of the RHEL8 clone

  • 2021-03-31
  • 2020-12-10
  • 2020-12-03
    • CentOS7 image updated to 7.9
  • 2020-10-26
  • 2020-10-05
  • 2020-08-11
  • 2020-05-20
    • CentOS, SL7 and pax images updated to 7.8
  • 2020-03-05
  • 2020-01-29
  • 2019-09-24
    • CentOS image updated to 7.7
  • 2019-08-30
    • SL-based images update to 7.7
  • 2019-06-12
    • SL-based images update to 7.6
  • 2018-12-12
  • 2018-10-29
    • SL-based images updated to 6.10 and 7.5
  • 2018-09-04
    • A new image for CentOS 7.5 was created
  • 2018-09-03
    • Singularity will be updated to release 2.6.0
  • 2018-08-01
    • A new image pax was created with an environment compatible with the parallel farm
  • 2018-07-05
    • Singularity will be updated to the security release 2.5.2 tonight - no negative impact on users is epected
    • the SL7 image was updated and no longer provides the "cluster" software/environment
      • this should enhance compatibility with ordinary WGSs and the (non-pax) batch farm
  • 2018-06-21
    • Gridengine directory /usr/gridengine is bind mounted on SL hosts to provide UGE batch system access
  • 2018-06-06
    • Overlayfs was reenabled
  • 2018-06-05
    • Initial Ubuntu 18.04 image released, needs host newer than SL6 to run
  • 2018-05-28
    • Singularity was updated to release 2.5.1 on all supported systems
  • 2018-05-01
    • Singularity was updated to release 2.5.0 on all supported systems
  • 2018-04-03
    • We had to disable the use of overlayfs for security reasons for the time being. Consequently, bind mounting additional paths will not work unless the mount point exists in the image. This will inhibit CUDA support as described below, at least until we have rolled out images containing /etc/OpenCL/vendors .
  • 2018-02-21
    • Singularity was updated to release 2.4.2 on all supported systems
    • A set of updated standard images was provided - now in squashfs format which makes them even snappier to start and use

Invocation

There are several ways to instantiate an Apptainer container and run software in it. The two most important ones are:

  • "Execute" an image
    Apptainer images are executable. Executing one of the standard images provided by DV will give you an interactive shell in the container environment: 

    [wgs03] ~ % lsb_release -d                     
Description:    Red Hat Enterprise Linux Server release 6.9 (Santiago)
[wgs03] ~ % /project/apptainer/images/U16.img
*************************************************************************
* You are now running a shell in a  Singularity container providing an  *
* Ubuntu 16.04 environment. Most things should just work as if you had  *
* just logged in to  such a system, but there are differences.          *
*                                                                       *
* For more info, please see https:://dvinfo.zeuthen.desy.de/Singularity *
*************************************************************************
[wgs03:U16] ~ % lsb_release -d
Description:    Ubuntu 16.04.2 LTS
[wgs03:U16] ~ %
    Note the modified prompt. It's an attempt to make it more transparent in which environment you're currently working.

  • Run an arbitrary command in the context of the image. 

    [wgs03] ~ % lsb_release -d
Description:    Red Hat Enterprise Linux Server release 6.9 (Santiago)
[wgs03] ~ % apptainer exec /project/apptainer/images/SL5.img lsb_release -d
Description:    Scientific Linux release 5.11 (Boron)
[wgs03] ~ %

Availability

Apptainer containers can be instantiated on any current user accessible Linux system at DESY Zeuthen. In particular, Apptainer is available on all

  • Linux Desktops
  • Workgroup Servers

  • Computes Nodes (Farm & Cluster)

The current set of standard images available in /project/apptainer/images includes

  • Scientific Linux 5 (SL5.sif)

  • Scientific Linux 6 (SL6.sif)

  • Scientific Linux 7 (SL7.sif)

  • AlmaLinux 8 (AL8.sif)

  • AlmaLinux 9 (AL9.sif)

  • Ubuntu 16.04 LTS (U16.sif)

  • Ubuntu 18.04 LTS (U18.sif)

These short image names are symbolic links to the current versions of the standard images for these operating systems. The specific images are being kept around for a few months, but not forever.

Note users can copy a current image to any location to make sure this exact version will remain available.

Also note users can bring their own images and thus don't have to rely on the ones provided by DV. Creating those is rather simple and documented upstream. It currently requires root access though, so cannot be done on supported Linux systems in Zeuthen. But have a look at https://singularity-hub.org .

Since Release 2.3, it's also possible to import docker images without root privileges. Note that the environment variable APPTAINER_CACHEDIR is used to place downloaded image layers, and it defaults to ~/.apptainer. Change the variable to something more sensible to avoid wasting a lot of home space.

Batch system support

In case you want to run your batch jobs directly under the standard images available in /project/apptainer/images, it is possible with these lines in the HTCondor submit file 

universe=container
container_image=/project/apptainer/images/<os>.sif

whereas <os> defines the image to be used.

Filesystems shared between Host and Container

Subject to availability on the host:

  • $HOME

  • /tmp

  • /afs

  • /lustre

  • /cvmfs

  • /batch

  • /acs, /pnfs (dCache)

  • /net (dCache)

  • /nfs

  • /run/user

  • /proc

  • /dev

  • /usr/gridengine

  • $PWD (the current working directory of the process invoking singularity)

Users can specify additional folders on the host to be bind mounted inside the container when the container is started. This shouldn't be required. But if it is, please see apptainer help run for info how to do it - and let DV know why you need to do this.

Docker image support

It is recommended to create an image file from docker before running it, for instance: 

export APPTAINER_CACHEDIR=/tmp/$(USER)/apptainer-cache
apptainer build /tmp/gpt.sif docker://gptdev/notebook
apptainer run /tmp/gpt.sif

CUDA support

To run CUDA code in apptainer, the CUDA libraries must be available both inside and outside the container. Apptainer can automatically bind the needed directories into the container like this: 

apptainer exec --nv -B /etc/OpenCL/vendors yourprogram

Caveats

  • The "standard images" are probably not final. If you need additional software or configuration, please let us know.

  • While we made an effort to provide applications running in a Apptainer container with the same set of significant environment variables - and in particular PATH and OS_ARCH should make sense - , environment variables inside the container may differ from expectation. DV can improve here if required.

  • While AFS just works, and the credentials (=tokens) inside a container are just the same as outside, the AFS sysname is always that of the host. There is no way to change this. Which is yet another reason not to rely on AFS sysname magic. To reiterate: using paths like /afs/zeuthen.desy.de/group/software/@sys/bin will not work as expected.

  • Automounted filesystems on SL6 hosts will have to be mounted before the container starts, thus the mount may have to be triggered outside the container. This affects cvmfs, lustre and dCache

  • Some special applications require compatibility between userland (tied to the container environment) and kernel (tied to the host). In particular:
    • CUDA

    • InfiniBand

    • MPI across hosts, especially over InfiniBand

    While all of those use cases are objectives of the Apptainer project, actually making them work hasn't been explored yet. In particular, our standard images currently don't have CUDA installed - but see the note on CUDA above.
  • Setuid/Setgid bits have no effect in the container. We haven't found a case where this causes real problems, but it can be surprising. Affected:

    • ping

    • screen (workaround: use tmux instead)

  • Since the upgrade to Apptainer, the containers run rootless, a Linux kernel mechanism called unprivileged user namespaces is used instead of a setuid root starter executable. This means that files in the container have a different ownership (your user instead of root). If this causes trouble, see the next item:

  • To work around the limitations of rootless apptainer, an image file can be unpacked before running by using the option --unsquash, e.g. for running ancient SL5 firefox: apptainer exec --unsquash /project/apptainer/images/SL5-management.img firefox .

Apptainer (last edited 2024-04-02 08:45:54 by GötzWaschk)