How to create an installer

Presentation

When deploying a node, installers allow to install softwares, copy configuration files, etc…

Today, installers use Saltstack to perform a series of actions on the deployed node.

An installer consists of a zip archive, containing the following files:

  • The Salt files
    • init.sls contains the actions to perform on the node under the Salt syntax
    • defaults.yaml may contain default values of eventual parameters used in init.sls
    • A folder files contains all the files used by init.sls (configuration files, expectfiles etc.)
  • The interface files
    • installer.ini contains the name and description of the installer
    • parameters.json contains the list of the parameters of the installer, as presented in the UI

Quick start

This archive contains a minimal example of an installer. It only installs GNU screen on a node, and has no parameter (and so no default values).

It can be uploaded through the UI in the installers tab, using the New Installer button. Once added to the installers list, it can be added to a node before its deployment.

Uploading installers

An Installer is registered with a UUID. The Installer content stays the same for its whole lifetime. In other words, we do not update an Installer; we’ll delete it and create a new UUID with the new content. The label, of course, can stay the same. There can be several Installer packages for the same label - the user will have to choose between them. This is a desired feature, so we can distinguish between “PBS Pro (Dario)”, “PBS Pro (Viktor)”, and the generic “PBS Pro”.

In order to upload a new Installer through the UI, we could support any of the following:

  • a file archive (.zip, .tgz),
  • a HTTP(S) link to such a file archive,
  • a Git address

Because the Installer content must not change during its lifetime, PBScloud.io’s Backend has to store its contents. If a link has been given, that means Backend must download its contents and store them at “upload” time. Later changes to the link target must not have any impact on PCLM. We do not know when Configurator will request/download the Installer from Backend; when it does, it must get the “one and only” version of the Installer. Otherwise our builds won’t be reproducible.

Uploading an Installer today

The file archive or the Git repository must contain all the files listed below: both Salt data and metadata.

The UI is very simple: only a link must be given or a file must be attached. The UI will do a POST /api/installer with “Content-Type: multipart/form-data”. This distinguishes it from the existing POST /api/installer with “Content-Type: application/json”.

Uploading an Installer tomorrow

For more user comfort, the user could edit the contents of “installer.ini” (see below) directly in a form. The “parameters.json” could be created in an online form editor. Only the formula and all its dependent data must be sent as a file archive or Git repository.

Managing installer at mid term

An external platform will handle installers to let user contribute/clone/modify installer to use them into their PCLM

Details of the installer

init.sls

Salt uses states to define what needs to be done on a node. They are represented like this:

state_name:
  salt.action:
    - parameter: value
    - parameter:
      - list
      - of
      - values

Examples

Install packages

To install the packages screen and htop on a node, the following state is used:

install_packages:
  pkg.installed:
    - pkgs:
      - screen
      - htop
Copy files

To add a configuration file for screen, the following state is now used:

screenrc:
  file.managed:
    - name: ~/.screenrc
    - user: centos
    - group: centos
    - require:
      - pkg: install_packages
    - source: salt:///files/screenrc
    - source_hash: sha512=2dc1b22a6ed4c24e90ad72ce7600efec1aa3b0de1feeba6d468de5acf17b699940ee826c0a885d31639c67c6c16d43b54d4735db973bf8ae1771a2fcfe2862ba

The file screenrc must be present in the files directory. Salt commands are run as root! This should be taken in account writing formulas. For example, the user field of file.managed allows to change the owner of the copied files.

Variables

Variables can be used in a Salt state. `````` is an a variable that refers to the folder where the init.sls file is located. This name will be a UUID and is unknown at the time init.sls is written. Custom variables can be defined as in this example:

{% set install_folder = 'folder/name' %}
clean_install:
  cmd.run:
    - onlyif: |
        [ -e /tmp/ ]
    - name: |
        rm -rf /tmp/ || exit 1
Run a bash command

The cmd.run state allows to directly run bash commands.

show_cpu_info:
  cmd.run:
    - name: |
        cat /proc/cpuinfo

More Salt states

Some examples of the most common used states are presented here: https://docs.saltstack.com/en/getstarted/config/functions.html The list of all Salt states and their usage can be found here: https://docs.saltstack.com/en/latest/salt-modindex.html#cap-s

defaults.yaml

The defaults.yaml file contains default values for variables.

tmp_dir: /tmp
install_dir: /opt
These values can be accessed like this:
{% set mydefaults = salt['defaults.get'] %}
{% set install_folder = 'folder/name' %}
clean_install:
  cmd.run:
    - onlyif: |
        [ -e / ]
    - name: |
        rm -rf / || exit 1

installer.ini

This file contains the general informations on the installer displayed by the UI. It also specifies the driver used for the node configuration (at the moment, this is always Salt).

label=Installer 1
description='Do several things to install stuff'
driver=configurator.drivers.salt
strict_mode=True
author="altair-pbscloud"
software="my-installer"
version="1.0"
revision=1

Label and Description are completely free-form and displayed by the UI. “software” is a more standardised form to describe what this Installer will install, and with “version” you can differentiate between versions of that software. The “author” field gives you a namespace for choosing “software” and “version” to your liking. By incrementing the revision field, you can provide newer versions of the Installer itself (which are supposed to be compatible with older ones). The UI will show only the latest revisions of each triple (author / software / version).

parameters.json

This file contains JSON-Schema for describing all the Installer parameters (for Salt: all pillars for the recipe) - data type and possible values. It is used by the UI to display a form allowing the user to enter parameters. JSON-Schema will be used internally, but it’s hard to enter manually. Can we find a more friendly text format that can be converted into JSON-Schema?

{
    "additionalProperties": false,
    "$schema": "http://json-schema.org/draft-04/schema#",
    "type": "object",
    "properties": {
        "a_parameter": {
            "type": "string",
            "title": "A parameter"
        }
    }
}

The values of the parameters can be accessed in the Salt file through the variable pillar, as the example bellow:

{% set mypillars = pillar[sls_path] %}

my_installer:
   file.managed:
     - source: 
     - source_hash: sha512=

When using pillars this way, an error is thrown when it has not been specified. But if you access them through salt[‘defaults.get’], non-specified pillars get their values from defaults.yaml.

Parameters can be unique or be part of a list. For example, the main ntp server is a unique parameter, other servers are optional and there are possibly several of them. Parameters in an array will allow the user to add new items through an “add” button.

{
    "additionalProperties": false,
    "$schema": "http://json-schema.org/draft-04/schema#",
    "type": "object",
    "properties": {
        "main_server": {
            "type": "string"
        },
        "server": {
            "items": {
                "type": "string"
            },
            "type": "array",
            "title": "other_servers"
        }
    }
}

A parameter can also be a boolean, represented as a checkbox in the UI.

{
    "additionalProperties": false,
    "$schema": "http://json-schema.org/draft-04/schema#",
    "type": "object",
    "properties": {
        "something_to_check": {
            "type": "boolean",
            "default": "True",
            "title": "Activate checkbox"
        }
    }
}

Finally, with a combo box you can let the user choose among a set of predefined values. They use a Map where the keys are the returned values, and the strings the displayed text.

{
    "additionalProperties": false,
    "$schema": "http://json-schema.org/draft-04/schema#",
    "type": "object",
    "properties": {
        "Combo Box": {
            "title": "Region",
            "type": "string",
            "enum": ["choice_1", "choice_2", "choice_3"],
            "enumNames": {
                "choice_1": "First choice",
                "choice_2": "Second choice",
                "choice_3": "Third choice"
            }
        }
    }
}

Complete example : Install a software from binaries

This section presents an installer (the archive is available here) that download and install Sublime Text from the web.

To install the software from binaries, the following steps are required:

  • Install the packages needed for the installation
  • Download and extract the binaries in the /opt folder
  • Create a symbolic link of the software executable in the /usr/bin folder

This Salt file do these steps one after the other:

{% set mydefaults = salt['defaults.get'] %}

install_packages:
  pkg.installed:
    - pkgs:
      - tar
      - bzip2

{% set sha521_file = 'b51f25c376c06b4261d9e797fb0374532d2fdf86c96af5a7cf0847cb6f7514a7ceda894a4e9b5c63efe0053f6651f52dced9f438a319ddc4991524199a5b65bc' %}
install_sublime:
  archive.extracted:
    - name: /
    - source: https://download.sublimetext.com/sublime_text_3_build_3103_x64.tar.bz2
    - source_hash: sha512=
    - archive_format: tar
    - tar_options: v
    - if_missing: /sublime_text_3/

/usr/bin/sublime_text:
  file.symlink:
    - require:
      - archive: install_sublime
    - target: /sublime_text_3/sublime_text

We define the default installation folder in the default.yaml file:

install_folder: /opt

The parameters can override the defaults values (defined in defaults.yaml), here the folder where to install the software:

{
    "additionalProperties": false,
    "$schema": "http://json-schema.org/draft-04/schema#",
    "type": "object",
    "properties": {
        "install_folder": {
            "type": "string",
            "title": "Installation folder"
        }
    }
}

Finally, we define the metadata of the installer in installer.ini:

label=Sublime Text
description='Download and install Sublime Text in /opt'
driver=configurator.drivers.salt
author="our-company"
software="sublime-text"
version="3"
revision=2

Installation validation

The Configurator module validates that an installer ran correctly. To do this, it uses the return values of each state of the installer. If a state named install_ok is present, its returned values is used to validate the installation.

Otherwise, all the states must return True to validation the installation. Finally, if the metadata strict_mode (in installer.ini) is set to False - and no state install_ok is present - then the installation is validate whatever the states returns are.

The following figure illustrates this behaviour: abstraction levels