Seminar “Selected Topics in Mathematical Optimization”

• Web version
• PDF version
• Repository: https://github.com/siko1056/optim-2021

N = 3;
[X,Y] = meshgrid (linspace (-N, N, 40));

% Gaussian probability density function (PDF)
GAUSS = @(sigma, mu)  1 / (sigma * sqrt (2*pi)) * ...
                      exp (-0.5 * ((X - mu(1)).^2 + (Y - mu (2)).^2) / sigma^2);

Z = 9 * GAUSS (0.6, [ 0.0,  2.0]) + 5 * GAUSS (0.5, [ 1.0,  0.0]) ...
  + 3 * GAUSS (0.4, [-0.5,  0.0]) - 3 * GAUSS (0.3, [-1.5,  0.5]) ...
  - 7 * GAUSS (0.5, [ 0.0, -2.0]);

surf (X, Y, Z);
colormap ('jet');
view (-55, 21);
axis off;

Rounding modes: standards and implementations

Switching the rounding mode is an important tool used by many verification methods based on floating-point arithmetic. The latter being only of finite precision, input data or results of numerical computations with “too long” fractional part have to be rounded to a floating-point number, i.e. a number that fits “nicely” into the computers memory.

The rules for rounding are defined by the IEEE Standard for Floating-Point Arithmetic (IEEE-754 2019, doi: 10.1109/IEEESTD.2019.8766229), which the C99, C++11, and later C/C++ standards refer to in an older version. Those rounding modes with respective C/C++ macros are:

• Roundings to nearest
  • FE_TONEAREST (“ties to even”)
  • (“ties away from zero” is not available in C/C++)
• Directed roundings
  • FE_TOWARDZERO = towards zero
  • FE_UPWARD = towards +infinity
  • FE_DOWNWARD = towards -infinity

C/C++ functions using the macros above are:

• since C99 fesetround and fegetround
• since C++11 std::fesetround and std::fegetround.

Under the hood, for example in the GNU C library (glibc 2.34) the functions fesetround.c and fegetround.c modify or read the active rounding mode from the global x87 FPU Control Status Register and SSE Control Status Register (MXCSR). The state of those registers is part of a running process. During a process context switch the register state (and thus the active rounding mode) is usually saved and restored properly.

With gaining popularity of multi-core processors, using light-weight threads has become more popular in software design. Context switches for threads are designed to be fast. For the sake of performance and depending on the implementation, some register states are not properly synchronized or at worst (partially) reset or cleared.

To overcome implementation-dependency, software libraries like BLAS/LAPACK have to manually take care of synchronizing those register states when starting a new thread. The situation for OpenBLAS is described below.

Further details about the specification and implementation of floating-point arithmetic can be found in many excellent articles and books. To name an exceptional all-encompassing source of information, see section 2.2, sub-section 3.4.4, chapter 6, and sub-section 6.1.3 in Muller et al. “Handbook of Floating-Point Arithmetic” (2018, doi: 10.1007/978-3-319-76526-6) or the respective Wikipedia article.

Changing the rounding mode from Octave/Matlab

Finally, the C function fesetround can be called from Octave/Matlab through the mex-file interface using the following code setround.c:

#include <mex.h>
#include <fenv.h>
#include <float.h>

#pragma STDC FENV_ACCESS ON

void mexFunction ( int nlhs, mxArray *plhs[],
                   int nrhs, const mxArray *prhs[] ) {

  int rnd = (int) mxGetScalar (prhs[0]);
  int mode = FE_TONEAREST;

  switch (rnd)
    {
      case -1:
        mode = FE_DOWNWARD;
        break;
      case 0:
        mode = FE_TONEAREST;
        break;
      case 1:
        mode = FE_UPWARD;
        break;
      case 2:
        mode = FE_TOWARDZERO;
        break;
      default:
        mode = FE_TONEAREST;
        break;
    }

  fesetround (mode);
}

which can be compiled with:

mex --std=c11 setround.c                          % Octave
mex ('CFLAGS="$CFLAGS --std=c11"', 'setround.c')  % Matlab

Switching the rounding mode to +infinity would be done with the call:

setround (+1);

See setround.c for analogous calls for other rounding modes.

This approach of switching the rounding mode is most notably used by INTLAB or the Interval package.

The issue: parallel matrix computation

For “small” (single-threaded) computations (thus mostly scalars) the function setround works fine.

e = eps () * eps ();  % smaller than machine precision
setround (+1);        % round to +inf
num2hex (1 + e)
ans = 3ff0000000000001

setround (-1);        % round to -inf
num2hex (1 + e)
ans = 3ff0000000000000

However, the strength of Octave and Matlab is the ease and speed of matrix and vector computations. Therefore, it is desirable to perform those calculations using directed rounding modes as well.

e = eps () * eps ();  % smaller than machine precision
setround (+1);        % round to +inf
a = ones(100) + e;
a_sup = a * a;        % use BLAS (multi-threaded)

setround (-1);        % round to -inf
a_inf = a * a;        % use BLAS (multi-threaded)

% The following value must be "0" (all values differ),
% if rounding mode switching works correctly.
sum (a_inf(:) == a_sup(:))

As outlined above, Octave and Matlab use in general multi-threaded BLAS/LAPACK implementations for matrix and vector computations. The thread synchronization of the rounding mode (in the global register state) is often neglected by those BLAS/LAPACK implementations for the sake of performance and the code listing above returns a value larger than 0 (which means computing with setround (+1) or setround (-1) makes no difference at all).

For example OpenBLAS offers an optional compile flag CONSISTENT_FPCSR = 1 for manual rounding mode synchronization. See driver/others/blas_server_omp.c:

static void exec_threads(blas_queue_t *queue, int buf_index){

// ...

#ifdef CONSISTENT_FPCSR
  __asm__ __volatile__ ("ldmxcsr %0" : : "m" (queue -> sse_mode));
  __asm__ __volatile__ ("fldcw %0"   : : "m" (queue -> x87_mode));
#endif

Using this compile flag has already been suggested by Masahide Kashiwagi (and information on Apple M1 chips) and by INTLAB.

However, most Linux distributions provide OpenBLAS libraries without this compile flag set. And the situation seems similar for other BLAS/LAPACK implementations like the Intel MKL.

Therefore the remainder of this article deals with how to replace the default OpenBLAS version with a custom one compiled with the CONSISTENT_FPCSR=1 flag set and, alternatively, how to ensure that only a single computation thread is used in Octave and Matlab, a slow and impractical last resort solution.

Octave on Microsoft Windows

It is assumed, that octave-6.4.0-w64-installer.exe from https://www.octave.org/download has been successfully installed.

C:\Octave\Octave-6.4.0\

wshSystemEnv("OMP_NUM_THREADS") = "1"

C:\Octave\Octave-6.4.0\mingw64\bin

Octave on macOS

It is assumed, that GNU Octave is installed via Homebrew https://brew.sh/:

brew install octave

OMP_NUM_THREADS=1 octave --gui

export EDITOR="open -a TextEdit"  # optional
brew edit openblas

ENV["CONSISTENT_FPCSR"] = "1"

export HOMEBREW_NO_INSTALL_FROM_API=1  # Ensure to use local changes
brew uninstall --ignore-dependencies openblas
brew reinstall --build-from-source   openblas

Octave on Linux

OMP_NUM_THREADS=1 octave --gui

cd /tmp
wget https://github.com/xianyi/OpenBLAS/archive/refs/tags/v0.3.19.tar.gz
tar -xf v0.3.19.tar.gz
cd OpenBLAS-0.3.19

make -j8             \
  CONSISTENT_FPCSR=1 \
  USE_THREAD=1       \
  USE_OPENMP=1       \
  NUM_THREADS=256

make install PREFIX=$HOME

LD_PRELOAD=$HOME/lib/libopenblas_haswellp-r0.3.19.so octave --gui
>> version -blas
ans = OpenBLAS (config: OpenBLAS 0.3.19 NO_AFFINITY USE_OPENMP HASWELL MAX_THREADS=256)

Matlab

In an undocumented feature-function Matlab offers an implementation, similar to setround above:

feature ('setround', mode);  % mode = 0, 0.5, +Inf, or -Inf

However, this function has the same limitations as setround and as it is no official function, it is not expected to work reliably. With every new Matlab release the thread synchronization of the underlying Intel MKL works more of less reliable. At the time of writing, Matlab R2020b is known to work, while Matlab R2021a is not.

A known reliable workaround is to force Matlab to only use a single computations thread with the startup option -singleCompThread. This works on MS Windows, macOS, and Linux.

Summary

In this blog post the issue of reliably changing the rounding mode for basic linear algebra operations within multi-threaded BLAS/LAPACK libraries using C/C++ instructions from Octave and Matlab was investigated.

While Octave can address this problem by using a customized OpenBLAS or slower reference BLAS/LAPACK implementation, each Matlab release must be evaluated for limitations of the underlying Intel MKL. A solution of last resort for both Octave and Matlab is enforcing a single computation thread, sacrificing all benefits of having multiple CPU cores available.

The proposed solutions do not apply for graphics processing units (GPUs), and associated numerical libraries, as the architecture and command set differs significantly from the CPU.

What is this project about?

The https://github.com/gnu-octave/docker repository defines Docker images - to build Octave (e.g. build-octave-6.docker), - to run a certain Octave release (e.g. octave-6.docker), and - to run software on top of Octave (e.g. JupyterLab).

Those Docker images are rebuilt monthly to incorporate maintenance and security updates and distributed via DockerHub: - https://hub.docker.com/r/gnuoctave/octave - https://hub.docker.com/r/gnuoctave/octave-build

The images can be used by Docker (= Podman) and Singularity. In the end of this article, some recommendations and experiences regarding the tools are given.

The following figure summarizes the available Octave Docker images and their dependencies.

What makes this project different?

The Octave Docker images are created with several usage scenarios in mind.

• Simple usage by (non-root) users

  An installation script is provided, that can be called directly with this shell command:

  ## Install
  /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/gnu-octave/docker/main/install.sh)" install -t singularity
  
  ## Uninstall
  /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/gnu-octave/docker/main/install.sh)" install -u -t singularity

  It creates links in $HOME/bin, as well as Desktop entries, to start the Octave and JupyterLab as if it was installed by the Linux distribution.

  

  

  Note: The system must have either Docker (= Podman) or Singularity installed and the user account must be setup to use those tools properly. Please adapt the shell command after -t respectively.

• “Time travels” (Docker only)

  Octave (package) developers, who want to test a specific Octave version X >= 4.0.0 (CLI or GUI) can pull the desired Docker image and don’t have to worry about incompatible libraries or overlapping installations.

  Hint: Using the installation script (described above), the desired version can be started very conveniently

  • OCTAVE_VERSION=4.0.0 octave --force-gui
  • OCTAVE_VERSION=4.2.0 octave --force-gui
  • OCTAVE_VERSION=4.4.0 octave --gui
  • OCTAVE_VERSION=5.1.0 octave --gui
  • OCTAVE_VERSION=6.1.0 octave --gui

  Please note that storing all those Octave version images is expensive. Therefore it is recommend to use Docker (Podman) for this task. In contrast to Singularity, Docker does not bundle all image layers into a single file. Image layers shared between images (e.g. Octave 4.0.0 and 4.0.1) do not have to be stored twice.

  After pulling all images:

  $ docker images
  REPOSITORY         TAG          IMAGE ID      ...  SIZE
  gnuoctave/octave   jupyterlab   c3f33109a9df  ...  3.81GB
  gnuoctave/octave   6.2.0        dc589b215991  ...  3.48GB
  gnuoctave/octave   6.1.0        635b3b411836  ...  3.47GB
  gnuoctave/octave   4.2.1        889f5cf8bbb9  ...  2.34GB
  gnuoctave/octave   5.1.0        56d1f94f3260  ...  2.83GB
  gnuoctave/octave   4.2.0        21e4bc9e07c0  ...  2.34GB
  gnuoctave/octave   5.2.0        8008f6c20bbe  ...  2.83GB
  gnuoctave/octave   4.4.1        dddef7f82523  ...  2.82GB
  gnuoctave/octave   4.2.2        c21b8daf6a96  ...  2.34GB
  gnuoctave/octave   4.4.0        ecf795465a96  ...  2.82GB
  gnuoctave/octave   4.0.1        aad035508ad4  ...  2.33GB
  gnuoctave/octave   4.0.2        9752ba4f54a9  ...  2.32GB
  gnuoctave/octave   4.0.3        df6af0aee010  ...  2.32GB
  gnuoctave/octave   4.0.0        d89e88593166  ...  2.33GB

  Summing up their sizes gives 38.38 GB. But thanks to shared layers

  $ docker system df
  TYPE     TOTAL  ...  SIZE      RECLAIMABLE
  Images   14     ...  15.06GB   15.06GB (100%)

  the real storage requirement is only 15.06 GB (39 %).

• CI projects

  Automated Continuous Integration (CI) for software testing is becoming popular these days. For example the Octave Packages index uses the latest stable Octave image with GitHub Actions to perform basic package checks.

Customizing the images or “your own Octave”

Off-topic note

When starting this blog post, unfortunate news regarding the shutdown of the “free” DockerHub Autobuild service were published.

It took lots of time to get the Autobuild setup right and it was planned to advertise it in this blog post as “the way” to customize and publish Octave images. Many thanks to the DockerHub maintainers for their great service. It is a shame that free services are being misused.

If the choice of bundled software or configurations is not satisfactory, there are options for customization:

FROM  gnuoctave/octave:6.2.0
LABEL maintainer="Your Name <e@mail.com>"

# If a privileged installation is necessary
USER root

# Install a missing Ubuntu package
RUN apt-get --yes update  && \
    DEBIAN_FRONTEND="noninteractive" \
    apt-get --no-install-recommends --yes install \
      missing-ubuntu-package

# Install a custom library (Note: GLPK is already installed!)
RUN GLPK_VERSION=5.0     && \
    mkdir -p /tmp/build  && \
    cd       /tmp/build  && \
    wget -q "https://ftp.gnu.org/gnu/glpk/glpk-${GLPK_VERSION}.tar.gz"  && \
    tar -xf glpk-${GLPK_VERSION}.tar.gz  && \
    cd      glpk-${GLPK_VERSION}         && \
    ./configure             \
      --with-gmp            \
      --prefix=/usr         \
      --libdir=/usr/lib  && \
    make -j8             && \
    make check           && \
    make install         && \
    rm -rf /tmp/build

# Ensure starting with an unprivileged default user.
USER jovyan

Bad news for oldtimers (RHEL/CentOS 7)

Even though containers are great to become independent of the host system, some elementary requirements remain, such as the Linux kernel. RHEL/CentOS 7 uses Linux 3.10 and is a dominant distribution in many companies and universities until the end of life in June 2024.

All Docker images up to Octave 5 work on RHEL/CentOS 7 without problems. In contrast, Octave 6 images (and all newer Octave images in the future) use Ubuntu 20.04 as base image. The qt libraries of Ubuntu 20.04 are compiled using a Linux kernel feature, which is not given in Linux 3.10, and any attempts to start the GUI or print graphics will end with this error message:

$ octave
/usr/libexec/octave/6.2.0/exec/x86_64-pc-linux-gnu/octave-gui: error while loading shared libraries: libQt5Core.so.5: cannot open shared object file: No such file or directory

If you only use the CLI version for computations and avoid graphics (which is an unrealistic use case for a desktop installation of Octave), you can sneak around this limitation.

$ octave-cli
GNU Octave, version 6.2.0
[...]
For information about changes from previous versions, type 'news'.

>>

As described in the last section, you are free to customize and publish Docker images building Octave 6 based upon the docker.io/gnuoctave/octave-build:5 image.

Summary

Another Linux distribution independent way to install and run GNU Octave was introduced. The approach offers new ways of using and sharing customized Octave installations.

For the average user both, Docker and Singularity, perform equally well. Docker is to favor, when the intention is to install several Octave versions in parallel (see “Time travels” above). Singularity is clearly the choice for archiving or sharing images. This is in particular interesting for research projects, where all Octave setups should be at best equal and unaffected by updates or system changes. Podman, despite all praises by RedHat, had problems when running the Octave Docker images as non-root user. For example dealing with the default internal container user “jovyan” or the GPU and graphics support has still some flaws.

Finally, thanks to GitHub Actions and DockerHub, maintaining the introduced infrastructure comes almost for free.

The SavannahAPI interface

First and most important: SavannahAPI is not a new GNU Savannah. SavannahAPI only provides improved read access to the issues (bugs and patches) for a particular GNU Savannah project. There is no possibility to edit or modify issues, neither it has a user or account management.

For an interface description, see the project README.md on GitHub.

While working on the Octave project with GNU Savannah for a few years, I often felt a lack of overview about the ever growing number of bugs. Now (March 2021) there are more than 10,000 issues (bugs and patches) accumulated in a time span of more than 10 years.

Some specific points that I missed with GNU Savannah are:

https://savannah.gnu.org/bugs/index.php?go_report=Apply&group=octave
  &func=browse&set=custom&msort=0&report_id=221&advsrch=1
  &resolution_id%5B%5D=0&bug_group_id%5B%5D=0&status_id%5B%5D=1
  &priority%5B%5D=0&severity%5B%5D=0&category_id%5B%5D=100
  &category_id%5B%5D=110&category_id%5B%5D=101&category_id%5B%5D=102
  &category_id%5B%5D=104&category_id%5B%5D=105&category_id%5B%5D=106
  &category_id%5B%5D=107&category_id%5B%5D=103&category_id%5B%5D=114
  &category_id%5B%5D=112&category_id%5B%5D=109&category_id%5B%5D=113
  &release_id%5B%5D=173&release_id%5B%5D=174
  &platform_version_id%5B%5D=0&history_search=0&history_field=0
  &history_event=modified&history_date_dayfd=25
  &history_date_monthfd=2&history_date_yearfd=2021&chunksz=100
  &spamscore=5&boxoptionwanted=1#options

The project in detail

Summary - Many new possibilities

Having a copy of all issue tracker data, it is very straight forward to create custom queries to obtain a desired subset (or all) of this data in various data formats (HTML, JSON, CSV).

Thus SavannahAPI is also a backup of GNU Savannah and enables a possible exit strategy, once the GNU Octave project decides for another code hosting service.

Despite existing JavaScript web application, the web API can be used by any arbitrary client, e.g. from a GUI programmed in Octave language itself.

Octave Forge developers do not have to monitor the whole bug tracker, they can query a subset of interest in the JavaScript web application, or embed the table in their own website, e.g.

• io-package (all bugs):
  • https://savannah.octave.org/?[octave%20forge]%20(io)
  • https://savannah.octave.org/api.php?Action=get&Format=HTMLCSS&Title=[octave%20forge]%20(io)
• io-package (all open bugs):
  • https://savannah.octave.org/?Action=get&Format=HTMLCSS&Title=[octave%20forge]%20(io)&OpenClosed=open
  • https://savannah.octave.org/api.php?Action=get&Format=HTMLCSS&Title=[octave%20forge]%20(io)&OpenClosed=open

Similarly, all other kind of permalinks to dynamic content can be created and shared. For example do you want to join a code-sprint on three particular bugs? - https://savannah.octave.org/?Action=get&Format=HTMLCSS&ItemID=60245,60237,60236

Any feedback on GitHub or Octave Discourse is very welcome.

Enjoy using SavannahAPI.

The code at a glance

%% Step 1: Install and load pkg-json.
if (isempty (pkg ("list", "pkg-json")))
  pkg install "https://github.com/gnu-octave/pkg-json/archive/v1.0.0.tar.gz"
end

pkg load pkg-json

%% Step 2: Define a helper function to plot an individual country.
function plot_country (name, color, countries, obj)
  idx = find (strcmp (name, countries));
  if (isempty (idx))
    error ("Country ''%s' not found.", name);
  end
  data_str = obj.data(idx){1}(10){1};
  data_str = data_str((1 + length("MULTIPOLYGON (((")):(end - 3));
  data_str = strsplit (data_str, ")), ((");
  for i = 1:length (data_str)
    data = textscan (data_str{i}, "%f %f,");
    fill (data{:,1}, data{:,2}, color);
    hold on;
  end
end

%% Step 3: Download the public data set.
json_str = urlread ("https://data.nasa.gov/api/views/yqhp-cuk8/rows.json?accessType=DOWNLOAD");

%% Step 4: Import JSON data to GNU Octave.
obj = jsondecode (json_str);

%% Step 5: Extract the country names.
countries = cell(length (obj.data), 1);
for i = 1:length (obj.data)
  countries{i} = obj.data(i){1}(9){1};
end

%% Step 6: Plot all available countries with random color.
for i = 1:length (countries)
    plot_country (countries{i}, rand (3,1), countries, obj)
end

The code explained

Step 3: Using urlread it is really straight forward to import the data provided by the NASA into a string.

json_str = urlread ("https://data.nasa.gov/api/views/yqhp-cuk8/rows.json?accessType=DOWNLOAD");

printf ("The string contains %.2f KB of data.", length (json_str) / 1024)

The string contains 253.02 KB of data.

Step 4: Thanks to Abdalla’s Google Summer of Code project the JSON data is available for further Octave processing with a single function call.

tic;
obj = jsondecode (json_str);
toc;

Elapsed time is 0.0160789 seconds.

The jsondecode function can of course not know about the structure of the NASA map data itself, thus it is necessary to explore the imported data using Octave.

The first step is to simply output the content of obj, which is quite verbose and thus omitted here.

As JSON is a structured data format, most of the content is converted to Octave structures. A handy function to get an overview of Octave structures is fieldnames.

fieldnames (obj)

ans =
{
  [1,1] = meta
  [2,1] = data
}

A splitting into a meta and data section can be observed. Some further investigation of the meta section revealed, that it is not very useful for the purpose of plotting the world map, thus it is not further explained here.

On the other hand the data section contains the desired polygon edge data of the several country borders.

Step 5: For the sake of convenience, the names of the individual countries are exported into an cell array of strings.

countries = cell(length (obj.data), 1);
for i = 1:length (obj.data)
  countries{i} = obj.data(i){1}(9){1};
end

With this approach, it is easy to map a country name to an index in obj.data, which is examined in the following for “Japan”.

japan = find (strcmp (countries, "Japan"))
usa   = find (strcmp (countries, "United States of America"))

japan =  88
usa =  173

obj.data(japan)

ans =
{
  [1,1] =
  {
    [1,1] = row-bxsn-brky.azms
    [2,1] = 00000000-0000-0000-1E4A-C7232D9BDD99
    [3,1] = 0
    [4,1] =  1495517988
    [5,1] = [](0x0)
    [6,1] =  1495517988
    [7,1] = [](0x0)
    [8,1] = { }
    [9,1] = Japan
    [10,1] = MULTIPOLYGON (((134.638428 34.149234, 134.766379 33.806335, 134.203416 33.201178, 133.79295 33.521985, 133.280268 33.28957, 133.014858 32.704567, 132.363115 32.989382, 132.371176 33.463642, 132.924373 34.060299, 133.492968 33.944621, 133.904106 34.364931, 134.638428 34.149234)), ((140.976388 37.142074, 140.59977 36.343983, 140.774074 35.842877, 140.253279 35.138114, 138.975528 34.6676, 137.217599 34.606286, 135.792983 33.464805, 135.120983 33.849071, 135.079435 34.596545, 133.340316 34.375938, 132.156771 33.904933, 130.986145 33.885761, 132.000036 33.149992, 131.33279 31.450355, 130.686318 31.029579, 130.20242 31.418238, 130.447676 32.319475, 129.814692 32.61031, 129.408463 33.296056, 130.353935 33.604151, 130.878451 34.232743, 131.884229 34.749714, 132.617673 35.433393, 134.608301 35.731618, 135.677538 35.527134, 136.723831 37.304984, 137.390612 36.827391, 138.857602 37.827485, 139.426405 38.215962, 140.05479 39.438807, 139.883379 40.563312, 140.305783 41.195005, 141.368973 41.37856, 141.914263 39.991616, 141.884601 39.180865, 140.959489 38.174001, 140.976388 37.142074)), ((143.910162 44.1741, 144.613427 43.960883, 145.320825 44.384733, 145.543137 43.262088, 144.059662 42.988358, 143.18385 41.995215, 141.611491 42.678791, 141.067286 41.584594, 139.955106 41.569556, 139.817544 42.563759, 140.312087 43.333273, 141.380549 43.388825, 141.671952 44.772125, 141.967645 45.551483, 143.14287 44.510358, 143.910162 44.1741)))
  }

}

From this data about Japan, we are only interested in the 10th entry, defining the “MULTIPOLYGON” data about three larger Japanese islands (areas). The format is as follows:

MULTIPOLYGON (
  ((x1 y1, x2 y2, ...))    // Area 1
  ((x1 y1, x2 y2, ...))    // Area 2
  ...
)

Using some straight forward Octave instructions, this sub format of the JSON data can be further processed by Octave.

%% Extract the 10th data entry.
data_str = obj.data(japan){1}(10){1};
%% Strip start and end of the numerical data.
data_str = data_str((1 + length("MULTIPOLYGON (((")):(end - 3));
%% Split area data into individual cells of an array.
data_str = strsplit (data_str, ")), ((");

The data in the three cells is still of type “string” and can finally be converted using textscan for example.

Now the x,y-data for the edges of each area is available in numerical form and can be simply plotted by Octave’s fill function in the same loop.

for i = 1:length (data_str)
  data = textscan (data_str{i}, "%f %f,");
  fill (data{:,1}, data{:,2}, "r");
  hold on;
end

Step 2: All steps above can conveniently be abstracted into a single function plot_country, defined in the first code section of this notebook.

A second more simplified function call will look like this, where countries is obtained by Step 5 and obj by Step 4. "g" (= “green”) is the color that should used.

plot_country ("United States of America", "g", countries, obj)

In Step 6 an example was given how all available countries can be plotted.

Benchmark setup

Create a directory to keep track of the mess.

mkdir ('benchmark');
cd ('benchmark');

Load the benchmark JSON files.

if (exist ('citm_catalog.json', 'file') ~= 2)
  urlwrite ( ...
    'https://github.com/RichardHightower/json-parsers-benchmark/raw/master/data/citm_catalog.json', ...
    'citm_catalog.json');
end

if (exist ('canada.json', 'file') ~= 2)
  urlwrite ( ...
    'https://github.com/mloskot/json_benchmark/raw/master/data/canada.json', ...
    'canada.json');
end

if (exist ('large-file.json', 'file') ~= 2)
  urlwrite ( ...
    'https://github.com/json-iterator/test-data/raw/master/large-file.json', ...
    'large-file.json');
end

Setup octave-rapidjson.

if (exist ('octave-rapidjson', 'dir') == 0)
  urlwrite ( ...
    'https://github.com/Andy1978/octave-rapidjson/archive/2d88511712032b14dea4c2272d82249e7547772a.zip', ...
    'octave-rapidjson.zip');
  unzip  ('octave-rapidjson.zip');
  rename ('octave-rapidjson-2d88511712032b14dea4c2272d82249e7547772a', ...
          'octave-rapidjson');
  cd ('octave-rapidjson')
  urlwrite ( ...
    'https://github.com/Tencent/rapidjson/archive/35e480fc4ddf4ec4f7ad34d96353eef0aabf002d.zip', ...
    'rapidjson.zip');
  unzip  ('rapidjson.zip');
  rename ('rapidjson-35e480fc4ddf4ec4f7ad34d96353eef0aabf002d', 'rapidjson');
  mkoctfile -Wall -Wextra -I./rapidjson/include load_json.cc
  mkoctfile -Wall -Wextra -I./rapidjson/include save_json.cc
  cd ('..')
end

Setup octave-jsonstuff.

if (isempty (pkg ('list', 'jsonstuff')))
  pkg install https://github.com/apjanke/octave-jsonstuff/releases/download/v0.3.3/jsonstuff-0.3.3.tar.gz
end

Setup JSONio.

if (exist ('JSONio', 'dir') == 0)
  urlwrite ( ...
    'https://github.com/gllmflndn/JSONio/archive/6c699a315ac2c578864d8b740a061bff47b718bf.zip', ...
    'JSONio.zip');
  unzip  ('JSONio.zip');
  rename ('JSONio-6c699a315ac2c578864d8b740a061bff47b718bf', 'JSONio');
  cd ('JSONio')
  mkoctfile --mex jsonread.c jsmn.c -DJSMN_PARENT_LINKS
  cd ('..')
end

Setup jsonlab.

if (exist ('jsonlab', 'dir') == 0)
  urlwrite ( ...
    'https://github.com/fangq/jsonlab/archive/d0fb684bd43165d312063345bdb795b628b2c679.zip', ...
    'jsonlab.zip');
  unzip  ('jsonlab.zip');
  rename ('jsonlab-d0fb684bd43165d312063345bdb795b628b2c679', 'jsonlab');
end

Benchmark run

The benchmark function reads the respective JSON file into a string and calls the libraries reading and writing function.

function t = benchmark (json_read_fcn, json_write_fcn)
  test_files = {'citm_catalog.json', 'canada.json', 'large-file.json'};
  N = length (test_files);
  t = nan (N, 2);
  for i = 1:N
    json_str = fileread (test_files{i});
    tic ();
    octave_obj = json_read_fcn (json_str);
    t(i,1) = toc ();
    tic ();
    json_str2 = json_write_fcn (octave_obj);
    t(i,2) = toc ();
  end
end

The results for the Matlab (R2020b, prerelease) have been measured on the same system without JupyterLab.

t.matlab = [
  0.0768, 0.0853;
  0.1510, 0.5405;
  1.2222, 0.6521];

Octave (7.0.0, development version)

t.octave = benchmark (@jsondecode, @jsonencode);

octave-rapidjson

addpath ('octave-rapidjson')
t.rapid_json = benchmark (@load_json, @save_json);
rmpath ('octave-rapidjson')

octave-jsonstuff: No results due to an error.

%pkg load jsonstuff
%t.jsonstuff = benchmark (@jsondecode, @jsonencode);
%error: cat: field names mismatch in concatenating structs
%error: called from
%    jsondecode>condense_decoded_json_recursive at line 116 column 9
%    jsondecode>condense_decoded_json at line 67 column 7
%    jsondecode at line 63 column 7
%    benchmark at line 8 column 16
%pkg unload jsonstuff

JSONio: Because of the long running time, the results of the first run are saved here.

addpath ('JSONio')
%t.jsonio = benchmark (@jsonread, @jsonwrite);
t.jsonio = [ ...
  0.9583,  30.5410;
  6.1333,  17.4022;
  4.3382, 552.8929];
rmpath ('JSONio')

Jsonlab: Because of the long running time, the results of the first run are saved here.

addpath ('jsonlab')
%t.jsonlab = benchmark (@loadjson, @savejson);
t.jsonlab = [ ...
   35.6242,  26.0625;
    6.1303,   0.7365;
  372.2456, 601.5318];
rmpath ('jsonlab')

Benchmark results

graphics_toolkit ('qt')
titles = {'citm\_catalog.json (2 MB, mixed)', ...
          'canada.json (2 MB, numeric)', ...
          'large-file.json (26 MB, mixed)'};

for i = 1:3
  subplot (3, 1, i);
  bar ([t.matlab(i,:); t.octave(i,:); t.rapid_json(i,:)]');
  legend ({'Matlab (R2020b, pre)', 'Octave (7.0.0, dev)', ...
           'octave-rapidjson'}, 'Location', 'bestoutside');
  xticklabels({'read','write'});
  ylabel ('time in seconds');
  title (titles{i});
end

for i = 1:3
  subplot (3, 1, i);
  bar ([t.jsonio(i,:); t.jsonlab(i,:)]');
  legend ({'JSONio', 'jsonlab'}, 'Location', 'bestoutside');
  xticklabels({'read','write'});
  ylabel ('time in seconds');
  title (titles{i});
end

The first figure compares the running times of Matlab, Octave, and octave-rapidjson. Both Octave and octave-rapidjson are based on RapidJSON and perform in many cases better than the Matlab implementation.

octave-rapidjson is using the SAX API of RapidJSON, while the new Octave implementation uses the DOM API. This decision was made to achieve best compatibility to Matlab, which was in some cases difficult with the SAX API and is no pririty for the octave-rapidjson project. The results show, that both APIs perform similarly in the tested data sets, even though there is a claim, that the SAX API is faster in some cases.

The results of JSONio and jsonlab are split into a second figure, as the running times are significantly larger than those of the first figure. For octave-jsonstuff we could due to an error not obtain any results and the maintainer is informed about it.

octave-buildbot: the final picture?

The initial project design from August has “matured” in the following aspects: 1. The Buildbot Master container can now be accessed from the Workers by SSH. This enables rsync file transfers with significant speed improvements. On the other hand, additional authentication is necessary. SSH key-based authentication was favored over plain username/password authentication via Secret. This gives the Buildbot Master more control about the Workers which can access the Docker container. The SSH port is publicly visible and not only Buildbot Workers might try to get access. 2. The storage location of the build artifacts on the Buildbot Master was moved inside a Docker Volume and is shared with a Nginx web server container. Testing on a local machine is now no different from the production system on https://octave.space and one does not need to worry about the Buildbot Master to mess up the server’s file system directly. The system’s native Apache web server now only serves as proxy and delegates requests from the world wide web to the respective container to serve the requested content.

The bill revisited

In a previous post the computational costs for the Buildbot Workers are listed in Table 1 there. This table has been revisited below. For the Buildbot Master there were no big changes.

Like in the previous table, a Buildbot Worker is expected to provide 4 CPU cores and 75 GB of disk storage.

The heterogeneity among the Workers is the reason for significantly different build times for the MS Windows installers (Octave-MXE). “Stronger” Workers, equipped with a modern server CPU, outperform those with a consumer CPU by hours of build time.

Table footnotes: 1. All builds are performed in a clean empty build folder. 2. Additional disk space is needed for ccache, which currently uses 13 GB of 20 GB, and 1.5 GB for the mxe-octave installer files, which are cached to avoid unnecessary downloads.

Despite the file transfer time reduction, there were two further minor improvements: 1. The “Builders” for Octave and the Doxygen documentation have been merged. This reduces the build time of both tasks by about 20%, as no additional build of Octave for the Doxygen documentation is necessary. Compared to the Octave-MXE build times, this reduction is not significant. 2. The usage of ccache for the Octave-MXE had some measurable improvement, which is shown by three examples for different types of Workers: - 5.3 hours down to 4.5 hours (-15%) for “strong” worker-03 after 2 repetitions. - 5.25 hours down to 4.75 hours (-10%) for “strong” worker-04 after 4 repetitions. - 8.3 hours down to 7.5 hours (-10%) for “weak” worker-01 after 3 repetitions.

Summary

The usage of rsync to copy the files from the Buildbot Workers to the Master is without doubt a huge step forward for the project and it’s very heterogeneous and (over continents) distributed infrastructure. Valuable machine hours are no longer wasted for unnecessarily slow file transfers.

Furthermore, the thorough usage of ccache for the Octave-MXE builds has a measurable impact for all types of Buildbot Workers and saves build time for repetitive compilation tasks without sacrificing “clean” release builds.

Currently the last outstanding issue of octave-buildbot is probably a library issue with modern CPUs. The progress on this issue is tracked on GitHub and hopefully it will be solved soon.

Happy brewing.

Everything in view

When releases happen that frequently, users and developers must be able to judge themselves, if an automatically built set of release files is “good” and complete.

After a Buildbot Worker finishes his duty, the relevant build artifacts are copied to the Buildbot Master, stored with the Octave Mercurial (hg) ID, as shown in the screenshot below.

This looks unattractive and this folder view lacks any information about the corresponding build processes done by the Buildbot Workers.

Fortunately, Buildbot allows to create custom dashboards. The result can now be seen on the landing page of https://octave.space or on the screenshot below.

In this dashboard the build artifacts are connected to the respective build processes including their status. The arrangement of those many MS Windows installers is based on the well-known Octave website https://octave.org/download#ms-windows. Furthermore, each Octave build entry clearly states the version label, hg id, and build date. Another feature is, that the GNU Octave user manual and Doxygen documentation can directly be viewed online to find potential flaws in them.

Trust me, I’m a Buildbot Worker

Especially, when automation and many systems are in the game, there should be a way to make sure, that the final installer downloaded from octave-buildbot is indeed that one “brewed” on the Worker.

To achieve this, before copying the files to the Buildbot Master, the SHA-256 hash sum of each file is displayed in the Buildbot Workers build log.

Admittedly, it is a little clumsy to find those hashes. Their main purpose can be seen as check for damaged or incompletely copied installer files to exclude sources of errors.

Speaking of errors, for the heavy weighting mxe-octave builds, even in the event of a build error the valuable log files are compressed and copied to the Buildbot Master. Because of this, there might sometimes be more than only one build log for a particular mxe-octave build available. They can be distinguished by the build number suffix. The installer files themselves have a timestamp in their file name.

The bill please

Running octave-buildbot binds a certain amount of computational resources. The numbers given in the following table can slightly change in the future, but didn’t change significantly in the last weeks.

Open issues, if you want to jump in

As of today (2020-10-16) there are four open issues about octave-buildbot on GitHub. It would be nice if they are fixed, but those do not disturb the release process significantly (workarounds exist).

• More efficient use of ccache.
• Speed up the file transfers.
  • Fun fact: Because of my limited resources (money) the server is located in Germany, while the great Workers are in Japan.
• Library issue with two of four Workers.
• Podman related deletion problem.

Summary

All in all I am very happy to enjoy a “fresh brewed Octave” every day and hope to be able to run the system for a long time on little own expenses.

Thanks to the use of (Docker) containers, this system can reliably release Octave on almost arbitrary machines with little setup and maintenance effort. On the other hand, the computational resources and storage requirements (4 CPUs and 75 GB per Worker and 70 GB for the Master) are not negligible and can hopefully be tuned a little in the future by binding a certain Worker to a particular task (see the small discussion with Markus).

Please contact me on GitHub if you have suggestions for improvements or you find any issues with this system.

Happy brewing.

Benchmark setup

Create a directory to keep track of the mess.

mkdir ('benchmark');
cd ('benchmark');

Load the benchmark JSON files.

if (exist ('citm_catalog.json', 'file') ~= 2)
  urlwrite ( ...
    'https://github.com/RichardHightower/json-parsers-benchmark/raw/master/data/citm_catalog.json', ...
    'citm_catalog.json');
end

if (exist ('canada.json', 'file') ~= 2)
  urlwrite ( ...
    'https://github.com/mloskot/json_benchmark/raw/master/data/canada.json', ...
    'canada.json');
end

if (exist ('large-file.json', 'file') ~= 2)
  urlwrite ( ...
    'https://github.com/json-iterator/test-data/raw/master/large-file.json', ...
    'large-file.json');
end

Setup octave-rapidjson.

if (exist ('octave-rapidjson', 'dir') == 0)
  urlwrite ( ...
    'https://github.com/Andy1978/octave-rapidjson/archive/2d88511712032b14dea4c2272d82249e7547772a.zip', ...
    'octave-rapidjson.zip');
  unzip  ('octave-rapidjson.zip');
  rename ('octave-rapidjson-2d88511712032b14dea4c2272d82249e7547772a', ...
          'octave-rapidjson');
  cd ('octave-rapidjson')
  urlwrite ( ...
    'https://github.com/Tencent/rapidjson/archive/35e480fc4ddf4ec4f7ad34d96353eef0aabf002d.zip', ...
    'rapidjson.zip');
  unzip  ('rapidjson.zip');
  rename ('rapidjson-35e480fc4ddf4ec4f7ad34d96353eef0aabf002d', 'rapidjson');
  mkoctfile -Wall -Wextra -I./rapidjson/include load_json.cc
  mkoctfile -Wall -Wextra -I./rapidjson/include save_json.cc
  cd ('..')
end

Setup octave-jsonstuff.

if (isempty (pkg ('list', 'jsonstuff')))
  pkg install https://github.com/apjanke/octave-jsonstuff/releases/download/v0.3.3/jsonstuff-0.3.3.tar.gz
end

Setup JSONio.

if (exist ('JSONio', 'dir') == 0)
  urlwrite ( ...
    'https://github.com/gllmflndn/JSONio/archive/6c699a315ac2c578864d8b740a061bff47b718bf.zip', ...
    'JSONio.zip');
  unzip  ('JSONio.zip');
  rename ('JSONio-6c699a315ac2c578864d8b740a061bff47b718bf', 'JSONio');
  cd ('JSONio')
  mkoctfile --mex jsonread.c jsmn.c -DJSMN_PARENT_LINKS
  cd ('..')
end

Setup jsonlab.

if (exist ('jsonlab', 'dir') == 0)
  urlwrite ( ...
    'https://github.com/fangq/jsonlab/archive/d0fb684bd43165d312063345bdb795b628b2c679.zip', ...
    'jsonlab.zip');
  unzip  ('jsonlab.zip');
  rename ('jsonlab-d0fb684bd43165d312063345bdb795b628b2c679', 'jsonlab');
end

Benchmark run

The benchmark function reads the respective JSON file into a string and calls the libraries reading and writing function.

function t = benchmark (json_read_fcn, json_write_fcn)
  test_files = {'citm_catalog.json', 'canada.json', 'large-file.json'};
  N = length (test_files);
  t = nan (N, 2);
  for i = 1:N
    json_str = fileread (test_files{i});
    tic ();
    octave_obj = json_read_fcn (json_str);
    t(i,1) = toc ();
    tic ();
    json_str2 = json_write_fcn (octave_obj);
    t(i,2) = toc ();
  end
end

The results for the Matlab (R2020b, prerelease) have been measured on the same system without JupyterLab.

t.matlab = [
  0.0768, 0.0853;
  0.1510, 0.5405;
  1.2222, 0.6521];

Octave (7.0.0, development version)

t.octave = benchmark (@jsondecode, @jsonencode);

octave-rapidjson

addpath ('octave-rapidjson')
t.rapid_json = benchmark (@load_json, @save_json);
rmpath ('octave-rapidjson')

octave-jsonstuff: No results due to an error.

%pkg load jsonstuff
%t.jsonstuff = benchmark (@jsondecode, @jsonencode);
%error: cat: field names mismatch in concatenating structs
%error: called from
%    jsondecode>condense_decoded_json_recursive at line 116 column 9
%    jsondecode>condense_decoded_json at line 67 column 7
%    jsondecode at line 63 column 7
%    benchmark at line 8 column 16
%pkg unload jsonstuff

JSONio: Because of the long running time, the results of the first run are saved here.

addpath ('JSONio')
%t.jsonio = benchmark (@jsonread, @jsonwrite);
t.jsonio = [ ...
  0.9583,  30.5410;
  6.1333,  17.4022;
  4.3382, 552.8929];
rmpath ('JSONio')

Jsonlab: Because of the long running time, the results of the first run are saved here.

addpath ('jsonlab')
%t.jsonlab = benchmark (@loadjson, @savejson);
t.jsonlab = [ ...
   35.6242,  26.0625;
    6.1303,   0.7365;
  372.2456, 601.5318];
rmpath ('jsonlab')

Benchmark results

Update 2020-08-29: Abdallah found out that the speed problem (described blow) was the call to matlab.lang.makeValidName not the chosen DOM API.

graphics_toolkit ('qt')
titles = {'citm\_catalog.json (2 MB, mixed)', ...
          'canada.json (2 MB, numeric)', ...
          'large-file.json (26 MB, mixed)'};

for i = 1:3
  subplot (3, 1, i);
  bar ([t.matlab(i,:); t.octave(i,:); t.rapid_json(i,:)]');
  legend ({'Matlab (R2020b, pre)', 'Octave (7.0.0, dev)', ...
           'octave-rapidjson'}, 'Location', 'bestoutside');
  xticklabels({'read','write'});
  ylabel ('time in seconds');
  title (titles{i});
end

for i = 1:3
  subplot (3, 1, i);
  bar ([t.jsonio(i,:); t.jsonlab(i,:)]');
  legend ({'JSONio', 'jsonlab'}, 'Location', 'bestoutside');
  xticklabels({'read','write'});
  ylabel ('time in seconds');
  title (titles{i});
end

The results are not as overwhelming as I initially hoped for (they are, see comment above.)

The first figure compares the running times of Matlab, Octave, and octave-rapidjson. Both Octave and octave-rapidjson are based on RapidJSON.

It must be the choice of the API (DOM vs. SAX) that slows down the current Octave implementation (DOM) by a factor of 10 to 100 (wrong, see comment above).

octave-rapidjson, using the SAX API, is for the mixed data case not slower than Matlab. But the implementation itself is less Matlab compatible than the current Octave implementation. The choice of DOM for the current Octave implementation was made to achieve best compatibility to Matlab.

On the positive side, in the case of more numeric data (canada.json) the DOM API outperforms the SAX API. Nevertheless, my humble assumption is that mixed data is more common for JSON data files.

The results of JSONio and jsonlab are split into a second figure, as the running times are significantly larger than those of the first figure. For octave-jsonstuff we could due to an error not obtain any results. I’ll inform the maintainer to hopefully in the future repeat this benchmark.

Regarding this benchmark Octave should seriously consider switching to the SAX API and additionally preserve the current Matlab compatibility (see comment above).

GSoC 2020 is over, and Abdallah enriched Octave with a great new feature. When he (or someone else) ports the Octave function matlab.lang.makeValidName to the C++ language, the performance of JSON decoding and encoding is great and compatible to Matlab.

1 How was Octave released in the past?

The basic workflow for the last major Octave 5.1 release was as follows:

1. Some Octave maintainer thinks it is time for a release and starts a topic on the maintainers mailing-list about it. This happened usually about once per year.
2. Another Octave maintainer (very often Rik, many thanks!) coordinated the release process by creating a wiki checklist and sometimes a wiki bug fix list to coordinate important open bug reports.
3. Finally, mostly jwe created release candidates, the final release tarballs, and MS Windows mxe builds on his machines.

2 Why not just continue like this?

As mentioned before, the Octave 6.1 release got stuck and I tried to summarize some problems of the current release process. The description is from the perspective of a totally uninvolved volunteer trying to release Octave on his own for the first time.

3 Octave-buildbot

In the past months, I thought about a system to address some of the weaknesses identified above. The key ideas are far from new:

1. Save the output (tarballs, logs, …) from the existing Octave Buildbots.
2. Use a reproducible build environment.

4 Summary and outlook

The project goal of octave-buildbot is that everybody should be able to build/release Octave using a fully documented, reproducible Docker environment running Buildbot.

The current state of the project is:

• Buildbot Master and Server can be run on a single local machine or distributed to a webserver and several PCs (see https://buildbot.octave.space, as of 2024-04-10, see https://nightly.octave.org).
• The Octave release tarballs and documentation are built and sent to the Master.

What is missing?

• Building Doxygen
• Better ideas how to faster send the 2 GB Doxygen zip-file to the Buildbot Master and where to store the builds for longer time.
• Implementing the MXE builds, lack of time so far.
• Interested developers supporting my efforts.

The final suggestion from section 3.2 is to make the Mercurial ID the only necessary release candidate identifier. For mxe-octave this means that only stable-octave.mk will be used, not release-octave.mk.

The test setup

To test the speed-up there is severe crime in the Octave language: “ijk Matrix multiplication” using triply nested loops

function C = matrix_mult (A, B)
  [m, r] = size (A);
  [R, n] = size (B);
  if (r != R)
    error ("matrix_mult: Bad arguments");
  endif

  C = zeros (m, n);
  for i = 1:m
    for j = 1:n
      for k = 1:r
        C(i,j) += A(i,k) * B(k,j);
      endfor
    endfor
  endfor
endfunction

As can be seen from a short test run, even for rather small dimensions matrix_mult.m is by a factor of about 10,000 slower than Octave’s builtin matrix multiplication.

n = 30;
A = rand (2*n, n);
B = rand (n, 3*n);

tic; C1 = A * B; toc
Elapsed time is 0.000144958 seconds.

tic; C2 = matrix_mult (A, B); toc
Elapsed time is 1.72018 seconds.

The interpretation of Octave code as in matrix_mult.m is rather slow, as many checks for index bounds, existing symbols, correct syntax, etc. and lots of memory operations have to be performed.

If there is no builtin routine available in Octave, as in the example above, one can study the vectorization chapter in the Octave manual.

The test result

OctaveCoder claims to speed-up things by factor three to four and indeed succeeds:

pkg load coder
octave2oct ("matrix_mult")

which matrix_mult
'matrix_mult' is a function from the file matrix_mult.oct

tic; C3 = matrix_mult (A, B); toc
Elapsed time is 0.54394 seconds.

What happens under the hood? There is an option to keep intermediate files during the compilation to an oct-file.

octave2oct ("matrix_mult", "KeepSource", true)

The output is an about 5000 lines C++-file containing many class definitions and strongly reminds of Octave’s interpreter implementation. According to the description by Hossein, he implemented a “lightweight interpreter”, that does all symbol look-ups at compile time, but still there is interpretation work left to be done. The compiled file reacts dynamically on bad input, for example.

Summary

• OctaveCoder keeps it’s promise and speeds up ordinary Octave m-code by factor three to four.
• It seems like a viable solution, if vectorization is not applicable or translation to C/C++ too tedious.
• Until Octave finally gets a proper working JIT compiler this solution can be really recommended.

Converting hg to git repositories

No worries, this step can be omitted.

mtmiller, the maintainer of two unofficial Octave forks on GitLab end GitHub, explained to me that he uses git-cinnabar to create and update these forks from the official GNU Octave Mercurial repository.

In the README.md of git-cinnabar and by own research on the web, I found that there are many tools for this purpose.

I gave it a short try myself after installing git-cinnabar:

hg  clone https://www.octave.org/hg/octave octave-hg
# The following is a git-cinnabar extension
git clone hg::octave-hg octave-git

But I continue to use the solution from mtmiller as upstream to not duplicate efforts.

Applying GitLab “merge requests”^(*) to Mercurial

^(*)For those who are more familiar with GitHub, GitLab “merge requests” are “pull requests” in GitHub slang.

First, I created a fork https://gitlab.com/siko1056/octave of mtmiller’s upstream repository on GitLab.

In my fork, I used the online editor from GitLab, to make a meaningless change on the “master” branch and created a merge request to mtmiller’s upstream repository at GitLab.

This “merge requests” or individual commits can be conveniently exported to patches from GitLab, without cloning the whole Octave fork, adding other remotes, etc. See the picture below showing the “merge request” view and how to download the associated patch.

Unfortunately, this export button does not exist in GitHub. A nice trick that works for GitLab and GitHub is to append .patch the commit ID in the URL, e.g.

https://gitlab.com/siko1056/octave/-/commit/4d43501fc2a442b179b2327ad8379a19d224323c.patch

This automatically generates a git patch and this feature can nicely be exploited by tools like wget or curl or be posted on Octave’s patch tracker.

Finally, I saved the exported git patch as bug-42424-git.patch on my machine.

Applying git patches to Mercurial works almost perfect!

Only the second blank line in the commit message gets stripped, which is part of Octave’s guidelines. Thus for importing git patches it is recommended to add --edit option

hg import --edit bug-42424-git.patch

to directly fix the commit message, or to call hg commit --amend after the import succeeded.

Basically, that is it.

Can the conversion be better/easier?

The short answer is: Not really.

In case a genuine Mercurial patch is needed, I did not find a great tool to convert git to hg patches. I succeeded using moz-git-tools and shortly explain what I did. But for me this is not the “easy way” to go.

One can either obtain the git patch as described above, or by cloning the GitLab fork:

git clone https://gitlab.com/siko1056/octave.git octave-git
cd octave-git
git format-patch -1 HEAD --stdout > ../bug-42424-git.patch
cd ..
cp bug-42424-git.patch bug-42424-hg.patch
git-patch-to-hg-patch  bug-42424-hg.patch  # moz-git-tools

Going to the Octave Mercurial clone to apply the converted patch preserves the newline in the commit message.

hg clone https://www.octave.org/hg/octave octave-hg
cd octave-hg
hg patch ../bug-42424-hg.patch

Finally, I exported the imported patch again to get an impression about the “quality” of the conversion.

hg export > ../bug-42424-real-hg.patch

The differences between all patches can be seen in this large picture.

Install required Python-software

The installation might require admin (sudo) privileges:

pip3 install --upgrade pip jupyterlab \
                           jupyterhub \
                           oauthenticator \
                           octave_kernel

Finally, one can check the installation by starting a local single-user Jupyter server with the JupyterLab interface:

jupyter lab

Configuring the public JupyterHub server

In the desired setup all configuration files for the server are located at /srv/www/jupyterhub and the server should be accessible by the public via https://www.domain.org:8888 as explained in a previous blog post.

A good starting point is to create a default configuration file that can be modified:

cd /srv/www/jupyterhub
jupyterhub --generate-config -f jupyterhub_config.py

The following options must be changed in jupyterhub_config.py to achieve the desired JupyterHub-server setup: - the port c.JupyterHub.port = 8888, - the location of the TLS-keys which are already established for the Apache2 webserver python c.JupyterHub.ssl_cert = '/etc/certbot/live/domain.org/fullchain.pem' c.JupyterHub.ssl_key = '/etc/certbot/live/domain.org/privkey.pem' - and the usage of the modern JupyterLab interface python c.Spawner.cmd = ['jupyter-labhub'] c.Spawner.default_url = '/lab'

Now one can test the public installation by starting JupyterHub with the custom configuration:

jupyterhub --config=/srv/www/jupyterhub/jupyterhub_config.py

If you want to give the classical notebook interface a try, just replace /lab by /tree:

https://domain.org:8888/user/localUser/lab?
                                      /tree?

Authenticate users

In general JupyterHub delegates the user authentication to some authenticator_class. By default this is the PAMAuthenticator, that just makes use of the authentication mechanism of the underlying server operating system:

c.JupyterHub.authenticator_class = 'jupyterhub.auth.PAMAuthenticator'

This has a serious impact: One does not only login to some sandboxed PHP-application (like a MediaWiki), one literally logs into the server! From Jupyter you can start a terminal window and can do any damage you could do from a ssh-login with your given user permissions. Keep this in mind.

If the desired setup demands more security, one should run Jupyter within some virtualization environment. But for my purpose using Docker, Kubernetes, or alike seems too bloated for a couple of predefined users.

To further control and restrict the user authentication, JupyterHub offers features such as a whitelist and a list of admin users:

c.Authenticator.admin_users = {'localUser'}
c.Authenticator.whitelist = {'localUser', 'localUser2'}

For more options, see https://jupyterhub.readthedocs.io/en/stable/api/auth.html. Unfortunately, those lists cannot be altered at runtime. Of course one can create custom Python-classes, if needed.

Another problem is that I don’t want users to login on my server via password at all. Fortunately, OAuthenticator provides several OAuth 2.0 based authenticator_classes. Among them:

• GitHub

  from oauthenticator.github import GitHubOAuthenticator
  c.JupyterHub.authenticator_class = GitHubOAuthenticator
  c.GitHubOAuthenticator.oauth_callback_url = 'https://domain.org:8888/hub/oauth_callback'
  c.GitHubOAuthenticator.client_id = 'xxxxxxxxxxxxxxxxxxxxxx'
  c.GitHubOAuthenticator.client_secret = 'xxxxxxxxxxxxxxxxxx'

• Google

  from oauthenticator.google import GoogleOAuthenticator
  c.JupyterHub.authenticator_class = GoogleOAuthenticator
  c.GoogleOAuthenticator.oauth_callback_url = 'https://domain.org:8888/hub/oauth_callback'
  c.GoogleOAuthenticator.client_id = 'xxxxxxxxxxxxxxxxxxxxxx'
  c.GoogleOAuthenticator.client_secret = 'xxxxxxxxxxxxxxxxxx'

Especially for Google it makes sense to map the authenticated user to a local server account:

c.Authenticator.username_map = {'local.user@gmail.com': 'localUser'}

But again, this map cannot be altered without restarting the JupyterHub server.

To sum up: New users (for example new colleagues) cannot be created on the fly by using the here described setup. This requires:

1. Creating a new local user account on the server

   useradd --create-home --gid jupyterhub <new colleague>

2. whitelist that local user account c.Authenticator.whitelist.

3. Map that local user account to the OAuth 2.0 authenticated user name c.Authenticator.username_map.

4. Restart JupytherHub.

On the other hand, not every user with a valid GitHub or Google account can log in the server.

Making JupyterHub a system service

The final target is to make the JupyterHub server run, even if you are logged out from your remote server.

There are other options such as running JupyterHub inside a screen-session that does not shutdown when the connection to the server is terminated. But personally I prefer the solution of creating a system service:

1. A service starts after each reboot when enabled.
2. A service can be logged by syslog.
3. A service runs in the background and cannot accidentally be shutdown while maintaining the system.

To create a service, just create a file /srv/www/jupyterhub/jupyterhub.service with the following content:

[Unit]
Description=JupyterHub
After=apache2.target

[Service]
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=jupyterhub
ExecStart=/usr/bin/jupyterhub --config=/srv/www/jupyterhub/jupyterhub_config.py
WorkingDirectory=/srv/www/jupyterhub

[Install]
WantedBy=multi-user.target

The service file has to be copied to the standard location and the file should be owned by root:

sudo chown root:root /srv/www/jupyterhub/jupyterhub.service
sudo cp /srv/www/jupyterhub/jupyterhub.service /etc/systemd/system/jupyterhub.service

Now one can use the standard mechanisms to start and stop JupyterHub as system service:

sudo systemctl enable jupyterhub  # Start service at system boot.
sudo systemctl start  jupyterhub
sudo systemctl stop   jupyterhub

and the server log can be inspected by

sudo journalctl -u jupyterhub

Further reading

This blog post was greatly inspired by https://pythonforundergradengineers.com/add-google-oauth-and-system-service-to-jupyterhub.html

Update 2019-01-13

• In the file /srv/www/jupyterhub/jupyterhub.service

  [Unit]
  -After=network.target nss-lookup.target
  +After=apache2.target

  and rather copy that file to /etc/systemd/system/ instead of just linking it. Occasionally I noticed that JupyterHub did not start automatically, as the location /srv/www/jupyterhub/ was not available to systemd on time.

Basic webserver setup

• Install the required packages:

  zypper install apache2 \
                 php7 php7-mysql apache2-mod_php7 \
                 mariadb mariadb-tools \
                 phpMyAdmin

• Enable Apache modules:

  a2enmod php7 rewrite

• Open the necessary ports in the Firewall (firewalld):

  firewall-cmd --permanent --zone=public --add-service=http --add-service=https
  firewall-cmd --permanent --zone=public --add-port=8888/tcp  # For Jupyter only!
  firewall-cmd --reload

• Start and enable (on each restart) all services:

  systemctl start  apache2 mysql
  systemctl enable apache2 mysql

• Configure the mySQL database and follow the instructions given:

  mysql_secure_installation

Setting up the directories

The used directory structure for the websites and services is as follows:

/srv/www/myMkDocsSite  (https://www.domain.org/)
        /myMediaWiki   (https://wiki.domain.org/)
        /jupyter       (https://www.domain.org:8888)

As Jupyter comes with it’s own web server, one only has to add Apache2 configurations for myMkDocsSite and myMediaWiki. Those configuration files are located in the

/etc/apache2/vhosts.d

directory and have the *.conf extension. The myMkDocsSite.conf may look like this:

<VirtualHost *:80>
    DocumentRoot "/srv/www/myMkDocsSite"
    ServerName www.domain.org
    ServerAlias domain.org
    ServerAdmin joe@mail.com
    ErrorLog /var/log/apache2/error_log
    TransferLog /var/log/apache2/access_log
    HostnameLookups Off
    UseCanonicalName Off
    ServerSignature Off
    <Directory "/srv/www/octave.space">
        Options Indexes FollowSymLinks
        AllowOverride None
        <IfModule !mod_access_compat.c>
            Require all granted
        </IfModule>
        <IfModule mod_access_compat.c>
            Order allow,deny
            Allow from all
        </IfModule>
    </Directory>
</VirtualHost>

Remember, that for dynamic websites the Apache2 server process might need write privileges. Therefore the owner of the respective directories should be wwwrun and can be set with:

chown -R wwwrun:www <directory>

Now the content of a file index.html should be displayed when typing “http://www.domain.org/” (note: HTTPS comes later!).

Webserver HTTPS setup

Fortunately, these days one has not to pay to be able to use the HTTPS protocol, one can easily make use of services like the Let’s encrypt project.

• Install the required packages:

  zypper install certbot python-certbot python-certbot-apache

• Enable the SSL module and ensure the correct server flags:

  a2enmod ssl
  a2enflag SSL

• Get the certificate by following the output of:

  certbot --apache

  This command usually updates the Apache2 configuration file, e.g. myMkDocsSite.conf above. If there are more websites, just call the certbot command above again to extend the certificate.

• Update the certificate regulary by establishing a cronjob:

  crontab -e

  and add the lines:

  # renew all certificates methode: renew
  10 5 1 * *  root    /usr/bin/certbot renew

Mediawiki setup

To setup a Mediawiki one can basically follow the installation guide. Additionally, the following extensions should be installed:

• https://www.mediawiki.org/wiki/Extension:MobileFrontend
• https://www.mediawiki.org/wiki/Extension:Math

With openSUSE Leap, the following extra packages are useful to be installed for all the above:

zypper install git ImageMagick make \
               php7-APCu php7-fileinfo php7-imagick php7-intl php7-mbstring

After that run

php myMediaWiki/maintenance/update.php

to update the MediaWiki database for that extension.