The number of FAIme jobs has reached 30.000. Yeah!
At the end of this November the FAIme web service for building customized ISOs turns 7 years old. It had reached 10.000 jobs in March 2021 and 20.000 jobs were reached in June 2023. A nice increase of the usage.

Here are some statistics for the jobs processed in 2024:

Type of jobs

3%		cloud image
11%		live ISO
86%		install ISO

Distribution

2%		bullseye
8%		trixie
12%		ubuntu 24.04
78%		bookworm

Misc

• 18%   used a custom postinst script
• 11%   provided their ssh pub key for passwordless root login
• 50%   of the jobs didn't included a desktop environment at all, the others used GNOME, XFCE or KDE or the Ubuntu desktop the most.
• The biggest ISO was a FAIme job which created a live ISO with a desktop and some additional packages This job took 30min to finish and the resulting ISO was 18G in size.

Execution Times

The cloud and live ISOs need more time for their creation because the FAIme server needs to unpack and install all packages. For the install ISO the packages are only downloaded. The amount of software packages also affects the build time. Every ISO is build in a VM on an old 6-core E5-1650 v2. Times given are calculated from the jobs of the past two weeks.

Job type		Avg		Max
install no desktop		1 min		2 min
install GNOME		2 min		5 min

The times for Ubuntu without and with desktop are one minute higher than those mentioned above.

Job type		Avg		Max
live no desktop		4 min		6 min
live GNOME		8 min		11 min

The times for cloud images are similar to live images.

A New Feature

For a few weeks now, the system has been showing the number of jobs ahead of you in the queue when you submit a job that cannot be processed immediately.

The Next Milestone

At the end of this years the FAI project will be 25 years old. If you have a success story of your FAI usage to share please post it to the linux-fai mailing list or send it to me. Do you know the FAI questionnaire ? A lot of reports are already available.

Here's an overview what happened in the past 20 years in the FAI project.

About FAIme

FAIme is the service for building your own customized ISO via a web interface. You can create an installation or live ISO or a cloud image. Several Debian releases can be selected and also Ubuntu server or Ubuntu desktop installation ISOs can be customized. Multiple options are available like selecting a desktop and the language, adding your own package list, choosing a partition layout, adding a user, choosing a backports kernel, adding a postinst script and some more.

At this years Debian conference in South Korea I've presented¹ the new feature of the FAIme web service. You can now build your own Debian live media/ISO.

The web interface provides various settings, for e.g. adding a user name and its password, selecting the Debian release (stable or testing), the desktop environment and the language. Additionally you can add your own list of packages, that will be installed into the live environment. It's possible to define a custom script that gets executed during the boot process. For remote access to the live system, you can easily sepcify a github, gitlab or salsa account, whose public ssh key will be used for passwordless root access. If your hardware needs special grub settings, you may also add those. I'm thinking about adding an autologin checkbox, so the live media could be used for a kiosk system.

And finally newer hardware is supported with the help of the backports kernel for the Debian stable release (aka bookworm). This combination is not available from the official Debian live images or the netinst media because the later has some complicated dependencies which are not that easy to resolve². At DebConf24 I've talked to Alper who has some ideas³ how to improve the Debian installer environment which then may support a backports kernel.

The FAI web service for live ISO is available at

      https://fai-project.org/FAIme/live

Debian Download Web Page

It's just a very tiny difference, but hopefully a big step forward for our users. Our main download web page (which still uses the URL https://www.debian.org/distrib/) now has the title "Download Debian". Hopefully this will improve the results in the search engines.

A brief history of this web page in time

• 1998: The title "Distribution" was added
• 2002: Title changed to "Getting Debian"
• 2024: Finally changed to "Download Debian"

Here are the screenshots of these three versions.

I like that we had a selection menu on the top right corner to select a mirror for downloading in the past.

A few days ago I've also removed the info "Internal ISDN cards are unfortunately not supported." from the netinst subpage. Things are moving forward, but slowly.

In two weeks DebConf24, the Debian conference starts in Busan, South Korea. Therefore I've added support for the Korean language into the web service of FAI:

https://fai-project.org/FAIme/

Another new feature of the FAIme service will be announced at DebConf24 in August.

A new FAI version was released and the FAIme service is using this new release. You can now also create installation images for Debian 13 (testing aka Trixie).

https://fai-project.org/FAIme/

Another new feature of the FAIme service will be announced at DebConf24 in August.

In January I've removed tens of thousands of web pages on www.debian.org. Have you noticed it?

In the past

From 1997 onwards, we had web pages for security announcements. We had to manually prepare a .data and a .wml file which then generated a web page for each security announcement (DSA or DLA). We have listed the 6 most recent messages in a short list that was created from these files. Most of the work that went into the Debian web pages was creating these files.

Our search engine often listed the pages with security announcements instead of a more relevant web page for a particular topic.

Preparation

At DebConf Kosovo (2022) I started with a proof of concept and wrote a script, that generates this list without using the .data/.wml files in the Git repository, but instead reading the primary sources of security information[1]. This new list now includes links to the security tracker and the email of the announcement.

Following web pages and scripts were also using these .data and .wml files:

• OVAL files
• RSS feeds for security announcements (and LTS)
• Apache config file for mapping URLs from dsa-NNN to YEAR/dsa-NNN
• A huge list of crossreferences between DSA and CVE numbers

Before I could remove all the security web pages, I had to adjust the scripts, that create the above information.

When I looked at the OVAL files and the apache logs of our web server, I saw that more than 99% of the web traffic was generated by these XML files (134TB of 135TB total in two weeks). They were not compressed and were around 50MB in size. With the help of Carsten Schönert we managed to modify the python scripts that generate this OVAL file without using the .data/.wml files and now we only provide bzip2 compressed XML files[2].

The RSS feeds are created by the new Perl script which reads the DSA/DLA list the security tracker and determines the URL of the email of all entries. This script also generates the list of the most recent DSA/DLA entries. Currently we show the last 350 entries which covers more than the last year and includes links to the announcement email and the security tracker.

The huge list of crossreferences is not needed any more, since the mapping of CVE to DSA is already included in the DSA list[3] of the security tracker.

The amount of translations of the DSA/DLA was very different. French translations were almost all done, but all other languages did translations for a couple of months or years only. E.g. in 2022, Italian had 2 translations, Russian 15, Danish 212, French and English each 279. But from 2023 on only French translations were made. By generating the list of DSA/DLA we lost the ability to translate these web pages, but since these announcements are made of simple, identical sentences it is easy to use an automatic translation service if needed.

Now the translation statistics of all web pages are more accurate. Instead of 12200 pages that need to be translated (including all these old DSA/DLA) there are now only 2500 pages to translate[4]. Languages that had a lot of old translations of DSA/DLA lost some percentage but languages that are doing translations of newer web pages won in the statistics of how many pages are translated. Examples:

Before

German (de)   3501  28.5%
Italian (it)  1005   8.2%
Danish (da)   6336  51.7%

After

German (de)   1486  59.0%
Italian (it)   909  36.1%
Danish (da)    982  39.0%

Cleanup of all the security web pages

Finally in January, I could remove all web pages of the security announcements in one git commit[5]. Using several git rm -rf commands this commit removed 54335 files, including around 9650 DSA/DLA data files, 44189 wml files, nearly 500 Makefiles.

Outcome

No more manual work is needed for the security team and we now have direct links from a DSA-NNN/DLA-NNN to the email in our mailing list archive. This was not possible before. The search results became more accurate.

But we still host a lot of other old content on the Debian web pages which may be removed in the future.

[1] https://www.debian.org/security/#infos

[2] https://www.debian.org/security/oval/

[3] https://salsa.debian.org/security-tracker-team/security-tracker/-/raw/master/data/DSA/list

[4] https://www.debian.org/devel/website/stats

[5] https://salsa.debian.org/webmaster-team/webwml/-/commit/2aa73ff15bfc4eb2afd85c

After more than one a year, a new minor FAI version is available, but it includes some interesting new features.

Here a the items from the NEWS file:

fai (6.2) unstable; urgency=low

• fai-cd can now create live images
• Use systemd during installation
• New feature: run FAI inside a screen or tmux session
• fai-diskimage: do not use compression of qemu-img which is slow instead provide .qcow2.zst, add option -C
• fai-kvm: add support for booting from USB storage
• new tool mk-data-partition adds a data partition to an ISO
• easy installation of packages from /pkgs/<CLASS> directories
• new helper functions for creating custom list of disks
• new method detect:// for FAI_CONFIG_SRC

In the past the command fai-cd was only used for creating installation ISOs, that could be used from CD or USB stick. Now it possible to create a live ISO. Therefore you create your live chroot environment using 'fai dirinstall' and then convert it to a bootable live ISO using fai-cd. See man fai-cd(8) for an example.

Years ago I had the idea to use the remaining disk space on an USB stick after copying an ISO onto it. I've blogged about this recently:

https://blog.fai-project.org/posts/extending-iso-images/

The new FAI version includes the tool mk-data-partition for adding a data partition to the ISO itself or to an USB stick.

FAI detects this data partition, mounts it to /media/data and can then use various configurations from it. You may want to copy your own set of .deb packages or your whole FAI config space to this partition. FAI now automatically searches this partition for usable FAI configuration data and packages. FAI will install all packages from pkgs/<CLASSNAME> if the equivalent class is defined. Setting FAI_CONFIG_SRC=detect:// now looks into the data partition for the subdirectory 'config' and uses this as the config space. So it's now possible to modify an existing ISO (that is read-only) and make changes to the config space. If there's no config directory in the data partition FAI uses the default location on the ISO.

The tool fai-kvm, which starts virtual machines can now boot an ISO not only as CD but also as USB stick.

Sometimes users want to adjust the list of disks before the partitioning is startet. Therefore FAI provides several new functions including

• smallestdisk()
• largestdisk()
• matchdisks()

You can select individual disks by their model name or even the serial number.

Two new FAI flags were added (tmux and screen) that make it easy to run FAI inside a tmux or screen session.

And finally FAI uses systemd. Yeah!

This technical change was waiting since 2015 in a merge request from Moritz 'Morty' Strübe, that would enable using systemd during the installation. Before FAI still was using old-style SYSV init scripts and did not started systemd. I didn't tried to apply the patch, because I was afraid that it would need much time to make it work. But then in may 2023 Juri Grabowski just gave it a try at MiniDebConf Hamburg, and voilà it just works! Many, many thanks to Moritz and Juri for their bravery.

The whole changelog can be found at https://tracker.debian.org/media/packages/f/fai/changelog-6.2

New ISOs for FAI are also available including an example of a Xfce desktop live ISO: https://fai-project.org/fai-cd/

The FAIme service for creating customized installation ISOs will get its update later.

The new packages are available for bookworm by adding this line to your sources.list:

deb https://fai-project.org/download bookworm koeln

Some years ago a customer needed a live ISO containing a customized FAI environment (not for installing but for extended hardware stress tests), but on an USB stick with the possibility to store the logs of the tests on the USB stick. But an ISO file system (iso9660) remains read-only, even when put onto an USB stick. I had the idea to add another partition onto the USB stick after the ISO was written to it (using cp or dd). You can use fdisk with an ISO file, add a new partition, loop mount the ISO and format this partition. That's all. This worked perfect for my customer.

I forgot this idea for a while but a few weeks ago I remembered it. What could be possible when my FAI (Fully Automatic Installation) image would also provide such a partition? Which things could be provided on this partition?

Could I provide a FAI ISO and my users would be able to easily put their own .deb package onto it without remastering the ISO or building an ISO on their own?

Now here's the shell script, that extends an ISO or an USB stick with an ext4 or exFAT partition and set the file system label to MY-DATA.

https://github.com/faiproject/fai/blob/master/bin/mk-data-partition

Examples how to use mk-data-partition

Add a data partition of size 1G to the Debian installer ISO using an ext4 partition
# mk-data-partition -s 1G debian-12.2.0-amd64-netinst.iso

Create the data partition using an exFAT file system on USB named /dev/sdb.
First copy (or dd) the ISO onto the USB stick. Then add the data partition
to the USB stick.
# cp faicd64-large_6.0.3.iso /dev/sdb
# mk-data-partition -F /dev/sdb

Create the data partition and copy directories A and B to it
# mk-data-partition -c debian-12.2.0-amd64-netinst.iso A B

The next FAI version will use this in different parts of an installation. A blog post about this will follow.

A new idea for our Debian installer ISO

Here are my ideas how the Debian installer could use such a partition if it automatically detects and mounts it (by it's file system label):

• Look for a preseed file and use this (without explicitly specifying it via boot parameters)
• User could provide its own set of packages that the installer will install
• d-i could show a menu (like tasksel) and the user can select packages from the data partition
• Save installation logs onto this partition
• Provide a postinst script, that is run during the first boot of the newly installed system

The advantage of this approach is that there's no need for the user to remaster the official Debian installer ISO, which is not easy for end users. We only have to extend the installer to use files from this data partition in some portions of the installation. Additional udebs, packages or firmware could automatically be used by the installer. Companies could easily create an OEM installer of Debian.

What do you think about this idea? Please send feedback to lange@debian.org

The FAI.me service for creating customized installation and cloud images has a new feature by a user requested it.

You can now enable installing recommended packages for your custom package list. By default FAIme does only install the dependencies needed, but not the recommended packages.

This was a very easy enhancement, only a few lines in the web interface and nearly no changes in the backend were needed.

The web interface of the FAI.me service is available at

https://fai-project.org/FAIme

The FAI.me service for creating customized installation and cloud images now supports the backports kernel for the stable release Debian 12 (aka bookworm). If you enable the backports option in the web interface, you currently get kernel 6.4. This will help you if you have newer hardware that is not support by the default kernel 6.1. The backports option is also still available for the older distributions.

The web interface of the FAI.me service is available at

https://fai-project.org/FAIme