In two weeks DebConf24, the Debian conference starts in Busan, South Korea. Therefore I've added support for the Korean language into the web service of FAI:

https://fai-project.org/FAIme/

Another new feature of the FAIme service will be announced at DebConf24 in August.

Posted Mon Jul 15 13:01:47 2024 Tags:

A new FAI version was released and the FAIme service is using this new release. You can now also create installation images for Debian 13 (testing aka Trixie).

https://fai-project.org/FAIme/

Another new feature of the FAIme service will be announced at DebConf24 in August.

Posted Tue Jun 25 14:39:36 2024 Tags:

In January I've removed tens of thousands of web pages on www.debian.org. Have you noticed it?

In the past

From 1997 onwards, we had web pages for security announcements. We had to manually prepare a .data and a .wml file which then generated a web page for each security announcement (DSA or DLA). We have listed the 6 most recent messages in a short list that was created from these files. Most of the work that went into the Debian web pages was creating these files.

Our search engine often listed the pages with security announcements instead of a more relevant web page for a particular topic.

Preparation

At DebConf Kosovo (2022) I started with a proof of concept and wrote a script, that generates this list without using the .data/.wml files in the Git repository, but instead reading the primary sources of security information[1]. This new list now includes links to the security tracker and the email of the announcement.

Following web pages and scripts were also using these .data and .wml files:

  • OVAL files
  • RSS feeds for security announcements (and LTS)
  • Apache config file for mapping URLs from dsa-NNN to YEAR/dsa-NNN
  • A huge list of crossreferences between DSA and CVE numbers

Before I could remove all the security web pages, I had to adjust the scripts, that create the above information.

When I looked at the OVAL files and the apache logs of our web server, I saw that more than 99% of the web traffic was generated by these XML files (134TB of 135TB total in two weeks). They were not compressed and were around 50MB in size. With the help of Carsten Schönert we managed to modify the python scripts that generate this OVAL file without using the .data/.wml files and now we only provide bzip2 compressed XML files[2].

The RSS feeds are created by the new Perl script which reads the DSA/DLA list the security tracker and determines the URL of the email of all entries. This script also generates the list of the most recent DSA/DLA entries. Currently we show the last 350 entries which covers more than the last year and includes links to the announcement email and the security tracker.

The huge list of crossreferences is not needed any more, since the mapping of CVE to DSA is already included in the DSA list[3] of the security tracker.

The amount of translations of the DSA/DLA was very different. French translations were almost all done, but all other languages did translations for a couple of months or years only. E.g. in 2022, Italian had 2 translations, Russian 15, Danish 212, French and English each 279. But from 2023 on only French translations were made. By generating the list of DSA/DLA we lost the ability to translate these web pages, but since these announcements are made of simple, identical sentences it is easy to use an automatic translation service if needed.

Now the translation statistics of all web pages are more accurate. Instead of 12200 pages that need to be translated (including all these old DSA/DLA) there are now only 2500 pages to translate[4]. Languages that had a lot of old translations of DSA/DLA lost some percentage but languages that are doing translations of newer web pages won in the statistics of how many pages are translated. Examples:

Before

German (de)   3501  28.5%
Italian (it)  1005   8.2%
Danish (da)   6336  51.7%

After

German (de)   1486  59.0%
Italian (it)   909  36.1%
Danish (da)    982  39.0%

Cleanup of all the security web pages

Finally in January, I could remove all web pages of the security announcements in one git commit[5]. Using several git rm -rf commands this commit removed 54335 files, including around 9650 DSA/DLA data files, 44189 wml files, nearly 500 Makefiles.

Outcome

No more manual work is needed for the security team and we now have direct links from a DSA-NNN/DLA-NNN to the email in our mailing list archive. This was not possible before. The search results became more accurate.

But we still host a lot of other old content on the Debian web pages which may be removed in the future.

[1] https://www.debian.org/security/#infos

[2] https://www.debian.org/security/oval/

[3] https://salsa.debian.org/security-tracker-team/security-tracker/-/raw/master/data/DSA/list

[4] https://www.debian.org/devel/website/stats

[5] https://salsa.debian.org/webmaster-team/webwml/-/commit/2aa73ff15bfc4eb2afd85c

Posted Mon May 6 16:58:01 2024 Tags:

After more than one a year, a new minor FAI version is available, but it includes some interesting new features.

Here a the items from the NEWS file:

fai (6.2) unstable; urgency=low

  • fai-cd can now create live images
  • Use systemd during installation
  • New feature: run FAI inside a screen or tmux session
  • fai-diskimage: do not use compression of qemu-img which is slow instead provide .qcow2.zst, add option -C
  • fai-kvm: add support for booting from USB storage
  • new tool mk-data-partition adds a data partition to an ISO
  • easy installation of packages from /pkgs/<CLASS> directories
  • new helper functions for creating custom list of disks
  • new method detect:// for FAI_CONFIG_SRC

In the past the command fai-cd was only used for creating installation ISOs, that could be used from CD or USB stick. Now it possible to create a live ISO. Therefore you create your live chroot environment using 'fai dirinstall' and then convert it to a bootable live ISO using fai-cd. See man fai-cd(8) for an example.

Years ago I had the idea to use the remaining disk space on an USB stick after copying an ISO onto it. I've blogged about this recently:

https://blog.fai-project.org/posts/extending-iso-images/

The new FAI version includes the tool mk-data-partition for adding a data partition to the ISO itself or to an USB stick.

FAI detects this data partition, mounts it to /media/data and can then use various configurations from it. You may want to copy your own set of .deb packages or your whole FAI config space to this partition. FAI now automatically searches this partition for usable FAI configuration data and packages. FAI will install all packages from pkgs/<CLASSNAME> if the equivalent class is defined. Setting FAI_CONFIG_SRC=detect:// now looks into the data partition for the subdirectory 'config' and uses this as the config space. So it's now possible to modify an existing ISO (that is read-only) and make changes to the config space. If there's no config directory in the data partition FAI uses the default location on the ISO.

The tool fai-kvm, which starts virtual machines can now boot an ISO not only as CD but also as USB stick.

Sometimes users want to adjust the list of disks before the partitioning is startet. Therefore FAI provides several new functions including

  • smallestdisk()
  • largestdisk()
  • matchdisks()

You can select individual disks by their model name or even the serial number.

Two new FAI flags were added (tmux and screen) that make it easy to run FAI inside a tmux or screen session.

And finally FAI uses systemd. Yeah!

This technical change was waiting since 2015 in a merge request from Moritz 'Morty' Strübe, that would enable using systemd during the installation. Before FAI still was using old-style SYSV init scripts and did not started systemd. I didn't tried to apply the patch, because I was afraid that it would need much time to make it work. But then in may 2023 Juri Grabowski just gave it a try at MiniDebConf Hamburg, and voilà it just works! Many, many thanks to Moritz and Juri for their bravery.

The whole changelog can be found at https://tracker.debian.org/media/packages/f/fai/changelog-6.2

New ISOs for FAI are also available including an example of a Xfce desktop live ISO: https://fai-project.org/fai-cd/

The FAIme service for creating customized installation ISOs will get its update later.

The new packages are available for bookworm by adding this line to your sources.list:

deb https://fai-project.org/download bookworm koeln

Posted Wed Jan 24 12:12:31 2024 Tags:

Some years ago a customer needed a live ISO containing a customized FAI environment (not for installing but for extended hardware stress tests), but on an USB stick with the possibility to store the logs of the tests on the USB stick. But an ISO file system (iso9660) remains read-only, even when put onto an USB stick. I had the idea to add another partition onto the USB stick after the ISO was written to it (using cp or dd). You can use fdisk with an ISO file, add a new partition, loop mount the ISO and format this partition. That's all. This worked perfect for my customer.

I forgot this idea for a while but a few weeks ago I remembered it. What could be possible when my FAI (Fully Automatic Installation) image would also provide such a partition? Which things could be provided on this partition?

Could I provide a FAI ISO and my users would be able to easily put their own .deb package onto it without remastering the ISO or building an ISO on their own?

Now here's the shell script, that extends an ISO or an USB stick with an ext4 or exFAT partition and set the file system label to MY-DATA.

https://github.com/faiproject/fai/blob/master/bin/mk-data-partition

Examples how to use mk-data-partition

Add a data partition of size 1G to the Debian installer ISO using an ext4 partition
# mk-data-partition -s 1G debian-12.2.0-amd64-netinst.iso

Create the data partition using an exFAT file system on USB named /dev/sdb.
First copy (or dd) the ISO onto the USB stick. Then add the data partition
to the USB stick.
# cp faicd64-large_6.0.3.iso /dev/sdb
# mk-data-partition -F /dev/sdb

Create the data partition and copy directories A and B to it
# mk-data-partition -c debian-12.2.0-amd64-netinst.iso A B

The next FAI version will use this in different parts of an installation. A blog post about this will follow.

A new idea for our Debian installer ISO

Here are my ideas how the Debian installer could use such a partition if it automatically detects and mounts it (by it's file system label):

  • Look for a preseed file and use this (without explicitly specifying it via boot parameters)
  • User could provide its own set of packages that the installer will install
  • d-i could show a menu (like tasksel) and the user can select packages from the data partition
  • Save installation logs onto this partition
  • Provide a postinst script, that is run during the first boot of the newly installed system

The advantage of this approach is that there's no need for the user to remaster the official Debian installer ISO, which is not easy for end users. We only have to extend the installer to use files from this data partition in some portions of the installation. Additional udebs, packages or firmware could automatically be used by the installer. Companies could easily create an OEM installer of Debian.

What do you think about this idea? Please send feedback to lange@debian.org

Posted Sun Dec 17 00:14:37 2023 Tags:

The FAI.me service for creating customized installation and cloud images has a new feature by a user requested it.

You can now enable installing recommended packages for your custom package list. By default FAIme does only install the dependencies needed, but not the recommended packages.

This was a very easy enhancement, only a few lines in the web interface and nearly no changes in the backend were needed.

The web interface of the FAI.me service is available at

https://fai-project.org/FAIme

Posted Tue Dec 5 22:08:49 2023 Tags:

The FAI.me service for creating customized installation and cloud images now supports the backports kernel for the stable release Debian 12 (aka bookworm). If you enable the backports option in the web interface, you currently get kernel 6.4. This will help you if you have newer hardware that is not support by the default kernel 6.1. The backports option is also still available for the older distributions.

The web interface of the FAI.me service is available at

https://fai-project.org/FAIme

Posted Fri Sep 8 11:30:01 2023 Tags:

The counter of the FAI.me build service has reached 20.000. This counter was added shortly after the service was started in November 2017. Since then, this service has built more than 21.000 installation images and more than 1300 cloud disk images. In the last few month we had averaged 100 requests per week.

Some statistics which settings are popular:

  • Language/keyboard layout selected

    12000 us
    4000 de
    2500 fr
    800 gb
    500 es
    300 ru
    300 cn
    200 pt

  • Desktop environments selected

    12000 NONE (without any desktop)
    5000 GNOME
    1800 XFCE
    800 KDE
    700 CINNAMON
    700 MATE
    500 LXDE

  • In April 2023, support for building your own Ubuntu installation ISO was added. Since then, 200 Ubuntu ISOs has been created.

  • Packages that are often added: tmux screen apt-transport-https build-essential sudo net-tools mc git wget htop vim curl

  • A postinst script was provided more that 1500 times even though it was not added until 2021.

  • Packages from backports were used 4000 times.

I still have some more ideas for the future: Build your own custom Live ISO

Thanks for all your feedback I got to improve this service.

The build service is available on the FAI project website at https://fai-project.org/FAIme

Posted Thu Jun 15 19:22:28 2023 Tags:

After Debian 12 aka bookworm was released yesterday, I've also created new FAI ISO images using Debian 12.

The defaut ISO (large) uses FAI 6.0.3, kernel 6.1 and can install the XFCE and GNOME desktop without internet connection, since all needed packages are included into the ISO. Additional you can install Ubuntu 22.04 or Rocky Linux 9 with this FAI ISO. During these installations, the packages will be downloade via network. There's also the variant FAI ISO UBUNTU, which includes all Ubuntu packages needed for a Ubuntu server or Ubuntu desktop installation.

If you need a small image, you can take the FAI ISO small, which only includes the packages for a XFCE desktop without LibreOffice. This ISO is only 880MB in size.

Currently I'm working on a new feature, so FAI can create Live images, that are bootable. It's like the tool live-build which Debian uses for their official Debian Live images. A first verison of the ISO using the XFCE desktop can be downloaded from

https://fai-project.org/fai-cd

There you also find all other FAI ISOs.

Posted Sun Jun 11 11:39:59 2023 Tags:

After the initial installation of a new machine, you often want to login as root via ssh. Therefore it's convenient to provide a ssh public key for a passwordless login.

This can now be done by just adding your user name from salsa.debian.org, gitlab.com or github.com. You can also give a customized URL from where to download the keys. Before it was only possible to use a github account name.

The FAI.me build service then creates a customized installation ISO for you, which will automatically install the ssh public key into the root account. Also the ready-to-boot cloud images support this feature.

The build service is available on the FAI project website at https://fai-project.org/FAIme

Posted Thu Apr 27 21:02:07 2023 Tags:

This blog is powered by ikiwiki.