The Fully Automatic Installation (FAI) is using a read-only nfsroot since it's very beginning. This is also used in diskless clients enviroments and in the LTSP (Linux Terminal Server Project).
During a network installation the clients are running as diskless clients, so the installation has full access to the local hard disk which is not in use. But we need some files to be writable on the read-only nfsroot. In the past we've created symlinks to a ram disk. Later we used aufs (another union fs), a kernel module for doing union mounts of several file systems. Putting a ram disk on top of the read-only nfsroot with aufs makes the nfsroot writable. But aufs was not available in kernel 4.X any more. It was replaced by overlayfs.
The initrd of FAI mounts the nfsroot read only and then puts a tmpfs ram disk on top of it using overlayfs. The result is a new merged file system which is writable. This works nicely since several years when using NFSv3. But when using NFSv4 we can read from a file, but writing always reported
openat(AT_FDCWD,....) = -1 EOPNOTSUPP (Operation not supported)
After some days of debugging overlayfs and NFS v4, I found that it's a complicated mixture of NFS and acl support (POSIX and nfs4 acl) and what overlayfs expects from the file systems in respect to certain xattr. Overlayfs uses calls like
setxattr(work/work, "trusted.overlay.opaque", "0", 1, 0x0) = 0
and writing to a file used
getxattr("/b/lower/etc/test1", "system.nfs4_acl", ....) = 80
without any errors. When talking to some overlayfs guys they ask me to disable acl for the exported NFS file system. There's an noacl option listed on nfs(5), but it's for NFS version 2 and 3 only, not for NFS v4. You cannot disable ACL on a NFS v4 mount.
In the end the solution was to disable ACL on the whole file system on the NFS server, which is exported to the clients. If you have a ext4 file system this works on the NFS server by doing
# mount -oremount,noacl $EXPORTED_FS
After that, overlayfs will detect that ACL's are not support on the NFS mount and behaves as expected allowing writes to a file.
You will need to use dracut instead of initramfs-tools for creating the initrd. The later is using busybox or klibc tools inside the initrd. Both do not support NFS v4 mounts (https://bugs.debian.org/409271).
Dracut is using the normal libc based executables. The Debian package
of dracut supports the kernel cmdline option rootovl. This is an
example of the kernel cmdline options:
rootovl ip=dhcp root=11.22.33.44:/srv/fai/nfsroot
This mounts a read only nfsroot and puts a tmpfs on top for making it writable.