View Issue Details

IDProjectCategoryView StatusLast Update
0000144CloudGeneralpublic2022-09-15 06:38
ReporterJean-François Landry Assigned ToNeil Hanlon  
PrioritynormalSeverityblockReproducibilityalways
Status assignedResolutionopen 
Summary0000144: Rocky 9 OpenStack image has no installed BIOS GRUB boot sector
Descriptionhttps://dl.rockylinux.org/pub/rocky/9/images/x86_64/Rocky-9-GenericCloud-9.0-20220706.0.x86_64.qcow2

is not bootable on OpenStack using the standard, default BIOS compatibility classic method.
The boot sector is empty. Looks like grub was never installed when provisioning the image.

Here is the xxd output of the first 512 bytes of the image when converted to RAW (only contains the partition table):

00000000: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000010: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000020: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000030: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000040: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000050: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000060: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000070: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000080: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000090: 0000 0000 0000 0000 0000 0000 0000 0000 ................
000000a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
000000b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
000000c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
000000d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
000000e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
000000f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000100: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000110: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000120: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000130: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000140: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000150: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000160: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000170: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000180: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000190: 0000 0000 0000 0000 0000 0000 0000 0000 ................
000001a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
000001b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
000001c0: 0200 eeff ffff 0100 0000 ffff 3f01 0000 ............?...
000001d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
000001e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
000001f0: 0000 0000 0000 0000 0000 0000 0000 55aa ..............U.
Steps To ReproduceUpload to OpenStack, attempt to launch an instance, the console stays stuck on "Booting from Hard Disk..."
TagsNo tags attached.

Relationships

has duplicate 0000154 closedLouis Abel Rocky-Linux-8 Rocky 9 x86_64 generic cloud image appears to be missing legacy boot support 

Activities

Louis Abel

Louis Abel

2022-08-30 21:20

administrator   ~0000487

Hello. Please try the following images:

http://dl.rockylinux.org/stg/rocky/9/images/x86_64/Rocky-9-GenericCloud-9.0-20220830.0.x86_64.qcow2
http://dl.rockylinux.org/stg/rocky/9/images/x86_64/Rocky-9-GenericCloud.latest.x86_64.qcow2
Jean-François Landry

Jean-François Landry

2022-08-31 15:57

reporter   ~0000491

Looks fine aside from a small locale / glibc langpack issue.

According to localectl: System Locale: LANG=en_US.UTF-8

Installed glibc langpacks: glibc-minimal-langpack-2.34-28.el9_0.x86_64

Which results in warning when running sudo dnf update and likely other commands: "Failed to set locale, defaulting to C.UTF-8"

Can be fixed by installing "glibc-langpack-en" or using "LANG=C.UTF-8" by default.

Otherwise, the image boots in BIOS mode on openstack, cloud-init/growpart work, seems totally functional.
Robert Cohen

Robert Cohen

2022-09-01 02:53

reporter   ~0000492

trivial issue.
Since this image is for use in virtual environments, it would be nice to have the virt-what package installed by default since some software uses it to detect
that this is a virtual environment.
Fritz Elfert

Fritz Elfert

2022-09-14 11:59

reporter   ~0000533

In addition to the missing glibc-langpack-en mentioned earlier, I just noticed another odd
behavior with the new image Rocky-9-GenericCloud-9.0-20220830.0.x86_64.qcow2:

With the old image, output of cloud-init was logged to the console (/dev/ttyS0) as well as to /var/log/cloud-init-output.log

With the new image, logging of cloud-init on the console is gone.

This affects automated provsioning of custom snapshots here, where we monitor a VMs console for specific patterns
in order to detect errors in an early stage and then prevent out automated provisioning process from createing a
possibly broken snapshot.

I just tested current Alma9 in regards to that and it still produces cloud-init output on the console. Since both provide
the same cloud-init package version, I assume this is not an issue of cloud-init itself but some side effect of a generic
configuration change. Will test the same with rhel-baseos-9.0-x86_64-kvm.qcow2 from RedHat later.

PS:
I would look into your image-creation process myself, but I can't find any public (git?) repo. The only one that
I found is https://github.com/rocky-linux/sig-cloud-instance-images but that one appears to be for docker.
So: Where the heck are the scripts that you use for generating the official images?
Neil Hanlon

Neil Hanlon

2022-09-14 13:20

administrator   ~0000534

Working on new images this week which will address these concerns. I'll have to look into the cloud-init one a bit more, but that shouldn't be too hard. I suspect something going on with the console parameter.

@Fritz regarding the scripts for image generation. we're moving things into our toolkit in a subproject. Specifically ,the image building script is here: https://git.resf.org/sig_core/toolkit/src/branch/devel/iso/empanadas/empanadas/scripts/build_image.py

I need to work on some docs for usage, but largely that's how the images get generated in our semi-hand-fed pipeline.
Louis Abel

Louis Abel

2022-09-15 03:32

administrator   ~0000536

Hello.

All kickstarts are here: https://git.resf.org/sig_core/kickstarts or the mirror here: https://github.com/rocky-linux/kickstarts

The langpack* packages have always been excluded in our images since we started making them. This was a carry over from how CentOS does it (including in stream 9). We usually remove and add packages that we may feel that do or don't fit as well.

For context with the locale piece: As far as I'm aware, the locale piece is actually a bug of cloud init that should be fixed in 22.2 (or a patched 22.1 that's currently in CentOS Stream 9). I did a hotfix for the next image we build to force the locale to be C.UTF-8 in the meantime. See this commit: https://git.resf.org/sig_core/kickstarts/commit/ebf7a38c6b3f81512daaf0f4938148c1a576750b

As for the console portion, it's a strange case. We have tried to adjust the parameters in the kickstart bootloader line and we have also tried to use sed and grubby within %post to try to resolve the problem. What ends up happening with the changes we try is we no longer get any console on ttyS0, not even a login prompt. We have also tried to inherit a fedora-like configuration specifying `console=tty1 console=ttyS0,115200n8` at the end of the bootloader configuration, but still no dice. Even more interesting, you can run a few grubby commands after logging in to "fix" the issue, but it defeats the purpose of not being able to see the cloud init messages on first boot.

As an aside, if you would like to try out how we build our images, you could try this (it may require you to make some directories or install packages on your host system; we also use Fedora in this example):

git clone https://git.resf.org/sig_core/toolkit.git -b devel
cd toolkit/iso/empanadas
podman build --build-arg BRANCH=r9 -t empanadas:r9 -f Containerfile.imagefactory
podman run -it --rm -e LIBVIRT_DEFAULT_URI -v /var/run/libvirt/:/var/run/libvirt/ -v /var/lib/imagefactory/:/var/lib/imagefactory/:rw --network host --security-opt label=disable --privileged --device fuse empanadas:r9 build-image --version 9 --type GenericCloud --debug

This will leave an image in /var/lib/imagefactory/storage with a .body name. It'll be in raw format. You can then use this against virt-install or otherwise.

I'll keep trying to look into where the issue is in the meanwhile. We are open to any suggestions or help as well to try to resolve the console issue.
Louis Abel

Louis Abel

2022-09-15 06:38

administrator   ~0000537

After a bit of work we believe we have the issues resolved. We'll have images built through the week/weekend and pushed to our mirrors and we'll notify when this happens.

Resolved in the following commits:

Remove kernel parameters from icicle configuration: https://git.resf.org/sig_core/toolkit/commit/eeae7ff5ab73f27229bbc578aeeb3750619101b9
Fixes grub2-pc package naming and removes sed in %post: https://git.resf.org/sig_core/kickstarts/commit/0a16a1b5cf1b3fbfe6bc8b95fcf19b645fef0389
Works around cloud-init bug that should be backported in 9.1's cloud-init or fixed in 22.2: https://git.resf.org/sig_core/kickstarts/commit/ebf7a38c6b3f81512daaf0f4938148c1a576750b
Adds virt-what to package list: https://git.resf.org/sig_core/kickstarts/commit/008bf056b923231c0f9100fabef54d2d407c4304

Issue History

Date Modified Username Field Change
2022-07-14 21:21 Jean-François Landry New Issue
2022-07-20 04:45 Louis Abel Assigned To => Neil Hanlon
2022-07-20 04:45 Louis Abel Status new => assigned
2022-07-20 04:45 Louis Abel Relationship added has duplicate 0000154
2022-08-30 21:20 Louis Abel Note Added: 0000487
2022-08-31 15:57 Jean-François Landry Note Added: 0000491
2022-09-01 02:53 Robert Cohen Note Added: 0000492
2022-09-14 11:59 Fritz Elfert Note Added: 0000533
2022-09-14 13:20 Neil Hanlon Note Added: 0000534
2022-09-15 03:32 Louis Abel Note Added: 0000536
2022-09-15 06:38 Louis Abel Note Added: 0000537