View Issue Details

IDProjectCategoryView StatusLast Update
0006040Rocky-Linux-9grub2public2024-06-03 14:31
ReporterDaniel Riek Assigned ToLouis Abel  
PrioritynormalSeveritymajorReproducibilitysometimes
Status acknowledgedResolutionopen 
Platformaarch64OSRockylinuxOS Version9.3
Summary0006040: Boot failure after recent update on aarch64 Hetzner Ampere server
DescriptionIssue: After upgrading a running Rocky 9.3 installation to the latest grub2 and kernel packages on a Hetzner Ampere aarch64 machine, the machine failed to boot. After stepping into booting the Linux kernel, we simply got an empty screen with a blinking cursor.

The packages we suspect are:
grub2* - 1:2.06-70.el9_3.2.rocky.0.3
kernel* 5.14.0-362.18.1.el9_3.0.1

We reproduced the issue on a second machine: Interactive deployment of 9.3 ISO works. After an upgrade to the above packages, the machine stops booting.

Machine details:

Hetzner RX220

$lscpu
Architecture: aarch64
  CPU op-mode(s): 32-bit, 64-bit
  Byte Order: Little Endian
CPU(s): 80
  On-line CPU(s) list: 0-79
Vendor ID: ARM
  BIOS Vendor ID: Ampere(R)
  Model name: Neoverse-N1
    BIOS Model name: Ampere(R) Altra(R) Processor
    Model: 1
    Thread(s) per core: 1
    Core(s) per socket: 80
    Socket(s): 1
    Stepping: r3p1
    Frequency boost: disabled
    CPU max MHz: 3000.0000
    CPU min MHz: 1000.0000
    BogoMIPS: 50.00
    Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs
Caches (sum of all):
  L1d: 5 MiB (80 instances)
  L1i: 5 MiB (80 instances)
  L2: 80 MiB (80 instances)
NUMA:
  NUMA node(s): 1
  NUMA node0 CPU(s): 0-79
Vulnerabilities:
  Gather data sampling: Not affected
  Itlb multihit: Not affected
  L1tf: Not affected
  Mds: Not affected
  Meltdown: Not affected
  Mmio stale data: Not affected
  Retbleed: Not affected
  Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1: Mitigation; __user pointer sanitization
  Spectre v2: Mitigation; CSV2, BHB
  Srbds: Not affected
  Tsx async abort: Not affected

Steps To Reproduce- Deploy Rocky9.3 on a Hetzner RX220
- Upgrade to
grub2* - 1:2.06-70.el9_3.2.rocky.0.3
kernel* 5.14.0-362.18.1.el9_3.0.1
- Reboot
TagsNo tags attached.

Activities

Daniel Riek

Daniel Riek

2024-03-05 15:46

reporter   ~0006304

Additional information on the machine.
screenshot_1709653513341.png (125,609 bytes)   
screenshot_1709653513341.png (125,609 bytes)   
Louis Abel

Louis Abel

2024-03-06 03:53

administrator   ~0006305

Thank you for the report. We unfortunately don't have access to these types of machines. We will try to and attempt to investigate. In the meanwhile, could you please do a fresh installation of 9.3 and run the following:

dnf install sos -y
sos report --clean

It will obfuscate system information as necessary and produce a tar ball that we can look over while we try to get access to a similar instance.

If possible as well, we'd like to see the output from keyctl. We don't expect secure boot, but it doesn't hurt to know whether or not this is the case.

dnf install keyutils
keyctl show %:.platform
cat /proc/keys
Sherif Nagy

Sherif Nagy

2024-03-06 10:35

manager   ~0006337

It is not secureboot related, we don't have signed shim for aarch64 out yet , and just to be clear, you do reach grub menu and then you select the new kernel and it just goes blank? or you don't reach grub menu?
Daniel Riek

Daniel Riek

2024-03-08 05:21

reporter   ~0006403

Thank you for looking at this. I reinstalled with Rocky 9.3 from vault and it works fine:

kernel-5.14.0-362.8.1.el9_3.aarch64
grub2-common-2.06-70.el9_3.1.rocky.0.2.noarch
grub2-tools-minimal-2.06-70.el9_3.1.rocky.0.2.aarch64
grub2-tools-2.06-70.el9_3.1.rocky.0.2.aarch64
grubby-8.40-63.el9.aarch64
grub2-efi-aa64-2.06-70.el9_3.1.rocky.0.2.aarch64
shim-aa64-15.6-1.el9.rocky.0.2.aarch64

I am not sure whether this is grub2 or kernel. My gut tells me it is actually kernel.

I will see if I can find a time slot to debug a bit more or maybe get Hetzner to help.
Daniel Riek

Daniel Riek

2024-06-03 14:31

reporter   ~0007327

I apologize that I wasn't able to debug further until now.

I can confirm that the issue is gone with current rocky:
kernel-core-5.14.0-427.18.1.el9_4.aarch64
grub2-efi-aa64-2.06-77.el9.aarch64
shim-aa64-15.8-2.el9.aarch64

Issue History

Date Modified Username Field Change
2024-03-05 14:34 Daniel Riek New Issue
2024-03-05 15:46 Daniel Riek Note Added: 0006304
2024-03-05 15:46 Daniel Riek File Added: screenshot_1709653513341.png
2024-03-06 03:53 Louis Abel Assigned To => Louis Abel
2024-03-06 03:53 Louis Abel Status new => acknowledged
2024-03-06 03:53 Louis Abel Note Added: 0006305
2024-03-06 10:35 Sherif Nagy Note Added: 0006337
2024-03-08 05:21 Daniel Riek Note Added: 0006403
2024-06-03 14:31 Daniel Riek Note Added: 0007327