gzip from Ubuntu Jammy doesn't execute

[iBug]

Version

Microsoft Windows [Version 10.0.19044.1586]

WSL Version

• WSL 2
• WSL 1

Kernel Version

4.4.0-19041-Microsoft

Distro Version

Ubuntu 22.04 "Jammy Jellyfish"

Other Software

GZip version 1.10-4ubuntu3 and 1.10-4ubuntu4 amd64.

Repro Steps

• apt install gzip=1.10-4ubuntu3 then run gzip.
• apt install gzip=1.10-4ubuntu4 then run gzip.

Expected Behavior

No error shows up.

Actual Behavior

sh: 1: gzip: Exec format error

The binary doesn't execute, so no strace.

Diagnostic Logs

• Report on Launchpad: LP#1966849
• Possibly related: gzip from Ubuntu Eoan doesn't execute #4461.

Similar to that one, the same binary runs perfectly OK on a native Ubuntu Jammy machine. However, this time the binary is 97520 bytes and no section points outside this range.

Dissected binary using Wireshark: https://paste.ubuntu.com/p/nc2v6ZSRHW/

The previous version gzip 1.10-4ubuntu1 is fine, so I've installed that one instead and setting apt-mark hold for now.

It's very interesting that only gzip is found problematic. And it's the same program as 3 years ago. Time to wonder if gzip has any magic to break on WSL1.

[hzqmwne]

The old version gzip 1.10-4ubuntu1 works well, and the new version gzip 1.10-4ubuntu3 triggers this issue.

I compare their program headers by readelf -l gzip command, and here are the LOAD segments:
For 1.10-4ubuntu1 (which has no bug):

  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  ...
  LOAD           0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000001fc0 0x0000000000001fc0  R      0x1000
  LOAD           0x0000000000002000 0x0000000000002000 0x0000000000002000
                 0x000000000000e405 0x000000000000e405  R E    0x1000
  LOAD           0x0000000000011000 0x0000000000011000 0x0000000000011000
                 0x00000000000035d0 0x00000000000035d0  R      0x1000
  LOAD           0x0000000000014690 0x0000000000015690 0x0000000000015690
                 0x0000000000000d88 0x0000000000000d88  RW     0x1000
  LOAD           0x0000000000000000 0x0000000000018000 0x0000000000018000
                 0x0000000000000000 0x00000000000ca810  RW     0x1000
  ...

and for 1.10-4ubuntu3 (which has the bug):

  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  ...
  LOAD           0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000001fa8 0x0000000000001fa8  R      0x1000
  LOAD           0x0000000000002000 0x0000000000002000 0x0000000000002000
                 0x000000000000e319 0x000000000000e319  R E    0x1000
  LOAD           0x0000000000011000 0x0000000000011000 0x0000000000011000
                 0x00000000000036a8 0x00000000000036a8  R      0x1000
  LOAD           0x0000000000016690 0x0000000000016690 0x0000000000016690
                 0x0000000000000d88 0x00000000000cc180  RW     0x2000
  ...

Notice that the buggy version has a strange LOAD segment with Align 0x2000, and after patching the 0x2000 to 0x1000 (by modifying only one byte of gzip binary at offset 0x189 from 0x20 to 0x10), the bug disappears and the patched binary works well!

So, maybe WSL1 makes a wrong assumption that the p_align value is 0x1000. It is just a bug in WSL1 rather than Ubuntu, and seems quite easy to fix.

[iBug]

Gzip 1.10-4ubuntu4 is out and is also affected. The binary is identical to 1.10-4ubuntu3 except for some "ID" part, namely:

• 20 bytes starting from 0x378
• 38 bytes starting from 0x17418
• 4 bytes starting from 0x17448

File length remains 97520 (0x17CF0)

The new binary, as you'd imagine, works well after manually patching byte 0x189 from 0x20 to 0x10.

[hzqmwne]

By reversing lxcore.sys, the real reason of this bug is that WSL1 assumes all p_align member in PT_LOAD program headers must be the same value, which is not correct. (See elf(5) — Linux manual page, there is not such assumption for p_align)

Part of decompilation pesudo C code of lxcore.sys (version 10.0.22000.1 from Windows 11 21H2 22000.556):
lxcore.sys.zip

__int64 __fastcall LxpElfInfoParse(__int128 *a1, unsigned __int64 a2, _OWORD *a3)    // RVA 0x1C004DC60
{
...
    if ( (_DWORD)v61 == 0x464C457F )    // "\x7fELF"
    {
...
      v46 = 0i64;
...
        v19 = *(__m128i *)v18;    // typeof(v18) is "Elf64_PHdr *"
        v58 = v19;
        v51 = v19;
        v20 = *((_OWORD *)v18 + 1);    // Elf64_PHdr.p_vaddr
        v60 = v20;
        v52 = v20;
        v21 = *((__m128i *)v18 + 2);
        v59 = v21;
        v53 = v21;
        v22 = *((_QWORD *)v18 + 6);    // Elf64_PHdr.p_align
        v54 = v22;
        v23 = _mm_cvtsi128_si32(v19);    // Elf64_PHdr.p_type
        if ( v23 == 3 )    // PT_INTERP
        {
...
        if ( v23 == 1 )    // PT_LOAD
        {
...
            if ( !v54 || (v54 & 0xFFF) != 0 )    // check p_align is multiple of page size
            {
              v5 = "LxpElfInfoParse: LocalProgramHeader.Align\n";
              v6 = 541;
              goto LABEL_5;
            }
            if ( (unsigned __int64)v52 % v54 != v24 % v54 )    // check "p_vaddr % p_align == p_offset % p_align"
            {
              v5 = "LxpElfInfoParse: LocalProgramHeader.VirtualAddress\n";
              v6 = 554;
              goto LABEL_5;
            }
            if ( v46 )
            {
              if ( v46 != v54 )    // Bug here! WSL1 assumes all `p_align` member in `PT_LOAD` program headers must be the same value, which is not correct. 
              {
                v5 = "LxpElfInfoParse: LoadHeaderAlignment\n";
                v6 = 567;
                goto LABEL_5;
              }
            }
            else
            {
              v46 = v54;
            }
...
}

LxpElfInfoParse function (RVA 0x1C004DC60) at lxcore.sys parses ELF file. See pesudo code above, 0x464C457F is the ELF magic number ("\x7fELF"), and v23 is p_type member of Elf64_Phdr. v23 == 1 means PT_LOAD (see here), and v54 is p_align member of Elf64_Phdr.
v46 is initialized to 0, and when it firstly meets v54, it will be set to its value. When v46 secondly meets v54, it checks v54 should equals to the old v46 value, which causes this issue.

So, for example, after patching all the p_align value from 0x1000 to 0x2000 (offset 0xE1, 0x119 and 0x151 of gzip 1.10-4ubuntu4), the new binary can also works well.

[Zenderable]

Same issue here (Ubuntu 22.04 & WSL1). I can't start VS Code Server due this issue:

Installing VS Code Server for x64 (dfd34e8260c270da74b5c2d86d61aee4b6d56977)
Downloading: 100%
/usr/bin/gzip: 1: ELF����: Permission denied
/usr/bin/gzip: 3: ���: Permission denied
/usr/bin/gzip: 4: Syntax error: "(" unexpected
tar: Child returned status 2
tar: Error is not recoverable: exiting now

tar is unable to read /home/stumski/.vscode-server/bin/dfd34e8260c270da74b5c2d86d61aee4b6d56977-1650739878.tar.gz. Either the file is corrupt or tar has an issue.
There's a known WSL issue with tar on Ubuntu 19.10.
See workaround in https://github.com/microsoft/vscode-remote-release/issues/1856.
Reload the window to initiate a new server download.

stumski@C-H50K6G3:~$ gzip
-bash: /usr/bin/gzip: cannot execute binary file: Exec format error

Older version works well:

sudo dpkg -i ./gzip_1.10-4ubuntu1_amd64.deb
stumski@C-H50K6G3:~$ sudo dpkg -i ./gzip_1.10-4ubuntu1_amd64.deb
dpkg: warning: downgrading gzip from 1.10-4ubuntu4 to 1.10-4ubuntu1
(Reading database ... 33879 files and directories currently installed.)
Preparing to unpack ./gzip_1.10-4ubuntu1_amd64.deb ...
Unpacking gzip (1.10-4ubuntu1) over (1.10-4ubuntu4) ...
Setting up gzip (1.10-4ubuntu1) ...
Processing triggers for install-info (6.8-4build1) ...
Processing triggers for man-db (2.10.2-1) ...

stumski@C-H50K6G3:~$ sudo apt-mark hold gzip
gzip set on hold.

stumski@C-H50K6G3:~$ gzip
gzip: compressed data not written to a terminal. Use -f to force compression.
For help, type: gzip -h
stumski@C-H50K6G3:~$ code
Updating VS Code Server to version dfd34e8260c270da74b5c2d86d61aee4b6d56977
Removing previous installation...
Installing VS Code Server for x64 (dfd34e8260c270da74b5c2d86d61aee4b6d56977)
Downloading: 100%
Unpacking: 100%
Unpacked 2341 files and folders to /home/stumski/.vscode-server/bin/dfd34e8260c270da74b5c2d86d61aee4b6d56977.
stumski@C-H50K6G3:~$

[dreamlayers]

So, for example, after patching all the p_align value from 0x1000 to 0x2000 (offset 0xE1, 0x119 and 0x151 of gzip 1.10-4ubuntu4), the new binary can also works well.

Actually those were all 0x1000 already. In gzip 1.10-4ubuntu4 I only had to change the value at offset 0x189 using
echo -en '\x10' | sudo dd of=/usr/bin/gzip count=1 bs=1 conv=notrunc seek=$((0x189))

[LostInBrittany]

It also happens in WSL2, I have just found it (unable to start VScode because of it)

Patch by @dreamlayers worked beautifully, thanks!

[Zenderable]

@LostInBrittany Are you absolutely certain that your distribution is running under WSL2? You can list all distros with wsl -l -v. I ask, because when switching to WSL2 it didn't happen to me.
I tried patch by @dreamlayers on newest gzip and it's working as it should!

[LostInBrittany]

@Zenderable OMG, you're right, my bad... I have two Windows 11 computers, both with Windows Insiders, both with WSL, and I was sure both of them used the same WSL version... And not, this one has WSL 1, and the other WSL 2 (and it hasn't the problem, you're right). Sorry about that!

[landall]

I trigger the same issue. WSL 2 is ok, WSL 1 is wrong.

[stong]

By reversing lxcore.sys, the real reason of this bug is that WSL1 assumes all p_align member in PT_LOAD program headers must be the same value, which is not correct. (See elf(5) — Linux manual page, there is not such assumption for p_align)
...

IDA Pro and Hexrays to the rescue again!