EDIT 2024-03-13: few days after our publication, BishopFox published on the same subject (blog entry).

Context

Optistream helps its clients improving the security of their infrastructure providing the means (technology & expertise) to build and maintain an efficient segmentation. As such, a key stake consists in a deep understanding of a wide variety of network technologies.

As a cybersecurity firm, beyond our core offerings, we are deeply committed to help our clients build and secure their IT landscape. We are therefore concerned by any security issues affecting appliances currently deployed by our customers. That’s why Optistream maintains a continuous monitoring of cyber threats and keeps a close eye on recent security flaws concerning several network and VPN solutions (e.g. Ivanti, Citrix, Fortinet).

We recently delved into analysis of FortiGate appliances, a network security equipment widely deployed and often Internet-facing making them a high-value target for attackers, and are pleased to share in this publication the key highlights of this R&D analysis.

Introduction

Fortinet products have recently encountered significant challenges marked by the disclosure of several highly critical vulnerabilities[2][3][4][5]. Prior to the release of proper updates, customers were left vulnerable to widespread exploitation campaigns targeting these products.

Recognizing the importance of understanding these vulnerabilities, Optistream researchers directed their efforts towards analyzing FortiGate firmware. During our research, we encountered difficulties in obtaining deep and unrestricted access to the FortiGate internals.

In response to these challenges, we present this analysis as a meticulous exploration of the integrity check mechanisms embedded within FortiOS. Additionally, we provide insights into establishing a debugging environment on the latest version of FortiGate-VM (version 7.4.3), which served as the foundation for our research endeavors.

FortiGate-VM

First requirement of our study is getting hold of FortiGate firmware.

Recently, Fortinet stopped to freely distribute its complete firmwares catalog, from now you need a properly registered equipment. However, for those who need to test Fortinet products (among FortiGate) you have possibility to download VMware images[6]. N.B.: In this way, you can only get the last updated release and cannot select a specific version.

Once downloaded, you can import the virtual machine into VMware. For our purposes, we configure port1 network interface with NAT enabled and set a static IP:

FortiGate-VM is now properly configured. We can stop the VM and start looking at the disk:

The files we are mainly interested in are flatkc and rootfs.gz. The former is FortiOS kernel that we’ll have to study, the latter is the ramdisk from which the appliance runs and contains main components making up FortiGate (including binaries historically affected by several vulnerabilities).

Until recently, it was possible to decompress this archive and have direct access to its content. But on most recent versions (those released around end of 2023), it seems that rootfs.gz is now encrypted:

The next part of this blog entry will follow the way this ramdisk is verified and decrypted before being loaded.

Ramdisk integrity

Having a look at the kernel command line confirms that rootfs.gz acts as the ramdisk:

One of the key tasks of the bootloader is to load this ramdisk into memory at a specific location that kernel has knowledge of. Reading Linux kernel source code teaches us that populate_rootfs[7] makes use of global variables __initramfs_start and __initramfs_end to locate it and handle its loading.

populate_rootfs job is to unpack the GZIP-compressed CPIO image of ramdisk and mount rootfs.

FortiOS is Linux-based and uses a modified kernel 4.19.13 (as of version 7.4.3), thus we can disassemble flatkc (after conversion to ELF format using vmlinux-to-elf[8]) and look for the function mentioned above:

We can safely labelize the addresses 0xFFFFFFFF81838080 and 0xFFFFFFFF81838078 with their right symbols, respectively __initramfs_start and __initramfs_end. We then take a look at the cross-references to these variables and get two interesting functions: fgt_verify_initrd and fgt_verify_decrypt.

These functions names suggest some checks are firstly done on ramdisk before it is decrypted.

Interesting enough, fgt_verify_decrypt function seems recently added to FortiOS.

fgt_verify_initrd

This function is responsible for both ramdisk signature check and decryption:

RSA key decryption

First, we notice a call to fgt_verifier_open which is responsible for decrypting and loading a 2048-bits RSA public key that will be used further for signature check. Key decryption happens in a function called fgt_verifier_pub_key:

Symbols reveal that ChaCha20 algorithm (stream-cipher) seems to be used for decryption. It is initialized with two parameters (key and IV) derived from a 32-bytes master key stored at 0xFFFFFFFF817932E0:

key = sha256(masterkey[3:32] + masterkey[:3])
iv  = sha256(masterkey[1:32] + masterkey[0])

As defined in RFC 8439, ChaCha20 uses a 256-bits key and a 96-bits IV. But it also uses a block counter initialized to 0 that is incremented after each 512-bits block processing. The implementation in FortiGate differs from RFC because it uses part of computed IV (first 4 bytes) to initialize this block counter to something different from 0.

Once initialized, ChaCha20 is then used to decrypt an RSA public key (270 bytes).

We have implemented[9] this algorithm according to highlighted modification are now able to decrypt .

After having extracted master key (0x4CF7…) and encrypted blob (0x97CE…), we can then retrieve plain RSA public key used for signature verification:

We recognize last digits which represent the standard RSA exponent (65537). Further code analysis tells us that the RSA public key is encoded using BER[10].

Decoded RSA public key (e, n)

Signature check

Back in fgt_verify_initrd code, we analyze next implemented steps following key decryption:

• SHA256 of ramdisk (minus block of signature) is computed
• signature (last 256 bytes block) is loaded into a BigInt (bg_sig_enc)
• signature decryption: modular exponentiation is calculated using mpi_pown (bg_sig_enc^e % n)
• signature check: validate padding, hash type and compare computed SHA256 hash of ramdisk against the one embedded in signature.

In case all these tests pass, v1 = 0 (valid signature).

This is a PKCS#1 v1.5 RSA signature that follows the format:

PKCS#1 v1.5 signature structure

We can check it by ourself.

We retrieve last signature block of ramdisk:

We decrypt signature using few lines of Python:

Finally, we decode part of signature following padding:

Decoded hash

Let’s check ramdisk SHA256:

Signature is verified.

Ramdisk decryption

Last function call within fgt_verify_initrd goes to fgt_verify_decrypt:

This is very similar to fgt_verifier_pub_key. The light difference resides in the way master key is derived inside fgt_verifier_key_iv to generate ChaCha20 parameters:

key = sha256(masterkey[4:32] + masterkey[:4])
iv  = sha256(masterkey[5:32] + masterkey[:5])

We also provide a tool[11] that implements decryption of ramdisk (rootfs.gz):

We now have access to main OS binaries:

This directory contains binaries for FortiGate services (httpsd, sslvpnd…). We can notice that most of them are actually symbolic links to /bin/init which is a big monolithic binary. Most of critical vulnerabilities target the latter and other services located in this directory.

Auxiliary tool: getrootfskey.py

To help analysts decrypting rootfs.gz, we developed a script[12] using Miasm framework[13] that helps automatically retrieving of master key from firmware in a generic way without need of disassembling.

The symbol address of fgt_verifier_pub_key is first retrieved. Then Miasm allows us to do symbolic execution on this function, the goal is to detect any call to sha256_update to which the master key is passed as an argument. Parts of master key are given to this function to compute hashes, so we register all these addresses and keep the minimum value that gives us the base address of master key in memory.

Usage example:

This tool has been tested on latest versions of FortiGate firmwares and supports x64 and aarch64 architectures.

Additional integrity checks

So far, we delved into ramdisk integrity verification handled by the kernel that is the earliest check happening during the boot process of the appliance.

In rootfs.gz, we’ve examined the content of bin.tar.xz archive that contains a binary called init. This is the first userland process executed by the kernel (PID 1).

We’ll see that this process also implements several integrity checks. We have to study what are the new mechanisms implemented in order to bypass them.

.db[x] files

The directory listing of .vmdk revealed two interesting files: .db and .db.x.

.db looks like this:

This file contains hashes (no signatures, only plain SHA512 hashes) of firmware main components: rootfs.gz, flatkc and datafs.tar.gz.

.db.x file contains blob data with strings related to certificates.

Among strings in /bin/init binary, we can find out an occurrence to “/data/.db”, we then locate under IDA Pro what is the only function referencing it:

These files appear to be read and their content passed to function sub_25029F0. Debug log functions calls let suppose this function is called ftnt_code_signing_verify_sig.

This code clearly shows us that .db file is checked against a signature stored in .db.x file. If this signature is validated, then .db JSON file is parsed and hashes of the three OS components are checked against the ones stored.

If we look at cross-references on this function, this leads us (several levels up) until the main function of /bin/init:

We can reasonably assume that we’ll have to disable this check to avoid OS shutdown during the boot process.

Another function has indirect usage of ftnt_code_signing_verify_sig:

It seems that /bin/init initializes some kind of cron tasks (“forticron”). This task might be a recent addition, and we’ll see that it is also mandatory to disable this check to avoid OS premature shutdown.

.chk files

In /bin/init entrypoint, we also notice a call to sub_455770:

This function looks for a subroutine called “Firmware integrity”, once the entry found in the array it will call the linked callback (we dubbed it __cb_fw_integrity):

Without going further in this function analysis, we can guess that .chk files contains another kind of signature data for associated files. Again, rootfs.gz and kernel integrity is checked.

Customizing firmware

As analysts, our ultimate goal is to be able to deploy custom debugging tools on appliance.

Fortigate CLI interface provides a limited shell to the user with a scoped set of commands sufficient for appliance configuration.

Equipped with the knowledge aggregated so far, our goal is now to obtain full root access, get a Bash shell and deploy our tools (busybox, gdbserver…).

Methodology

Below steps are necessary:

• mount .vmdk disk
• retrieve ramdisk encryption key from kernel (flatkc) using getrootfskey.py and decrypt rootfs.gz
• decompress bin.tar.xz to have access to /bin/init binary
• patch /bin/init to disable the previously spotted integrity checks
• deploy our tools into /bin
• repack bin.tar.xz, replace it into rootfs.gz
• debug FortiGate kernel to disable ramdisk signature check.

We’ve already done the first three steps, so let directly see how to disable /bin/init security mechanisms.

Patching /bin/init

We previously spotted three locations where integrity checks were performed by init process. At assembly level, we spot the instructions we need to disable by patching them with NOP instructions.

Example of init patches

Deploying tools

To get a proper shell we’ll use a popular tool often met in embedded devices: busybox. This will offer us basic Unix commands to interact with the OS. We’ll install a bind shell whose purpose is to attach busybox to a socket so we are able to remotely connect to the FortiGate.

FortiGate forbids us to open arbitrary ports. The trick here is to kill legitimate (and non-necessary) services and re-use their port. For that, we configure the interface to enable ssh and telnet:

Now, we need a way to execute our bind shell. Several researchers[14][15] already gave a solution for that: it exists standalone binaries (i.e. not symbolic links to /bin/init) that it is possible to replace and to start from limited FortiGate CLI. One of them is /bin/smartctl that we can trigger using “diag hardware smartctl” command. We replace it with our bind shell and link busybox to /bin/sh:

We’ll also need a static gdbserver for debugging purpose, we can also put in in /bin/.

We can now rebuild the customized ramdisk as a whole (and encrypt it!):

We can now safely replace rootfs.gz within .vmdk and unmount it.

Debugging the kernel

Kernel is the first component that checks ramdisk integrity. We need to debug it to force this validation.

We take a closer look at the assembly of fgt_verify_initrd and locate the precise ASM instruction responsible for this test:

We’d like to force call to decryption routine while skipping machine_halt call. We’ll set a breakpoint at 0xFFFFFFFF81709877 and force r12d = 0.

We first enable GDB stub for our VM (.vmx file):

Then we set the breakpoint, start the VM and connect gdb (we have a few seconds window before init executes):

Finally, and despite warning messages, the boot process is completed and we got a privileged shell:

Conclusion

The security mechanisms outlined in this paper are designed to prevent arbitrary modifications to FortiGate firmware, thereby safeguarding against potential attackers willing to compromise the equipment.

However, while these redundant checks and encryption layers may be perceived as attempts to complicate the work of researchers and serve as obfuscation layers, they may not provide adequate protection against determined actors.

It is likely that Fortinet products, along with those of other vendors, will remain highly targeted. Therefore, Optistream advises maintaining updated customer appliances and applying security integrity check procedures based on IOCs provided by vendors.

Links

[1] https://github.com/optistream/fortigate-crypto
[2] https://www.fortiguard.com/psirt/FG-IR-23-097
[3] https://www.fortiguard.com/psirt/FG-IR-23-183
[4] https://www.fortiguard.com/psirt/FG-IR-24-015
[5] https://www.fortiguard.com/psirt/FG-IR-24-029
[6] https://support.fortinet.com/Download/VMImages.aspx
[7] https://elixir.bootlin.com/linux/v4.19.13/source/init/initramfs.c#L602
[8] https://github.com/marin-m/vmlinux-to-elf
[9] https://github.com/optistream/fortigate-crypto/blob/main/decrypt_rsapubkey.c
[10] https://en.wikipedia.org/wiki/X.690#BER_encoding
[11] https://github.com/optistream/fortigate-crypto/blob/main/decrypt_rootfs.c
[12] https://github.com/optistream/fortigate-crypto/blob/main/getrootfskey.py
[13] https://github.com/cea-sec/miasm
[14] https://ioo0s.art/2023/02/07/%E5%88%A9%E7%94%A8VMware%E8%8E%B7%E5%8F%96shell-%E8%BF%9B%E9%98%B6/
[15] https://forum.butian.net/share/2166