7 Questions and Answers about Firmware and Firmware Security

What is a firmware image?

Although the name would suggest that it is somewhere in the middle ground between hardware and software, firmware can be classified as a type of software used in IoT devices, and embedded systems. Its main responsibilities are controlling the device's hardware and providing its functionality.

A firmware image is a file that contains the firmware code for a specific device or system. It will usually include all the necessary firmware code for a device, along with any configuration settings, data structures, and other information required for the device to function properly. Firmware images are typically distributed by the manufacturer of the device, and they may get updated from time to time to fix bugs, add new features, or improve performance.

Firmware images are commonly used in devices such as routers, printers, cameras, and other electronic devices that require specialized software to control their hardware. However, firmware is not limited to consumer devices only; it is used in industrial control systems, aviation, the automotive industry, medical systems, military technology, and more.

In the case of consumer-facing applications, they may be downloaded from the manufacturer's website, or they may be installed automatically as part of a firmware update process over whatever network infrastructure is used in that particular use case.

Firmware images are different from traditional application code because they contain all the necessary systems software to make hardware work. They may or may not include an operating system to fulfill this task. This is the very thing that makes them super important in terms of IoT security: a piece of firmware is driving the whole behavior of an IoT device, it is essentially its “soul”. Therefore a single failure hidden among all the complexity could be detrimental in terms of software integrity, leading to potentially huge security implications.

Mainstream desktop and mobile operating systems provide abstraction layers to hide the low-level details from application developers, while in IoT operating systems there are usually many constraints that do not make this (application vs. system software decoupling) possible.

On top, these platforms have to be flexible to run on hardware configurations corresponding to an increasingly heterogenous IoT technology stack. Each hardware component vendor has its own secure coding culture, tooling, and development process, leading to an ecosystem without central control over crucial parts of the resulting products.

So-called bare-metal firmware also exists, although it is becoming less common, used only in the lowest unit-cost devices. This means that there isn’t an external programming interface, but the code is directly using hardware resources. Creating such an image will usually entail compiling pieces of C code and linking them in a strictly platform-dependent way, allowing the developer to directly use resources via memory-mapped IO, interrupts, and so on. This, obviously involves a lot of platform-specific datasheet reading and the low-level nature of these pieces of firmware requires additional care from the developer.

Firmware developers primarily deal with C and C++ source code, or even some assembly from time to time. These engineers need to be comfortable with understanding the details of the binaries their compiler produces. These particularities are often very important when debugging some malfunction with the device.

Linux kernel and FreeRTOS are popular choices for non-bare metal systems. In this case, the work includes dealing with and writing platform-specific drivers, and also using all the “comfortable” facilities, abstraction layers, and driver frameworks that these OS’s support to develop maintainable code adhering to best practices. On rich operating systems and application CPUs, firmware developers can enjoy better isolation from the low-level system via software execution privilege levels, operating-system-level virtualization, and a range of other measures.

In any case, writing firmware code and maintaining firmware images is a daunting task.

What are the most common firmware image formats?

Today, there are different types of firmware image formats being used, which shows how diverse the firmware technology ecosystem is. Still, let’s try to summarize what is out there in the wild.

Bare metal

For bare-metal firmware and small RTOS systems, some of the most common ones include:

• Binary file: the end and the beginning. A binary file, of course, is a simple file format that consists of a sequence of bytes with no universal overall structure. Binary files are the simplest and most basic type of firmware image format.
• Intel HEX: a file format that is commonly used for transferring firmware images to microcontrollers and other embedded systems. The file format consists of ASCII text records that encode the data and address information for each memory location in the firmware image.
• Motorola S-record: The Motorola S-record format is similar to the Intel HEX format, but it was originally developed by Motorola for use with their microcontrollers. S-records encode the address and data information in a simple ASCII format that is easy to read and parse.

Rich OS

For embedded systems with a rich operating system, the firmware image generally embeds a variety of binaries. These binaries correspond to the operating system kernel, platform-specific configuration partitions and data, and often complete filesystems. The filesystems contain a wide range of assets including ELF binaries, shell scripts, text files, and image assets for vendor branding, among others. A common way to describe firmware images for complex systems is to compare them to matryoshka dolls. When you open and decompress an outer layer, you'll find another layer that looks quite similar, and it can also be opened up to reveal even more parts, and so on. For examples of possible file formats in this category, here are the ones that we support with BugProve:

Vendors are very creative with their own particular headers and container layouts, but it is still common that most of the inner layers and binary formats are going to involve some kind of compression to minimize the firmware’s size. This is necessary because the flash storage chip has a unit cost that becomes expensive for large batches of products.

These are just a few examples of the many different firmware image formats in use today. The choice of format depends on the specific requirements of the device or system, as well as the preferences of the manufacturer.

Who writes the firmware?

Firmware is typically written by specialized firmware engineers who have expertise in low-level programming, embedded systems, and hardware design. These engineers work closely with the hardware designers and system architects to develop firmware that can control the device's hardware and provide the necessary functionality.

As we mentioned already, firmware development requires a deep understanding of the hardware architecture of the device, as well as the specific requirements of the application that the device will be used for. Firmware engineers use programming languages such as C and assembly language to write code that can interact directly with the device's hardware, accessing registers, memory-mapped I/O, and other low-level features.

Firmware development in a rich environment

Some higher-complexity devices actually require more or less traditional system software skills. High-level C++ is a popular tool for firmware engineers, not only for GUI-related tasks but also for its performance and high-level abstractions that enable large teams to collaborate on projects.

Some firmware engineers can enjoy not only the comforts of a rich operating system (e.g Linux) but also an application-specific middleware, written by other companies in the supply chain. For instance, a set-top-box provided by your cable TV operator (if you still have one since Netflix) would be an example of such a design. In such a scenario, a firmware developer does not have to reimplement things like the TV guide menu that your remote brings up, or scheduled recordings. Instead, they can focus on features that the vendor or the operator considers as a value-add, as well as customize aspects provided by the middleware.

Complex systems usually have many third-party components in their final firmware image, including open-source software, middleware sub-contractor IP, and hardware drivers. This results in a complex supply chain. SoC vendors (System-on-a-Chip – the „brain” of the device consisting of CPUs and essential IP cores that deliver the primary application features on the hardware level) are crucial in this as they are designing and shipping the application-specific SDK and development kits. Although they usually don’t take too much responsibility for their sample code, firmware developers working with their code are inclined to reuse whatever they have access to when facing strict deadlines.

In summary

The complexity in the firmware supply chain, shortage of security manpower, unclear responsibilities, cost-minimizing efforts, and go-to-market time pressure have all contributed to the long-standing IoT security problem. Thus, it's time to discuss firmware security.

What is firmware security?

There isn’t a universally accepted definition of firmware security, but it generally refers to the protection of firmware from exploitation, modification, and unauthorized access.

As we discussed previously, firmware is a critical component of many electronic devices, and if it is compromised, it can lead to serious security vulnerabilities that can be difficult to detect and mitigate.

Firmware security involves several different aspects. Design-time and implementation-time concerns are separate categories. Post-deployment firmware security is ideally part of DevSecOps, an approach to culture, automation, and platform design that integrates security as a shared responsibility throughout the entire lifecycle.

Secure firmware development

Firmware developers must follow secure coding practices and design the firmware with security in mind from the outset. This includes using encryption, authentication, and access control mechanisms to protect the firmware from unauthorized access or tampering. It is also crucial to be mindful of all the memory safety vulnerabilities that C/C++ has a bad reputation for in terms of security.

In our experience, these along with OS command injections are the most important practical vulnerability classes in most IoT system implementations. It is very hard to fix devices post-market which weren’t designed and implemented with firm control over secure coding practices.

Producing a truly secure device requires more than just secure coding. To achieve this, a minimum set of security features must be enabled, and ideally, all security measures offered by the hardware platform and software toolchain should be utilized. The specific list of security concerns and features is device and application-specific, as ususal in IoT. However, there are a few common security features that should be considered.

Firmware operational security concerns

Secure boot

Secure boot is a process that ensures that only trusted firmware and software is executed on a device. Secure boot involves verifying the integrity of the firmware before it is executed and preventing the device from booting if the firmware has been tampered with. As with everything in hardware and software design, implementation details can contain various vulnerabilities even if the design was working in principle. In practice, bootrom and bootloader exploits are pretty common and hard to fix.

Firmware updates

Firmware updates are critical for addressing security vulnerabilities and fixing bugs. However, the update process itself can also introduce security risks if it is not properly secured. The integrity of firmware updates must be protected to prevent attackers from injecting malicious code into the device.

Device attestation is an increasingly popular solution, supposed to meet the operational demands of Internet of Things (IoT) and cyber‐physical system (CPS) devices. It can be used to verify cryptographically signed evidence about the state of the device, providing information on whether it is securely booted, or if there is any evidence of tampering. This enables the relying party, because it is cryptographically signed, to verify that it is a particular device from a particular manufacturer and that it has not been tampered with before it is connected to the network. This doesn’t necessarily protect against issues when the device’s software integrity is compromised in runtime, but it can help with certain tampering or outright spoofing vectors.

More on implementation

Physical security

Firmware can be vulnerable to physical attacks, such as side-channel attacks that exploit vulnerabilities in the hardware itself. Physical security measures, such as tamper-proof seals and secure storage, can help protect firmware from these types of attacks. These mechanisms might sound like black magic, completely alien to the usual concerns of manufacturers, but in highly sensitive domains involving piracy and fraud (e.g financial and digital entertainment sector), these measures are much more common than one would think, even for fairly low-cost devices.

Hardening measures

Firmware images can have operating systems embedded in them, and a lot of binary code is present in each case for sure. These should be hardened via the options offered by the compiler toolchain and the system. The principle of least privilege should help with selecting, e.g. the minimum necessary Linux kernel modules to minimize the attack surface. On top, almost all compiler toolchains will support a range of flags that can be used to increase resilience against exploitation attempts. If you are interested in how these checks could work in a tool, you can have a look at our documentation

Cryptography

It is crucial to properly implement cryptography in IoT devices to ensure the confidentiality, integrity, and authenticity of the data being transmitted over the internet. Without proper cryptography, sensitive information such as personal data, financial information, and even critical infrastructure can be compromised. Engineers should follow industry-standard cryptographic protocols, use strong and unique keys for each device, and regularly update and patch the cryptographic libraries in use, such as OpenSSL. It is also recommended to utilize trusted execution environments and other platform security features to secure cryptographic assets when this is possible.

Can you reverse engineer an encrypted firmware?

In general no, no one can really decrypt encrypted firmware without additional information. On the other hand, one can reverse plaintext firmware, and there might be ways to extract such an image from a device, even if firmware is encrypted at rest.

Many Manufacturers consider the firmware their IP, even though it will usually contain IP from various vendors of the complex supply chain. As a result, some consider firmware as a highly confidential asset, while others are completely fine with uploading plaintext firmware images to their support site.

If implemented correctly, there are valid threat models where it may not be necessary to encrypt the entire device firmware. Instead, security-sensitive components, such as cryptographic keys and sensitive code, can be protected separately, perhaps by loading them into a trusted execution environment (TEE) from a separate partition using robust hardware measures. Meanwhile, "active" assets, like a baby monitor's camera feed or sensitive authentication information in transit, can be safeguarded using secure software, hardware-protected keys, cryptographic protocols, and a strong public key infrastructure (PKI) in an ideal setup.

The software should primarily be secure, or in security jargon, it should be impossible to compromise the local software integrity of the device. This, however, does not mean that plaintext code that is separately updatable could not do its job.

The usual objection to the above argument is that software can be reverse-engineered. This is certainly true, in theory, even the most complicated things that can make a CPU tick could be analyzed and understood to the extent that a semantically equivalent piece of code can be reconstructed in source form.

This is also how attackers can find vulnerabilities even in complex binaries. Accordingly, neither IoT pentest labs nor BugProve require source code for firmware security evaluations, since we have everything present in the firmware that is needed to test for practical vulnerabilities.

Due to reverse engineering concerns, therefore, it is often a requirement to store all pieces of code encrypted at rest to get ahead of these types of attacks.

We have analyzed a range of possibilities of how an attacker could bypass firmware encryption and how vendors can protect against these here.

What are the most dangerous vulnerabilities in firmware?

Although difficult to have an objective toplist, confirmed buffer overflows, injections are considered critical since they can easily lead to remote code execution attacks. Let’s see these in more detail.

Firmware vulnerabilities can be particularly dangerous because they can provide attackers with persistent access and control over a device. Some of the most dangerous vulnerabilities in firmware include those that allow so-called Remote Code Execution (RCE) attacks. Zero-day vulnerabilities that are also RCEs are especially dangerous, since by definition, these are issues that were discovered before security researchers and developers became aware of them. Zero-days pose a higher risk to users since cybercriminal race to exploit these vulnerabilities to cash in on their schemes and these affected IoT devices are exposed until a patch is issued by the vendor (Considering that some IoT manufacturers completely lack a product security incident response management process, this is bad news).

What is the root cause of these vulnerabilities, then?

Buffer overflows and friends

One common class of IoT RCEs involves some kind of networking service, such as an http server or some kind of proprietary message-parsing protocol or maybe vendor-specific changes in well-known open-source networking daemon, such as udhcpd.

It is not hard in C or C++ to run into a memory corruption issue while parsing networking data that is controlled by an attacker. Paired with the lack of hardening measures, this results in situations where issues such as stack-based buffer overflows end up being remotely exploitable on a large portion of IoT devices. This class of vulnerability was already well understood in 1996 (yours truly was just born 4 years earlier) and should have disappeared from the face of the earth by now.

Obviously, memory safety issues are still an open problem in very modern and complex applications written in modern C++, so it is unreasonable to expect this will be missing entirely, no matter how well-versed the workforce is in secure coding. However, in IoT systems, there is usually no need for attackers to find multiple vulnerabilities and to implement complex exploit chains starting with some relatively moderate corruption primitive. Unfortunately, the reality is that other types of bugs, such as logic flaws, authentication bypasses or incorrectly designed NAT traversal hacks often worsen the impact of vulnerabilities that would have only allowed for local attacks.

Injections

Another significant class of vulnerabilities is injections, which can be characterized as weaknesses where user input is being interpreted in a context where it shouldn’t be.

Linux-based IoT devices have many service binaries that carry out operations by executing commands in the system. Unfortunately, standard library calls such as system() do not enforce any kind of sanitization of input data, which often results in situations where the engineer does not perform this before using untrusted user data in their code. This allows for super simple attack vectors, where the attacker just has to find a field or parameter that she suspects will influence such a system() invocation and insert some special characters to make the shell interpreter treat these pieces of data as commands.

The situation is often made worse by the complete lack of privilege isolation in some vulnerable systems, where low code quality services are started with root privileges, never dropping their rights. An RCE of this type is completely detrimental to the device, while the attacker has a fairly easy job when it comes to vulnerability research and exploitation.

Threat intel services such as Greynoise regularly discover opportunistic exploitation of such IoT vulnerabilities that are essentially hitting the whole internet (well, at least the full IPv4 range).

If you are interested in these in more detail, we covered them in a separate article focusing on the most dangerous vulnerabilities.

Design problems

The above classes are usually considered as implementation vulnerabilities, however, some IoT systems have actually a complete disregard for any kind of access control when it comes to some of their sensitive networking services, or even the full video and audio stream of IP security cameras. Other times, there are hardcoded credentials in the firmware, for example: vendorname:12345 or operator:pass. These act as backdoors into these systems, and once someone finds such a combination, they could take control over all affected devices that expose their interfaces on the public internet.

What are firmware security best practices?

Many practices will help firmware developers to design and implement a secure IoT system.

It cannot be stressed enough that firmware developers should follow secure coding practices, such as input validation and carefully managing memory and other resources to prevent common vulnerabilities, such as buffer overflows and injection attacks.

Secure coding does not stop at looking for common classes of errors, the software development cycle should organically incorporate security in it, including thorough code reviews, and even more rigorous patch reviews. (It is often the case that a fix to a certain problem introduces two additional vulnerabilities that are now essentially zero-days and will be found with ease with differential analysis by any serious attacker).

Static Application Security Testing (SAST) and Software Composition Analysis (SCA) tools are widely used in application security, and we believe there are some great options available for C/C++ code that should also work for firmware code with certain limitations. However, firmware analysis solutions can offer a much more holistic overview of the firmware’s security posture, which manufacturers may find valuable. In this article, we covered in detail the benefits and limitations of SAST tools.

It is also good practice to enable all the hardening measures that are offered by the hardware platform and the compiler toolchain. These include things like NX, PIE, stack protection, SafeStack, CFI, and the list goes on. ASLR should be enabled on most systems, even though it is known to be less effective on 32-bit platforms.

Modern platforms such as recent ARM designs offer hardware measures to combat exploitability, such as UXN, PXN, MTE, BTI that developers should learn about and utilize when working on such a system.

The measures we listed above in "Firmware security" should all be used. I'm not going to cover everything, but here are some essential security practices to keep in mind.

• Secure development practices: Firmware developers should follow secure coding practices, such as input validation and error checking, to prevent common vulnerabilities, such as buffer overflows and injection attacks.
• Secure boot: Firmware should be protected by a secure boot process that ensures that only trusted firmware and software are executed on the device. The secure boot process should include measures such as cryptographic verification of firmware and software images and protection against unauthorized firmware modification.
• Firmware updates: Firmware updates should be authenticated to prevent unauthorized modification, and updates should only be obtained from trusted sources. Additionally, firmware updates should be tested before deployment to ensure that they do not introduce new vulnerabilities.
• Access control: Firmware should implement access control mechanisms to limit the ability of unauthorized users to access or modify the firmware or impact software integrity in a security-sensitive way.
• Monitoring and auditing: Devices should be monitored for firmware modifications, and firmware changes should be logged and audited to detect any unauthorized modifications.
• Physical security: Devices should be protected against physical tampering, such as by using tamper-evident seals, secure storage, and anti-tamper measures.
• Vulnerability assessments: Regular vulnerability assessments should be performed to detect and address any potential vulnerabilities in the firmware.
• Education and training: All stakeholders, including developers, administrators, and end-users, should receive education and training on firmware security best practices, including how to identify potential threats and how to mitigate them. If you are looking for materials, we have collected books, podcasts, and courses here.

By following these best practices, organizations can help to minimize the risk of firmware-based attacks and ensure that their devices are secure and reliable.

Conclusion

In conclusion, firmware security is a critical aspect of device security that must be taken seriously by device manufacturers and developers. The protection of firmware from exploitation, modification, and unauthorized access is essential to prevent serious security vulnerabilities that can have devastating cross-country effects. (This is the reason we see regulatory efforts like the EU Cyber Resilience Act and US Cyber Trust Mark coming into effect.)

Designing and implementing a secure IoT system requires a comprehensive approach that includes numerous steps we listed above, such as secure coding practices, hardware and software hardening, secure boot and firmware updates, access control mechanisms, etc. Firmware developers must prioritize security at every stage of the software development cycle, from code reviews to patch reviews, to ensure that any vulnerabilities are detected and addressed early on. By implementing these best practices, organizations can significantly reduce the risk of firmware-based attacks and protect their devices' security and reliability.