From 4dbb18bf5eba0d40c181c049d208cad7e72e3043 Mon Sep 17 00:00:00 2001 From: Norman Feske Date: Mon, 27 May 2019 13:49:56 +0200 Subject: [PATCH] Release notes for version 19.05 --- doc/release_notes-19-05.txt | 886 ++++++++++++++++++++++++++++++++++++ 1 file changed, 886 insertions(+) create mode 100644 doc/release_notes-19-05.txt diff --git a/doc/release_notes-19-05.txt b/doc/release_notes-19-05.txt new file mode 100644 index 000000000..dc968ad89 --- /dev/null +++ b/doc/release_notes-19-05.txt @@ -0,0 +1,886 @@ + + + =============================================== + Release notes for the Genode OS Framework 19.05 + =============================================== + + Genode Labs + + + +The Genode release 19.05 is primarily focused on platform support. +It adds compatibility with the 64-bit ARM architecture (AARCH64), +comes with improvements of the various kernels targeted by the framework, +and extends the list of supported hardware. The increased diversity of base +platforms calls for unifications to keep the hardware and kernel landscape +manageable. + +On that account, Genode uses one reference tool chain across all kernels +and CPU architectures. The current release upgrades this tool chain to +*GCC 8.3* with C++17 enabled by default +(Section [Tool chain based on GCC 8.3.0 and binutils 2.32]). + +To increase the velocity of Genode system scenarios across different boards +of a given CPU architecture, the release introduces the notion +of *board and kernel-agnostic build directories* presented in Section +[Unified build directories for ARM]. Once built for one particular +CPU architecture, the same binaries can be deployed at any supported board or +kernel without recompilation. This vastly accelerates the workflow when +targeting multiple boards and emulators at once. + +As another major unification effort, the current release introduces a new +*kernel-agnostic virtualization* interface. Up until now, virtualization +used to be inherently tied to a specific kernel. Thanks to the new interface, +however, one virtual machine monitor implementation can be combined with +kernels as different as NOVA, seL4, or Fiasco.OC. No recompilation needed. +As outlined in Section [Kernel-agnostic virtual-machine monitors], Genode +has now become able to run the Seoul VMM on all those kernels, while +VirtualBox is planned to follow. + +On our [https://genode.org/about/road-map - road map], we originally +planned several user-facing features related to Sculpt OS. However, in the +light of the major platform efforts, we decided to defer those topics instead +of rushing them. +That said, the release is not without new features. For example, our port +of *OpenJDK* has become able to host the Spring framework and the Tomcat web +server, there are welcome improvements of the *package-management tooling*, +and we added new options for user-level networking. + +Finally, version 19.05 is accompanied with the annual revision of the *Genode* +*Foundations book* (Section [New revision of the Genode Foundations book]), +which is now available as an online version in addition to the regular PDF +document. + + +Kernel-agnostic virtual-machine monitors +######################################## + +Since the introduction of Genode's +[https://genode.org/documentation/release-notes/17.02#Genode_Application_Binary_Interface - Application Binary Interface] +in the 17.02 release, +Genode components can be assembled once for a given hardware platform and +executed without further adjustments on all the supported kernels. However, at +that time, the supported virtual machine monitors - a port of VirtualBox 4 & 5, +Seoul, and our +[https://genode.org/documentation/articles/arm_virtualization - custom VMM] - +remained kernel specific. + +Of course, last remaining bastions tempt to be taken! So last year, we started +the venture to unify our virtualization interface across different kernels. +Starting point was the already existing Genode VM interface of our custom VMM +on ARM. We took it and extended the interface with caution to the x86 world. +Having an eye on the requirements of our already supported VMMs on NOVA(x86), +namely VirtualBox and Seoul, the VM interface got extended with missing +features like multiple vCPU support and specific VM handlers per vCPU. + +In parallel, we started to investigate the other x86 microkernels with regard +to hardware-assisted virtualization features, namely seL4 and Fiasco.OC. +Over several weeks, we iteratively extended the interface. On the one hand +we familiarized ourself with the kernel interfaces of seL4 & Fiasco.OC while +on the other hand considered known requirements of the NOVA microhypervisor. +Additionally, we kept our custom VMM for ARM still compatible with the new VM +interface. + +During this time, it became apparent that the control flow on a VM resume/pause +and a VM event(exit) are different between seL4/Fiasco.OC and NOVA/base-hw. +For seL4 and Fiasco.OC, a VM is resumed by making a blocking syscall on the +kernel. On a VM event, the blocking syscall would return. Logically, on both +kernels the VMM 'calls' into the VM. +On base-hw and NOVA, it is the other way around. Whenever a VM causes a VM +event, the kernels set up either an asynchronous notification (base-hw) or a +synchronous IPC call (NOVA) to the VMM. In both cases the VMM executes a prior +registered VM event handler as response. +Upon return of the VM event handler, the kernel resumes the VM. Logically, on +NOVA and base-hw the VM 'calls' into the VMM. The following two figures +contrast the different flows of control between a user-level virtual machine +monitor and the respective kernels. + +[image vm_seq_foc_sel4] + Control flow of handling virtualization events on Fiasco.OC and seL4 + +[image vm_seq_nova_hw] + Control flow of handling virtualization events on NOVA and the base-hw kernel + +Hiding this differences behind a common VM interface was the challenge we were +faced, accepted, and won. Finally, at one point in December we had all 3 +x86 kernels running with a test VMM - without re-compilation. The toy VMM +(vmm_x86.run) runs multiple vCPUs on multiple physical CPUs and tests several +VM events/exits. + +After this major breakthrough, we spent the days left before Christmas to +adjust the Seoul VMM to the new VM interface, freeing it from the ties to the +NOVA kernel. The choice to start with Seoul stems from the fact that it is - +compared to VirtualBox - much smaller and therefore easier to debug if things +go wrong in the beginning. After one week, the Seoul VMM became in principle +kernel independent and worked again on NOVA. After some more days, it started +to hobble on seL4 and Fiasco.OC as well. + +With the New Year, VirtualBox was the next target where all NOVA kernel +specific calls were replaced with the new Genode VM interface. Mid of January, +the work showed first results by having a prototype running simple VMs on NOVA +again. At this point, it became apparent that this venture is not anymore an +adventure. All the findings and technical details so far got condensed to a +[https://fosdem.org/2019/schedule/event/microkernel_virtualization - presentation] +given and recorded at the [https://fosdem.org/2019 - FOSDEM 2019] in Brussels +in February in the +[https://fosdem.org/2019/schedule/track/microkernels_and_component_based_os - Microkernel and Component based OS] +developer room. + +At this point, we started transforming our prototype for the 4 kernels into a +clean solution to be featured in Genode 19.05. Eventually, the kernel-agnostic +Seoul VMM runnable on seL4, Fiasco.OC, and NOVA entered Genode master. In the +Genodians article +[https://genodians.org/alex-ab/2019-05-09-seoul-vmm - Seoul VMM and the new VM interface], +we conserved the current state and a few performance measurements. + +Shortly before this release, the kernel-agnostic VirtualBox VMM version on +Genode/NOVA got ready. The kernel-agnostic version is in principle capable to +run Linux VMs and Windows 7/10 VMs on Genode/NOVA. Currently, this version +must still be considered as experimental and does not run on seL4 or +Fiasco.OC. + +Because of the experimental nature of the kernel-agnostic VirtualBox VMM +version, we decided to keep the kernel-specific version for NOVA for the +moment. This gives us time to test and improve the kernel-agnostic version. It +also allows us to compare both versions to each other. +If time and interest permits, we will consider bringing the virtualization +support on Genode/seL4 and Genode/Fiasco.OC on par with Genode/NOVA. + +When building VirtualBox with Genode 19.05, +you will find both the 'virtualbox5-nova' and the new 'virtualbox5' binaries +in the build directory. The former relies on NOVA's kernel interface whereas +the latter uses Genode's kernel-agnostic VM interface. Nightly tested run +scenarios with the new VM interface are named 'vbox5_vm*.run' and can be found +in the 'repos/ports/run' directory. + + +Broadened CPU architecture support and updated tool chain +######################################################### + +With the major update of Genode's tool chain and library infrastructure in +tandem, the framework gains a consistent architecture support across x86-32, +x86-64, ARM-32, RISC-V, and the newly added AARCH64. This includes the tool +chain (Section [Tool chain based on GCC 8.3.0 and binutils 2.32]), the base +framework, the dynamic linker, and the C runtime +(Section [Updated dynamic linker and C runtime]). + +Together with this update, we took the chance to wrap up our long-time move +away from board-specific build directories to one generic build directory +shared by multiple kernels and boards for a given CPU architecture +(Section [Unified build directories for ARM]). + + +Tool chain based on GCC 8.3.0 and binutils 2.32 +=============================================== + +Genode uses a tailored tool chain based on GCC and binutils that is used +across all supported kernels and architectures. Since the previous tool-chain +update in version +[https://genode.org/documentation/release-notes/17.05#Tool_chain - 17.05], +we relied on GCC 6.3. After two years, it was time for an update, motivated by +three major reasons. First, the C++17 standard is common-place now. We Genode +developers anticipate the improvements that come with it. Second, RISC-V and +AARCH64 are now supported by mainline GCC. Up till now, we had to maintain a +custom patch set for Genode's RISC-V support. AARCH64 was not supported yet. +Third, our increasing engagement with SPARK depends on recent improvements of +the Ada compiler that is part of GCC. + +With Genode 19.05, the tool chain is now based on binutils version 2.32, GCC +version 8.3.0, GDB version 8.2.1, gcov version 8.3.0, standard C++ library +version 8.3.0. + +The tool chain supports x86 (32 and 64 bit), ARM, AARCH64, and RISC-V. + +For C++ code, the C++17 standard is enabled by default. + +The update of the tool chain provided a perfect opportunity to replace the +former use of gnatmake with a much more natural integration of Ada in Genode's +build system, using a custom ali2dep dependency-extraction tool developed +by [https://github.com/Componolit/ali2dep - Componolit]. + +In contrast to the previous versions, we switched to a versioned installation +directory for the new tool chain. By default, it is now installed to +_/usr/local/genode/tool/19.05/_. This eases the use of different tool-chain +versions for different development branches. + +:Tool-chain installation: + + [https://genode.org/download/tool-chain] + +Caveats +------- + +The tool-chain update required a number of adaptations throughout the source +tree, and may affect Genode users too: + +* The silent fall-though within switch statements must now be replaced + by an explicit annotation of the form + ! [[fallthrough]] +* The 'register' keyword is no longer valid with C++17. Hence, it must + be removed from the code. +* Types marked as 'Noncopyable' can no longer have an implicit default + constructor. A default constructor must be provided manually. + + +Updated dynamic linker and C runtime +==================================== + +The tool-chain update is accompanied with a major update of the dynamic linker +and the C runtime to cover both the AARCH64 and RISC-V architectures in +addition to the traditional x86 and ARM architectures. + +FreeBSD 12 supports AARCH64 and RISC-V. Hence, by updating our C runtime to +this version, Genode's libc support extends to those architectures now. + +Until now, Genode's dynamic linker supported only the eager binding of symbols +at loading time on the *RISC-V* architecture. With the current version, we +lifted this limitation in favor of lazy binding as used on all other CPU +architectures. + + +Unified build directories for ARM +================================= + +In version +[https://genode.org/documentation/release-notes/17.02#Genode_Application_Binary_Interface - 17.02], +we introduced unified build directories for x86, which allow us to build and +run Genode scenarios on various kernels while using only one build directory. +This concept leverages Genode's cross-kernel binary compatibility to make +the switch from one kernel to another - like developing on base-linux and +deploying on base-nova - a seamless experience. + +On ARM, this concept was held back by a third dimension. The +system-integration step does not only depend on the CPU architecture and +the kernel but also on the used board. Our traditional approach was the +use of one build directory per board. Granted, within such a build directory, +one could easily switch between different kernels like Fiasco.OC and seL4. +But on ARM, we find an extreme proliferation of different board +configurations, which share the same CPU architecture but demand different +integration steps. This ensues large redundancies among different build +directories. Switching from one board to another - even when most binaries +happen to be exactly the same - requires an additional rebuilding effort. + +With version 19.05, we took the chance to generalize the unified build +directory concept to support multiple different boards per build directory, +greatly reducing the friction when switching kernels and boards for a given +CPU architecture (like ARMv7a). This change has the following implications: + +* Drivers no longer depend on the SPEC values as configured for a build + directory. + +* All *binaries* are now *named unambiguously*. For example, the USB drivers + for the Panda (OMAP) and Arndale (Exynos) boards were formerly called + 'usb_drv' but were different programs. They just never happened to + appear in the same build directory. In the new version, they are named + 'panda_usb_drv' and 'arndale_usb_drv' respectively and can thereby + peacefully co-exist within the same 'armv7a' build directory. + + Note that this binary renaming will likely affect existing run scripts. + +* Include paths no longer hide the board details, which makes the included + code much more easy to follow. + +* Run scripts need to pick the right binary, depending on the used board. + Since the board is no longer tied to a build directory, the selection + of the used board has become a build-time variable 'BOARD' following + the successful pattern of how we specify the targeted 'KERNEL'. + +To avoid the pollution of run scripts with difficult conditions, we wrap +the drivers needed for a particular board and use case into so-called +_drivers_ packages. Such a package can be instantiated within a generic +scenario using a nested init instance. The details about the drivers and +how they access the hardware remain nicely hidden inside this building block. + +Currently, there exist _drivers_ packages for two distinct use cases: + +:drivers_interactive pkgs: contain all drivers needed for simple + interactive scenarios, including graphical output and user input. + +:drivers_nic pkgs: contain the drivers needed for communication over the + network. + +Whenever a run script fits one of these use cases, it can rely on the +corresponding ready-to-use drivers packages via: + +! import_from_depot [depot_user]/src/[base_src] \ +! [depot_user]/pkg/[drivers_nic_pkg] \ +! ... + +With the drivers package incorporated, the drivers subsystem can be +instantiated as follows (note the absence of any board or kernel-specific +details): + +! +! +! +! +! +! +! +! +! +! +! + + +Using the 'BOARD' build variable +-------------------------------- + +The new 'BOARD' variable selects the board to use. It can be specified either +as a 'make' command-line argument (or environment variable), or defined in the +build-directory configuration (_etc/build.conf_). The following boards are +available: + +:arm_v6: rpi +:arm_v7a: arndale, imx53_qsb, imx53_qsb_tz, imx6q_sabrelite, imx7d_sabre, + nit6_solox, odroid_x2, odroid_xu, panda, pbxa9, usb_armory, + wand_quad, zynq_qemu +:arm_v8a: rpi3 +:x86_64: pc, linux, muen +:x86_32: pc, linux +:riscv: spike + +Please note, when running Genode on Linux or the Muen separation kernel - +although it is run on common x86 PC hardware - we treat both runtime +environments as separate "boards" because their device driver environments +are fundamentally different. + + +New revision of the Genode Foundations book +########################################### + +The "Genode Foundations" book received its annual update, which is actually +rather a refinement than a revision. The noteworthy additions and changes are: + +:
+:

+:

+: +: +: +:
+:

+ +* Component health monitoring +* Static code analysis +* Documentation of --depot-user and --depot-auto-update +* Minor adjustments in the under-the-hood chapter +* Changes of the build system +* Updated tool requirements +* Updated API reference + +:
+ +To examine the changes in detail, please refer to the book's +[https://github.com/nfeske/genode-manual/commits/master - revision history]. + + +New online version of the book +------------------------------ + +We are happy to announce that the Genode Foundations book is now available +as an online version in addition to the regular PDF version. + +:Browse the Genode Foundations book online: + + [https://genode.org/documentation/genode-foundations/19.05/index.html] + +Thanks a lot to Edgard Schmidt for creating the tooling for the HTML version +of the book! + + +Base framework and OS-level infrastructure +########################################## + +Modernized block-storage interfaces +=================================== + +With the current release, we revisited Genode's interfaces for accessing +block devices to ease the implementation of asynchronous I/O, to accommodate +zero-copy block drivers, and to support trim and sync operations. + + +Revised RPC interface +--------------------- + +The 'Block::Session' RPC interface remained untouched for a long time. +We have now rectified long-standing deficiencies. + +First, *sync requests* used to be handled as synchronous RPCs. This is bad +for components like part_block that multiplex one block device for multiple +clients. One long-taking sync request of one client could stall the I/O for +all other clients. The new version handles sync requests as asynchronous +block-request packets instead. + +Second, the new version allows a server to dictate the *alignment* of +block-request payload. This way, a driver becomes able to use the payload +buffer shared between client and server directly for DMA transfers while +respecting the device's buffer-alignment constraints. + +Third, we added support for *trim* as an asynchronous block operation. +However, as of now, this operation is ignored by all servers. + +Fourth, each block operation can now be accompanied with a client-defined +request tag independent from the other parameters of the operation. The tag +allows a block-session client to uniquely correlate acknowledgments with +outstanding requests. Until now, this was possible for read and write +operations by taking the value of the request's packet-stream offset. However, +sync and trim requests do not carry any packet-stream payload and thereby lack +meaningful and unique offset values. By introducing the notion of a tag, we +can support multiple outstanding requests of any type and don't need to +overload the meaning of the offset value. + + +New client-side API +------------------- + +We have now equipped the 'Block::Connection' with a framework API for the +implementation of robust block-session clients that perform block I/O in an +asynchronous fashion. + +An application-defined 'JOB' type, inherited from 'Connection::Job', +encapsulates the application's context information associated with a block +operation. + +The life cycle of the jobs is implemented by the 'Connection' and driven by +the application's invocation of 'Connection::update_jobs'. The 'update_jobs' +mechanism takes three hook functions as arguments, which implement the +applications-defined policy for producing and consuming data, and for the +completion of jobs. + +We plan to gradually move the existing block clients to the new API to benefit +from the latency-hiding effects of asynchronous I/O. The first updated client +is the _block_tester_ component located at _os/src/app/block_tester/_, which +received a number of new features like the choice of the batch size. Please +refer to the accompanied README for a detailed description of the +block-tester. + + +Unified types for time values +============================= + +[https://genode.org/documentation/release-notes/17.05#New_API_for_user-level_timing - Two years ago], +we introduced the so-called timeout framework to provide a general solution +for requirements unmet by the bare timer-session interface - most notably +timer-session multiplexing amongst multiple timeouts, and microseconds +accuracy. Up to this day, the timeout framework has proved itself many times +in both real-life appliances and artificial tests and has become the standard +front end for timing in Genode applications. + +With this release, we solved one of the few remaining limitations with the +framework by enabling timeouts of up to 2^64 microseconds (> 500000 years) +across all supported architectures. In order to achieve this, we replaced the +former machine-word-wide types used for plain time values by unsigned 64-bit +integers. We did this not only inside the timeout framework but also to almost +all code in the basic Genode repositories that uses the framework. + +By doing so, we also paved the way for a second step, in which we are planning +to replace plain time values as far as possible with the abstract 'Duration' +type. With this type in place, the user wouldn't have to worry anymore about +any plain-integer implications when calculating with time values. + + +Support for chained EBR partitions +================================== + +Having an active community around Sculpt leads to bugfixes in unexpected +places. By now we prefer to use a GPT rather than an MBR based partition table +and although we test 'part_block', the component that parses the tables, on +regular basis, the handling of chained EBR's was flawed. Community member +[https://genodians.org/valerius/index - Valery Sedletski] who relies on such a +setup encountered this flaw and provided a bug report, which enabled us to +quickly reproduce and fix the problem. + + +IP forwarding with port redirection +=================================== + +The NIC router can now be used to redirect to individual destination ports on +port-forwarding. To express the redirection, the new 'to_port' attribute can +be added to '' and '' rules in the NIC router +configuration. If the new attribute isn't added, the rules behave as usual and +forward with an unaltered destination port. + + +Libraries, languages, and applications +###################################### + +Ada/SPARK runtime and SPARK-based cryptography +============================================== + +The SPARK runtime has been updated to GCC 8.3. SPARK components do not require +'Genode::Env' or a terminal session anymore. Debug messages can still be +printed using 'GNAT.IO', which uses 'Genode::log' and 'Genode::error' +internally now. + +Threading support, which was never fully implemented, has been removed to +further simplify the runtime. This simplification allowed us to prove absence +of runtime errors for the secondary stack allocator and other parts of the +runtime. + +[https://github.com/Componolit/libsparkcrypto.git - Libsparkcrypto] is a +library of common cryptographic algorithms implemented in SPARK. It is +free-standing and has a very small footprint. The port of libsparkcrypto for +Genode has been added to the libports repository. Thanks to Alexander Senier +and Johannes Kliemann of [https://componolit.com - Componolit] for maintaining +the Ada/SPARK runtime and libsparkcrypto. + +To accommodate the use case of block encryption, we added the small wrapper +library 'aes_cbc_4k' around libsparkcrypto that provides a simple C++ +interface for the en/decryption of 4 KiB data blocks. It uses AES-CBC while +incorporating the block number and the private key as salt values. + + +Improved resilience of the sequence tool +======================================== + +We have a simple component that starts other components sequentially. It +will exit whenever one of those components has exited with an error. +However, this component is used by our [https://genodians.org - Genodians] +appliance where it controls the content-update mechanism. Since updating +involves fetching content via HTTP/S depending on external events, e.g., +the remote site is not reachable, the sequence tool might exit. In a long +running appliance, this is obviously not a useful action where no one is +in place to restart the sequence tool. Rather than increasing the overall +complexity of the appliance by introducing such a management component, we +added a _keep-going_ feature to the sequence tool that will instruct it +to carry on even if one of the started components has failed. + +Please look at _repos/os/src/app/sequence/README_ for instructions on +how to use the feature. + + +NIC-bus server for private LANs +=============================== + +The 'nic_bus' server was added to the world repository as an alternative +to the 'nic_router' and 'nic_bridge' components. The name may be a slight +misnomer, but this component acts neither as a hub, switch, or router. +The 'nic_bus' implements unicast and multicast Ethernet packet switching +between sessions, but drops any unicast packet not destined for a session +attached to the bus. This is in opposition to the behavior of a typical +Ethernet switch and is intending to create simple, software-defined +local-area-networks for native components as well as virtual machines. +In practice the component has been used for attaching VMs to the +[https://yggdrasil-network.github.io/ - Yggdrasil] overlay network via +a bus-local IPv6 prefix. + + +Distributed Genode +================== + +In +[https://genode.org/documentation/release-notes/16.08#Network-transparent_ROM_sessions_to_a_remote_Genode_system - 16.08], +we initially released the _remote_rom_ components that act as communication +proxies. A communication proxy transparently relays a particular service to +another Genode system. As the name suggests, the remote_rom relays ROM +sessions. + +Originally implemented as a proof of concept using bare IP packets, broadcast +MACs and static configuration of IP addresses, we added several improvements +to allow a more general use. First, we adopted the size-guard idea for packet +construction and processing from the NIC router. Furthermore, we adopted the +single-threaded implementation style that was already established in other NIC +components. Thanks to Edgard Schmidt for this contribution. Second, we +implemented ARP requests to eliminate broadcasting. Third, we moved from bare +IP packets to UDP/IP and implemented a go-back-N ARQ strategy in order to +reliably transmit larger ROM dataspaces. + +As the remote_rom proved valuable for distributing functionality across +multiple Genode devices, we also applied this concept to the LOG session in +order to transmit LOG output from a headless Genode device to a +[https://genode.org/download/sculpt - Sculpt] system for instance. The udp_log +component provides a LOG service and sends the LOG messages as UDP packets to +another machine. The log_udp reverses this process by receiving these UDP +packets and forwarding the messages to a LOG service. An example can be found +in the world repository at _run/udp_log.run_ and _run/log_udp.run_. + + +Seoul and VirtualBox virtual machine monitors +============================================= + +Besides the conversion of the Genode back end of Seoul to the new VM +interface, we added mouse-wheel support to the PS/2 model and changed the VMM +to request a single GUI/nitpicker session rather than distinct framebufer and +input sessions. + +Similar to the Seoul VMM, the VirtualBox VMM was adjusted to the new VM +interface and now uses the GUI/nitpicker session. The original kernel-specific +VirtualBox version tied to the NOVA kernel is still available. Both versions +can be used simultaneously. + + +Use of Nim decoupled from Genode build system +============================================= + +With this release, all integration with Nim tooling has been removed from the +Genode build system as a result of maturing support for additional languages +via Genode SDKs. Building Nim components independently of the Genode source +tree has the benefit of smaller upstream checkouts and faster build times, and +has yielded components such as the +[https://genodians.org/ehmry/2019-03-22-depot_9P - 9P server] used in some +Sculpt developer workflows. An example of an independent build system for Nim +components is +[https://genodians.org/ehmry/2019-04-27-nim_packaging - documented on the Genodians blog]. + + +OpenJDK improvements +==================== + +Within the 19.05 release cycle, we further improved Genode's OpenJDK support +by enabling additional networking infrastructure required by the +[https://spring.io - Spring Framework]. The improvements especially concern +support for SSL connections, which enabled us to successfully execute an embedded +[https://docs.spring.io/spring-boot/docs/current/reference/html/howto-embedded-web-servers.html - Tomcat] +server natively on Genode x86 and ARMv7 platforms using the same JAR archive. +This line of work continues our Java for embedded systems effort as described in +our [https://genodians.org/ssumpf/2019-02-27-java-19-02 - Boot2Java] article. + +Having these features in place, our Java efforts will continue in the direction +of Java Swing and the support of input devices in the future, with the ultimate +goal of seamless Java application integration into +[https://genode.org/download/sculpt - Sculpt OS]. + + +Device drivers +############## + +Improved Zynq board support +=========================== + +The initial support of the Xilinx Zynq-7000 SoC was added to our custom kernel +in 15.11. Since then, the support of this hardware has been incrementally +extended. The definitions of memory maps, frequencies, and RAM sizes for +different Zynq-based boards are found in the world repository. + +One of the major additions in this release is the initialization of the L2 +cache. In this context, we also added a simple cache benchmark at +_repos/os/run/cache.run_ that measures the access times for memory regions of +different size and thereby reveals the number of cache levels and their sizes. + +With the latest improvements of the network driver in 18.11, a zero-copy +approach was introduced as an effort to eliminate bottlenecks in the driver's +performance. However, this modification also introduced a kernel dependency of +the driver in order to flush packet-buffer memory from the cache before handing +it over to the DMA-controller. With this release, we moved back to using +uncached dataspaces in order to eliminate the cache flushes and the kernel +dependency. Interestingly, we could not recognize a significant impact on the +driver's performance, which confirms the presumption that flushing the cache +nullifies the gain from using cached dataspaces. + +In order to enable the continuous operation of the network driver, we extended +the driver-internal error handling that is necessary to recover the network +driver in certain situations. + +_Thanks to Johannes Schlatow for contributing and maintaining Genode's Zynq support!_ + + +Updated Intel network drivers +============================= + +As a result of recurring issues with modern Intel i219 laptop NICs, we +updated the driver sources for Intel chipsets to the latest upstream +iPXE version. This update also enables all NIC variants, which were +missing from our manually maintained PCI ID whitelist before. + + +New drivers-nic and drivers-interactive depot packages +====================================================== + +As already described in section [Unified build directories for ARM], +_drivers_nic_ packages nicely hide the driver configuration internals needed for +a specific board to communicate over the network. Until now there was only one +package available for x86 based PCs. Now, additional _drivers_nic_ packages +are available for: + +:boards: imx53_qsb imx6q_sabrelite linux muen pbxa9 rpi zynq + +Beside the formerly available _drivers_interactive_ packages for linux, pbxa9 +and pc, there are now additional ones for the following: + +:boards: imx53_qsb rpi muen + + +Platforms +######### + +For most kernel environments, the core component provides a ROM module named +'platform_info', which comprises information provided by either the kernel or +the bootloader. The information entails e.g., the TSC clock frequency and +framebuffer dimensions. Most of the information is of interest for special +device driver components only. + +Over the time, there was an increasing need to incorporate the information +about which kernel Genode runs on top of. Thereby, special test components, +like depot_autopilot could use the information to, e.g., skip certain tests +on kernels known to not support them. Moreover, there are rare corner-cases +where kernels behave differently, for instance, interrupts are enumerated +differently on certain ARM platforms. Rather than maintaining multiple driver +binaries with different names depending on specific kernels, the +'platform_info' ROM module can now be used to differentiate between kernels +when necessary. + + +Execution on bare hardware (base-hw) +==================================== + +This release comes with fundamental optimizations and corrections for +executing Genode on bare hardware when using the core component as the actual +kernel. + +In the past, we could observe some serious peculiarities regarding the timing +behavior on the hw kernel. After a careful review, we identified the obstacles +that led to time drifts on several platforms and to quite different runtime +execution. + +First and foremost, we limited the CPU-load wasted by the kernel, which +unnecessarily made new scheduling decisions quite often. When the hw kernel +was started as an experiment, there was less focus on performance, but more on +simplicity. Instead of caring about state changes that make a scheduling +decision necessary, the scheduler was asked for the next execution context +unconditionally, whenever the kernel was entered. Now, the scheduler gets +invoked only whenever an execution context gets blocked, or unblocked, or if +the kernel's timer fires due to a timeout. This dramatically influences the +CPU-load caused by the hw kernel in a positive way. + +The timing accuracy got increased by reworking most hardware timer drivers +used in the kernel to let the timer never stop counting. Moreover, we limit +the scope in between reading the clock and adjusting the next timeout to a +minimum. The whole internal time representation got widened to 64-bit. + +In some rare use cases, we could observe components that do I/O polling, and +thereby actively ask for pending signals, to starve. The reason was a gap in +the hw kernel's syscall API. Beside the ability to wait for signals, the +base-library offers the ability to check for pending signals without blocking +in the case of no available signals. The equivalent call in the kernel was +still missing, and is now present and integrated in the base-library of +base-hw. + +ARM architecture +---------------- + +With this release, we add the i.MX 7 Dual SABRE reference board to the rich +hardware zoo Genode runs directly on top of. This includes the use of the +virtualization extensions available on this platform. + +Apart from the new board support, several optimizations were added +specifically for the ARM architecture. Several unnecessary cache maintenance +operations were eliminated, which resided in the code base since the time when +the kernel used a separate address-space only. Moreover, the kernel-lock - +used when several execution contexts on different CPU-cores try to enter the +kernel - does not spin anymore. Instead, the CPU goes into a sleep-state to +save energy. As a side-effect, multi-core scenarios become usable when +executed in Qemu. + +X86 architecture +---------------- + +Since the newly used compiler version makes aggressive use of FPU instructions +including the core component, the kernel itself makes use of FPU registers and +state. Therefore, lazy FPU switching becomes a no go for base-hw. Although, we +incorporated eager FPU switching into the ARM-specific part of the hw kernel +already, the x86 version was still missing it. Now, the FPU context of a thread +gets saved and restored on every kernel entry and exit on x86 too. + + +Updated Muen separation kernel +============================== + +The Muen port has been updated to the latest development version, which comes +with many improvements under the hood. Most notably this version of Muen brings +support for Linux SMP subjects, GNAT Community 2018 toolchain support as well +as much improved build speed, which is most noticeable during autopilot runs. + +Additionally, the debug server buffer size in the Genode system policy has been +increased to avoid potential message loss in case of rapid successive logging. + +_Thanks to Adrian-Ken Rueegsegger of [https://codelabs.ch - Codelabs] for_ +_this welcome contribution!_ + + +NOVA microhypervisor +==================== + +The kernel got updated due to the tool-chain update to GNU G++ 8.3.0. +Additionally, several issues reported by Julian Stecklina regarding FPU and +page-table synchronization got addressed. The kernel memory allocation at boot +time got more flexible to address target machines with fragmented physical +memory. Additionally, the vTLB implementation is no longer used on AMD +machines whenever nested paging is available. + + +seL4 microkernel +================ + +With this release, we extend the variety of hardware to run Genode on top of +the seL4 kernel with NXP's i.MX 7 Dual SABRE reference board. To do so, we had +to update the seL4 tools used to craft a bootable ELF image to a state that is +consistent with the currently supported seL4 kernel version 9.0.1. + +As a side-effect of this development work, the General Purpose Timer (GPT) used +in the i.MX series can now be used as a timer service component. + + +Fiasco.OC microkernel +===================== + +As with base-hw and seL4, we add the i.MX 7 Dual SABRE reference board to the +list of working hardware for Genode running on top of the Fiasco.OC +microkernel. Moreover, with Fiasco.OC it is now possible to take the first +steps using Genode on the ARM 64-bit architecture. Therefore, we add Raspberry +Pi 3 as a candidate board to be used with Genode/Fiasco.OC. Currently, only +basic tests without peripheral dependencies are supported. + + +Tooling and build system +######################## + +Improved handling of missing ports +================================== + +The depot tools _tool/depot/create_ and _tool/depot/extract_ now detect and +report all missing third-party sources - called ports - for a given set of +archives at once. Additionally, the user can tell the tools to download and +prepare such missing ports automatically by setting the argument +'PREPARE_PORTS=1'. Please be aware that doing so may cause downloads and +file operations in your _contrib/_ directory without further interaction. +These features make building archives with dependencies to many ports more +enjoyable. If you merely need a list of ports that are missing for your +archives, you can use the new tool _tool/depot/missing_ports_. + +For more details you may read the +[https://genodians.org/m-stein/2019-05-21-depot-missing-ports - article on genodians.org]. + + +Automated depot management +========================== + +When using the 'import_from_depot' mechanism of the run tool, one frequently +encounters a situation where the depot lacks a particular archive. Whenever +the run tool detects such a situation, it prompts the user to manually curate +the depot content via the _tool/depot/create_ tool. The need for such manual +steps negatively interferes with the development workflow. The right manual +steps are sometimes not straight-forward to find, in particular after +switching between Git branches. + +To relieve the developer from this uncreative manual labor, we extended the +run tool with the option '--depot-auto-update' for managing the depot +automatically according to the needs of the executed run script. To enable +this option, use the following line in the build configuration: + +! RUN_OPT += --depot-auto-update + +If enabled, the run tool automatically invokes the right depot-management +commands to populate the depot with the required archives, and to ensure the +consistency of the depot content with the current version of the source tree. +The feature comes at the price of a delay when executing the run script +because the consistency check involves the extraction of all used source +archives from the source tree. In regular run scripts, this delay is barely +noticeable. Only when working with a run script of a large system, it may be +better to leave the depot auto update disabled. + +Please note that the use of the automated depot update may result in version +updates of the corresponding depot recipes in the source tree (recipe hash +files). It is a good practice to review and commit those hash files once the +local changes in the source tree have reached a good shape. +