Systemd as central control for the Linux system

Cadaverhan, sxc.hu

Cadaverhan, sxc.hu

Strong Glue

Some Linux followers see systemd as the best thing since sliced bread – for others, it's the work of the devil. However, it also has the stuff to clear out old trenches and form a unified base for Linux.

Systemd is the result of the work of many developers, but, if there's a public face to be given to the software, it's certainly that of Lennart Poettering.

The valiant developer has already checked off a number of important projects on his to-do list, including Avahi and the PulseAudio sound system. With systemd, he's been working since 2010 on getting central services on a modern, uniform basis – emanating from a new approach for the init system.

Poettering doesn't shy away from unpopular decisions, which has earned him the reputation of using dictatorial methods. The decision to develop systemd only for Linux brought criticism, scorn, and insults. For him, meanwhile, the decision has a clearly understandable foundation.

Along with his work on systemd, Poettering is advocating for a unified Linux desktop. His newest project, which he's pursuing together with kernel guru Greg Kroah-Hartman, deals with integrating the Dbus message system as Kdbus in the kernel. His announcement at Google Plus [1] that systemd, together with Kdbus and all userspace tools in Fedora, starts without a problem and, thus, represents a new milestone, resulted in yet another hailstorm of harsh criticism.

Such opposition, however, has never diverted the Red Hat developer from his course (see the "Lennart Poettering Interview" box). He likes to speak his mind (Figure 1) as a free-thinker with enough backbone and support to see his vision through and perhaps remove a few old Linux relics.

Figure 1: Lennart Poettering's t-shirt shows his opinion on the sayings of Mark Shuttleworth.

To the Roots

If you're dealing with an init system, you very quickly recognize why working on this key component can cause so much turmoil. The term comes from "initiating" and encompasses, on Unix-like operating systems, the system startup and orderly shutdown.

Init is the first process the kernel generates, and it starts all other child processes. It's therefore assigned process ID 1 (PID 1). Among init's tasks are essentially starting all necessary services, mounting the filesystem, and setting up the network.

The SysVinit [2] init system used on Linux and other Unix derivatives since the 1990s was developed in 1988, when AT&T needed a start mechanism for its System V operating system in version R4.

In 2004, Solaris replaced SysVinit with its successor, Service Management Facility (SMF) [3], followed for the Mac OS X in 2005 by launchd [4]. Canonical has used Upstart [5], developed in-house based on SysVinit, since 2007. With Gentoo's FreeBSD offshoot, OpenRC [6] has been an alternative since 2007.

Born of Necessity

Systemd essentially comes from Red Hat developers Lennart Poettering and Kay Sievers. Before the two set upon the task of remaking the init system under Linux in 2007, they took a close look at launchd and SMF, because these init systems provided parallel processing startup capabilities that the static SysVinit didn't have.

Because Upstart also had this ability, the original idea of the developers was to enhance Ubuntu's init system. Legal reasons that had to do with Canonical's Contributor License Agreement (CLA) [7] prohibited this – in that neither Poettering nor Sievers was willing to sign the agreement.

Thus, systemd was born of necessity. Two fundamental decisions of the time still raise hackles: Systemd was deliberately and without compromise designed for Linux only. Systems with other kernels such as BSD and Hurd simply stood on the sidelines; a porting to BSD was possible, but expensive.

The other decision grated against the old Unix principle of "one tool for one job" in that it set some of the jobs pretty close together. This may loosen up slightly in the future so that functions are available only individually, as Ubuntu currently does with Upstart in conjunction with logind.

More than init

Systemd is short for system daemon, that is, a background process that provides services for the system. The init process was at the beginning of the development and still gets most of the attention. Other components such as logind and the journal, however, are up there as well.

Systemd's init process is based on a slim PID-1 service that does only the bare minimum. Thanks to parallel processing, the computer's startup and shutdown runs much faster. It's based on a technology that Apple developed for launchd: socket activation.

Socket activation allows all services to start concurrently. The process also got simpler because developers and administrators no longer need to be aware of which service starts at which point during bootup with which dependencies. The trick is that systemd itself creates the communication sockets with the services to be started and buffers the data until the service is running and accepts it. The SysVinit concept rather forces dependent services to wait for each other, because the init system doesn't store any data itself.

Because all services start concurrently, systemd contributes greatly to simplifying the structure of the service files. These replace the SysVinit shell scripts that often went wild. Figures 2 and 3 show the differences based on the Kexec [8] example.

Figure 2: SysVinit: The Kexec startup script elicits constructs at various places where the purpose isn't readily apparent.
Figure 3: Systemd: The Kexec service file needs only a few lines whose function is clearer based on the variable names.

Systemd organizes all tasks pending at startup – such as readying the hardware, mounting the filesystem and starting the services – in units. The service files of the various units have their own filename extensions, which give a clue as to their functions.

Files with the .mount extension take care of mounting and unmounting the filesystems; those with the .service extension handle the background services. These units are stored uniformly in /lib/systemd/system . For a modification, copy the files in question manually to /etc/systemd/system/ and edit them as needed; the system automatically uses them from then on.

An important decision in designing systemd that contributed to its being used only in Linux was the integration of cgroups [9]. These control groups allow allocating and limiting kernel resources, such as CPU time, data throughput, and process group memory usage.

Systemd therefore provided new unit types as of version 205, including scope units and slice units. The former collect the system services or applications and all their dependencies into cgroups and manage them. The slice units provide the units with the required resources.

In the long term, the developers want to hand over the use and control of the cgroups completely to the daemon. In this way, systemd would become a new intermediate layer between the kernel and applications. Poettering himself sees it as a "glue." In any case, the structure thereby acquires some complexity (Figure 4).

Figure 4: The different layers of the systemd infrastructure make it difficult to consider it a simple service.

Journal

Unlike SysVinit, systemd logs the boot messages right from the beginning. Thus, screenshots of the boot process captured by smartphone for support forums become a thing of the past. Systemd instead creates a socket [10] for it at runtime, on which a minimal logging service runs and records the messages from the early boot phase. It later passes the socket to the regular protocol services and the message is journaled.

The journal replaces the long-used Syslog. In a first demonstration, it received much criticism because the journal deviated from the ongoing concept of storing text files in a flat hierarchy. Instead the files are stored as binary.

Apart from that, the journal has some hard-to-ignore advantages. Besides the aforementioned boot process logging, it allows for highly targeted queries. Thus, the command on the first line below shows the messages since the last system startup; whereas the second line shows what the web server put out in messages since midnight. In a blog entry, Poettering shows other more detailed queries [11].

$ journalctl -b
$ journalctl -u httpd --since=00:00 --until=9:30

As the name logind [12] suggests, the daemon takes care of user and session management and replaces the no longer developed ConsoleKit [13]. Thus, systemd begins to enter userspace, which again was not met without criticism.

In the end, Poettering wants to go still further by using the same mechanism for the processes starting Gnome or KDE and its applications. Apple's Mac OS X already does this. The first approaches can already be seen in Gnome 3, which has recently been pulling in parts of systemd by integrating logind.

Neverending Story

Distributions such as Fedora and openSUSE, but also Debian derivatives such as Siduction, Semplice, and soon Tanglu, have already switched over to systemd (Figure 5).

Figure 5: Front ends already exist to exhibit systemd activities.

The dependencies to parts of systemd were the catalyst for hard decisions about the future init system in Debian. The discussion began in October 2003 with news on the Debian developer mailing list [14]: A complaint was filed about the dependency of the Gnome settings daemon on systemd – introduced by the hope that it wouldn't "turn into some kind of flame war."

But, that's just what happened. The discussion became heated and with no result, until it became clear after a week that no solution was possible. Another developer requested that the Debian Technical Committee (CTTE) [15] should take on the problem [16] and come up with a decision [17].

The CTTE familiarized itself with the technical issues and tried to find a solution supported by all its members. The board comprised eight members – of which four soon supported systemd, with the other four supporting Upstart. Of the latter, two were currently working at Canonical, and a third was a former Ubuntu employee.

The Debian project has its particular problems because of its many supported architectures. Thus, it was clear from the beginning that the Hurd and KFreeBSD platforms weren't working with systemd. This proved to be a problem technically and politically for the largest distribution that decides things without any corporate backing.

Of the three candidates that could replace SysVinit, only one could come to the rescue: OpenRC, developed in Gentoo. Although it currently has little chance of establishing itself as the default init system, with some effort, it could run both architectures.

In the case of Upstart, a fork of the software would apply, because Canonical only allows collaboration when the aforementioned CLA is signed (see the "Canonical's Gag" box). Mark Shuttleworth would probably rub his hands if Debian took over the further development of Upstart.

Canonical's Gag

The controversial Canonical CLA allows having software under a dual license. The developer might retain his copyright, but gives Canonical the right to put the code under another license and no longer develop it further under the original one.

In the case of systemd, the discussion over Canonical in general and CLAs in particular was rekindled. Not only did Sievers and Poettering express themselves, but so did Linux Torvalds [18], Greg Kroah-Hartman [19], and Matthew Garrett [20] in some detail.

Whereas Garrett didn't find CLAs to be all that bad – after all, the Apache Foundation uses one of them – Torvalds contradicted him and considered these agreements as "fundamentally broken." Systemd co-developer Sievers brought his pragmatic view of things sarcastically to the fore: "Without CLA we would still be working on Upstart instead of getting sensible things done on Systemd."

Early February saw a second set of ballot initiatives [21] that committee member Ian Jackson (a Canonical colleague) proposed, without resolution, only to be followed by a third. The third vote was like the first, a simple proposal – with one difference: It included a clause that allowed Debian developers to adopt the ultimate decision-making tool at Debian, the General Resolution (GR) and thereby repeal the decision of the CTTE with a simple majority [22].

Normally, such a decision needs a two-thirds majority at Debian. The GR needs six official Debian developers to agree. In mid-February, the third ballot led to almost four months of discussion. Finally, Bdale Garbee, chair of the committee, broke the stalemate by invoking for the first time a special voting procedure for such situations and declared systemd a winner [23].

Whether the developers tilt the vote again with another GR has yet to be seen. Systemd would probably win again but would have a broader base. Thus, the argument of an arbitrary decision would be removed from the hands of any critics, with Upstart having hardly a chance in the process. The procedure, however, showed that the CTTE statutes might require some reform. Attempts by one or the other members to drag out the process and torpedo it are hard to overlook.

In his blog, Canonical founder Mark Shuttleworth conceded a few days after the decision to be the "graceful loser" and announced that even Ubuntu would make the switch to systemd in the long run, provided developers declared it to be as stable as the previous solution, Upstart.

Thus, with Debian and Ubuntu, the last major distributions are on course with systemd. Together they form a critical mass to influence its development in keeping with Debian. There may also be less contention among the distribution warehouses. Gentoo and Slackware are the sole outsiders; Gentoo continues to rely on OpenRC, while Slackware will probably keep the faith with SysVinit.

Optimizing the Boot Process with Systemd

Systemd provides a few tools to analyze the boot process and optimize it. These tools all work under a user account, which means no root privileges – the corresponding command is systemd-analyze .

In our test, on a workstation with KDE and a good many services and SSD for the whole system, the result was not convincing (Listing 1, line 1). A freshly installed current notebook with KDE started faster (Listing 1, line 2). On a lighter desktop environment, the values were in the three-second range.

Listing 1

Results of systemd-analyze

01 02 Startup finished in 10.432s (kernel) + 6.287s (userspace) = 16.720s
03 Startup finished in 1.327s (kernel) + 3.116s (userspace) = 4.443s

If the startup takes too long, you can use systemd-analyze blame to test which processes and services are to blame. On an older notebook, the blame option revealed that the NTP service was stuck in a loop for more than 30 seconds. As it turned out, it was trying to contact a server that no longer existed, so a configuration correction fixed the problem.

If necessary, you can create a graph with systemd-bootchart that shows the behavior at boot time (Figure 6). The command is as follows:

Figure 6: Using a boot chart, you can tell at first glance which services are coming up on startup.
$ systemd-analyze plot > test.svg; inkscape test.svg

In this case, the Inkscape application helped in viewing the graph. Alternatives include image viewers such as Gthumb or the Firefox web browser.

Conclusion

The switchover to systemd does not present high hurdles to either distributions or users. In the near future, systemd remains compatible with Sysvinit so that the well-known commands for manipulating services work just as well as those for shutting down, restarting, and changing run levels.

Systemd proves to be a good solution for Debian and Ubuntu, to stand among the others, such as Fedora, openSUSE, Arch Linux, and Mageia, who are already using it. If the developers have any voice, systemd will help standardize processes for all distributions, providing advantages for developers and administration alike. More details on systemd are provided in Lennart Poettering's blog [24].

Some developers are accusing Red Hat of striving for more control over Linux with systemd. Upstart, on the other hand, would pull in a system that is totally under a company's control. Viewed objectively, systemd provides the best technically suited init system for the foreseeable future. Debian would be best to recognize this and be up front about it.l

Lennart Poettering Interview

Linux User: Why do you think systemd is viewed so critically, even though it's implemented more consistently than some other software we use every day?

Lennart Poettering: Systemd may evoke a controversy, but overall the assessment has been far more positive than negative. The recent decision from Debian in favor of systemd makes it clear that not only do individual distributions opt for the approach, but the clear majority of all major distributions do as well.

Of course, there is a significant minority in the community that criticizes or even totally rejects systemd. The reasons for it vary. A big part of it is that the concept cuts off some old heads and represents a major change in the classic Unix system.

Another reason lies in the fact that systemd not only optimizes the actual boot process, but unifies many of the system's low-level components.

Still others in the community reject it for economic reasons. Thus, Canonical has put a lot of effort into developing Upstart and was understandably interested in seeing to its own project's success.

LU: Debian, after much strife, has settled on some sensitivities for systemd. Do you see the last few months as a democratic process or simply as evidence of a lack of a benevolent dictator?

LP: The various distributions are organized differently. Debian is set on a democratic process, Ubuntu rather has the concept of a "benevolent dictator." Fedora is somewhere in the middle, a kind of technocracy.

Certainly the debate on this subject was largely unpleasant at Debian. But after all the chaotic discussion, it remains to be determined that this exactly constitutes the democratic process and that democracy works perfectly well.

The process takes its time and is sometimes no fun, but this is the deal: If we want to create a universal operating system, it means that we have to bring in a lot of people, which necessarily leads to extensive discussion and a wider range of contributions – positive as well as negative.

LU: Systemd is to extend further into userland, and you want to start processes in the desktop environment with the same mechanisms as the system itself. Will systemd be brewing our coffee?

LP: What exact tasks belong to systemd and which don't is debatable. There are few modules we're still integrating. Generally speaking, components of what we call the Plumbing Layer come into play – not the kernel or interface, but the "glue" between them.

For us, it's always the general usability, so we want a generic solution, not a specific one. Systemd is thus never a product, but something from which other developers can build products. At best, it should hold the system together without users being in direct contact with it.

To come back to the coffee. According to this definition, systemd is certainly a useful component, among many others, in building an Internet-capable coffee machine – but it never brews the coffee. That would be too specialized and no longer the "glue" that holds the coffee machine together.

I personally favor Club-Mate as the coffee source. That means that there's nothing further from my mind than to build coffee machine functionality into systemd.

LU: Not only your software, but you yourself are controversial. You deal with it pretty casually. Do you understand why so many criticize you personally, and how does that affect you?

LP: Anyone trying to swap out a key component of the Linux ecosystem is open for criticism; that's unavoidable. The community does have strong convictions and if you contradict them, it's easy to become a bogeyman.

Systemd was not the first project that tried to turn the boot system inside out – there've been dozens. We've often asked ourselves why we've been successful in advocating our approach. We'd like to believe it's based on technical grounds. At least in our view systemd definitely represents the better system.

How we've managed to deal with the community's pressure also plays a major role. My colleagues and I have probably been better at maintaining a good sense of humor and developing clear convictions, and that with a deep breath.

The Linux community sometimes behaves badly. If you want to set something in motion, you can't tread too softly. It would certainly be nice if it weren't that way!

Infos

  1. Poettering on Kdbus: https://-plus.-google.-com/-u/-0/-+LennartPoetteringTheOneAndOnly/-posts/-13JZ7GpyVDb
  2. SysVinit: http://-en.-wikipedia.-org/-wiki/-Init#-SysV-style
  3. SMF: http://-en.-wikipedia.-org/-wiki/-Service_Management_Facility
  4. Launchd: http://-de.-wikipedia.-org/-wiki/-Launchd
  5. Upstart: http://-de.-wikipedia.-org/-wiki/-Upstart
  6. OpenRC: http://-en.-wikipedia.-org/-wiki/-OpenRC
  7. CLA: http://-en.-wikipedia.-org/-wiki/-Contributor_License_Agreement
  8. Kexec: http://-en.-wikipedia.-org/-wiki/-Kexec
  9. Cgroups: http://-en.-wikipedia.-org/-wiki/-Cgroups
  10. Socket: http://-en.-wikipedia.-org/-wiki/-Network_socket
  11. Journal: http://-0pointer.-de/-blog/-projects/-journalctl.-html
  12. Logind: http://-www.-freedesktop.-org/-software/-systemd/-man/-systemd-logind.-service.-html
  13. ConsoleKit: http://-www.-freedesktop.-org/-wiki/-Software/-ConsoleKit/
  14. Developers list discussion: https://-lists.-debian.-org/-debian-devel/-2013/-10/-msg00444.-html
  15. Debian Technical Commmittee: https://-www.-debian.-org/-devel/-tech-ctte
  16. CTTE request: https://-lists.-debian.-org/-debian-devel/-2013/-10/-msg00703.-html
  17. Bug report to CTTE: https://-lists.-debian.-org/-debian-devel/-2013/-10/-msg00825.-html
  18. Torvalds on CLAs: http://-www.-muktware.-com/-2014/-01/-linus-torvalds-cla-fundamentally-broken/-19811/-2
  19. Kroah-Hartman on CLAs: https://-plus.-google.-com/-111049168280159033135/-posts/-NstZfwXbAti
  20. Garrett on CLAs: http://-mjg59.-dreamwidth.-org/-29160.-html
  21. Second set CTTE: http://-bugs.-debian.-org/-cgi-bin/-bugreport.-cgi?-bug=727708#-5389
  22. General Resolution: http://-www.-debian.-org/-vote/
  23. Final decision of CTTE: http://-bugs.-debian.-org/-cgi-bin/-bugreport.-cgi?-bug=727708#-6734
  24. Blog on systemd: http://-0pointer.-de/-blog/-projects/-systemd.-html