FreeBSD Journal
FreeBSD Journal
FreeBSD Journal
JOURNAL
Nov/Dec 2018
Introducing
FreeBSD 12.0
The Daemon’s Dozen
ZPool Checkpoint
Debugging the Kernel
®
Nov/Dec 2018
3 Foundation
Letter Welcome to
the last FreeBSD Journal
issue of 2018, and the
22 We Get Letters
launch of FreeBSD 12.0.
By George Neville-Neil
of FreeBSD.
debugging even if you are not a FreeBSD developer.
By Mark Johnston
In this issue, we shine a spot-
light on Sergey Kozlov who
received his ports bit in
September and Vinícius Zavam
who received his ports bit in
ZPool Checkpoint................. 18
31 Events Calendar
In March 2018, Alexander Motin ported the Pool
October. By Dru Lavigne
Checkpoint feature of OpenZFS from Illumos to
FreeBSD. This article provides a high-level description of
32 svn Update
By Dru Lavigne what happens under the hood during each administra-
tive operation. By Serapheim Dimitropoulos
For a while there was not
much action in HEAD, as the
FreeBSD 12.0 Toolchain
20
release engineering team had
frozen the tree to create a
branch of what will become
12-STABLE. That freeze has
Update.....................................
been lifted and we once again FreeBSD 12.0 continues the trend in recent FreeBSD
are seeing lots of new features releases of transitioning away from obsolete GPLv2-
and exciting changes. licensed toolchain components to modern ones.
By Steven Kreuzer By John Baldwin and Ed Maste
Nov/Dec 2018 1
Support
FreeBSD
®
JOURNAL LETTER
Editorial Board from the Board
John Baldwin • FreeBSD Developer, Member of the
FreeBSD Core Team and Co-Chair of
FreeBSD Journal Editorial Board.
Brooks Davis • Senior Computer Scientist at SRI
International, Visiting Industrial Fellow
at University of Cambridge, and mem-
ber of the FreeBSD Core Team.
Bryan Drewery • Senior Software Engineer at EMC Isilon,
member of FreeBSD Portmgr Team,
elcome to the last FreeBSD
Justin Gibbs •
and FreeBSD Committer.
Founder of the FreeBSD Foundation,
Director of the FreeBSD Foundation
Board, and a Software Engineer at
Facebook.
W Journal issue of 2018 and the
launch of FreeBSD 12.0. But first, we’ve
Daichi Goto • Director at BSD Consulting Inc.
(Tokyo). got a big announcement. Beginning with •
Joseph Kong • Senior Software Engineer at Dell EMC
and author of FreeBSD Device Drivers. the January/February 2019 issue, the
Steven Kreuzer • Member of the FreeBSD Ports Team.
Dru Lavigne • Director of Storage Engineering at FreeBSD Journal will be free to everyone.
iXsystems, author of BSD Hacks and
The Best of FreeBSD Basics. Yes, Free.
Michael W Lucas • Author of Absolute FreeBSD.
Ed Maste • Director of Project Development,
FreeBSD Foundation. • Why? you might ask. A big part of the FreeBSD
Kirk McKusick • Treasurer of the FreeBSD Foundation
Board, and lead author of The Design Foundation’s mission is to raise awareness of
and Implementation book series.
FreeBSD throughout the world. We know that the
George V. Neville-Neil • President of the FreeBSD Foundation
Board, and co-author of The Design
and Implementation of the FreeBSD
FreeBSD Journal is the best venue for keeping those
Operating System. interested in FreeBSD up-to-date. Our articles are
Philip Paeps • Secretary of the FreeBSD Foundation
timely and informative. Therefore, the Foundation
Board, FreeBSD Committer, and
Independent Consultant.
has decided that the benefit of making it accessible
Hiroki Sato • Director of the FreeBSD
Foundation Board, Chair of Asia to everyone at no charge will outweigh the cost of
BSDCon, member of the FreeBSD
Core Team, and Assistant producing it. However, donations will be accepted
Professor at Tokyo Institute of
Technology. if you’d like to help support the Journal.
Benedict Reuschling • Vice President of the FreeBSD
Foundation Board, a FreeBSD Docu-
mentation Committer, and member
of the FreeBSD Core Team.
• That’s not all. We’re also debuting an enhanced
Robert N. M. Watson • Director of the FreeBSD Foundation browser-based edition that will provide a much
Board, Founder of the TrustedBSD
Project, and University Senior Lecturer better experience for our readers.
at the University of Cambridge.
By Michael W LUCAS
The
Daemon’s
Dozen
Z
FS users are probably familiar with the beadm(8) package, the boot environment administration pro-
gram. We’ve previously covered it in the FreeBSD Journal. FreeBSD now includes bectl(8), a boot envi-
ronment control program. It retains much of the beadm(8) syntax, letting you create, activate, list,
and destroy boot environments. Additionally, it lets you mount unused boot environments and create jails
from those boot environments. I’ve used jails to troubleshoot problematic boot environments, so this is a
welcome addition to the base system.
Many people have used GELI to encrypt disks. Doing hard drive encryption properly is tricky, but GELI
• handily reduces the risk of exposing confidential data when your laptop gets stolen. GELI was painful to
use on systems with many hard drives, though. Each disk needed to be configured separately. With
FreeBSD 12.0, GELI can now configure multiple disks simultaneously so long as they use the same encryp-
tion key, easing this annoyance for sysadmins who administer such machines. Also, in the GELI realm, the
installer can now encrypt disks on hosts booting off of UEFI.
FreeBSD’s GEOM system lets us stack storage transformations on top of one another in exactly the way
you need for your application, hardware, and environment. If you want a single massive striped and
encrypted storage array, you could stripe the disks together and then encrypt them—or you could encrypt
each disk separately and then stripe them. GEOM empowers you. The catch, of course, comes when
you’re trying to figure out what you did with that power. The new geom -t command displays the host’s
GEOM stack.
4 FreeBSD Journal
geom -t
dropped from the current kernel. Many
Geom Class Provider drivers for 10/100 Mbs cards such as
ada0 DISK ada0 ed, ep, pcn, and so on are deprecated
ada0 DEV and salted for removal in FreeBSD 13.0.
ada0 PART ada0p1
Twenty years ago, I was a big fan of
ada0p1 DEV
NE2000-compatible cards. They were
ada0p1 LABEL gpt/gptboot0
essentially indestructible. The problem
gpt/gptboot0 DEV
ada0 PART ada0p2 is that was twenty years ago. It’s time
ada0p2 DEV to buy a new network card.
swap SWAP FreeBSD has had a long-running
ada0 PART ada0p3 problem with video drivers. Video cards
ada0p3 DEV come out more frequently than FreeBSD
zfs::vdev ZFS::VDEV releases. Users sensibly want to use all
the fancy features of their new video
cards and would get frustrated when a
You can easily see how the providers and consumers
two-month-old FreeBSD didn’t support those drivers.
get stacked onto each other, what goes where, and
FreeBSD solved this by removing the advanced video
the labels applied to everything.
drivers from the base operating system and making
Similarly, if you want to see all of the information
them a port instead. Ports are much more easily
about a particular GEOM container, use the new
updated than base system components, meaning
geom -p command. Here I tell GEOM to spill every-
users get accelerated video and 3D support and all
thing it knows about da2p1.
the other fancy stuff they want. Install the drm-kmod
geom -p da2p1 package for the latest video drivers and see the
FreeBSD Graphics Wiki at https://wiki.freebsd.org/
Some storage changes are not so obvious. A TRIM
Graphics for details on how to use it.
request is used to tell an SSD or a memory filesys-
If you need to extract information from last(1)
tem that a block is no longer used. FreeBSD 12.0
and lastlogin(8), these programs now support libxo.
batches and consolidates those requests, reducing
Try last --libxo xml to get a sample of the out-
disk I/O. You’ll see dozens of such minor perform-
put and use your organization’s standard XML tools
ance enhancements throughout the release.
to gather the information you need.
FreeBSD 12.0 has one major new feature for stor-
There’s a bunch of other new features and
age, though: a parallel NFS (pNFS) server. Parallel
details. The boot loader now includes Lua rather
NFS can greatly enhance NFS performance but
than Forth, so all you Lua hackers can customize the
requires a pNFS-aware client. FreeBSD’s NFS client is
boot process. Some older, standard device drivers
pNFS-aware. It’s a brand-new feature, though, and
have undergone improvements. While the console
rather than telling you to deploy it immediately I’d
could previously display text faster than I can read,
encourage you to thoroughly test it in your environ-
the vt(4) driver now runs up to four times as fast as
ment. pNFS also requires NFSv4, so if you’re still
in FreeBSD 11.0. Contributed programs like ntpd(8)
using an older version of NFS it’s time to look at
and mandoc(1) have been upgraded. The random
upgrading.
number generator random(4) has seen moderniza-
NFS touches on the network, so let’s talk net-
tion, and the integrated OpenSSL will get support
working next.
for several years. The default pager is now less(1),
The FreeBSD 12.0 kernel enables the virtual net-
and newsyslogd(8) ignores setuid and executable log
working stack VIMAGE by default, allowing you to
files—as it should.
use multiple routing tables. You can now have your
FreeBSD 12.0 has many smaller enhancements,
host use its standard routing table but create a new
improvements, and tweaks. Really, the best way
routing table just for jails running on that host. Also,
to see them all is to install it and see what
you can now use PF within a jail, and PF’s HFSC traf-
happens. I’ve been happy with it. I think
fic shaping can now handle up to 100 Gbs.
you will be as well.
Eight-bit 10Mbs network cards are no longer
popular enough for FreeBSD to support, and their
Michael W Lucas has been using BSD since the
presence in the kernel creates roadblocks for devel-
1980s. His most recent book is the third edition of
opment of current features. Drivers for purely
Absolute FreeBSD. Learn more at https://mwl.io.
10Mbs cards like lmc, ixgb, and nxge have all been
Nov/Dec 2018 5
SEE
TEXT
O N LY
DeBUGGING
the FreeBSD Kernel
By Mark Johnston
As I write this, the long-awaited FreeBSD 12.0 release is
within days of shipping. Of course, FreeBSD developers have
been meticulously polishing and testing 12.0's kernel for months
and we're confident that every last bug has been squashed.
or the typical FreeBSD user, the subject of kernel debugging is just an intellectual curiosity, right? This
F is how it should be, at least; FreeBSD's stability is one of its main selling points and contributes a lot
to its reputation. The stability of an operating system kernel is especially important since a fault in the
kernel will typically bring down the entire system. In truth, however, no matter how many bugs were fixed
in the lead-up to 12.0, the harsh reality is that some are still lingering and will emerge later to bite users,
often with a stress-inducing kernel panic.
It is useful to have some familiarity with FreeBSD kernel debugging even if you are not a FreeBSD devel-
oper. The permissiveness of the BSD license has allowed many companies to build products on top of the
FreeBSD source tree, often directly extending the kernel. These extensions will, of course, contain bugs or
may expose latent bugs in FreeBSD itself. Moreover, developers working on such a code base generally
cannot work directly with the FreeBSD community and thus would need to do some debugging them-
selves, perhaps before reporting the issue upstream. Another very common scenario is the case where
developers do not have direct access to the system exhibiting a problem, so live debugging is not possible.
Instead, any data required for debugging must be collected by an administrator or user. Knowing what to
collect and how to collect it can greatly speed up the debugging process.
Finally, familiarizing yourself with our debugging tools is impor-
tant for becoming a proficient kernel developer; as a newcomer you
will inevitably make mistakes, some trivial and some not, and debugger
can often reveal a problem more quickly than staring at one's own code.
Kernel bugs can of course have many different causes, but the high-level
approach to resolving them is always the same:
1. Find the root cause of the bug or get as close as possible.
2. Find a way to work around the bug, if both possible and necessary.
3. Fix the bug.
Step 2 might be much more important than step 3 if the bug is affecting a produc-
tion workload, and step 3 might not be achievable; hardware failures and firmware bugs
are common sources of what appear to be kernel bugs, and generally cannot be fixed
directly. However, step 1, which generally consists of kernel debugging, is a prerequisite
of both steps 2 and 3. The task of debugging an operating system kernel might seem
daunting, and, indeed, it poses some unique challenges, but methodologically it is the
same as debugging any userspace application.
The FreeBSD kernel is a large body of code by most standards, and one cannot rely on
code inspection and debug-by-printf to get very far with anything except trivial bugs. The
fact that one must reboot in order to test a new kernel also constrains these kinds of primi-
tive debugging techniques. FreeBSD contains various facilities to do post-mortem debug-
ging and to help find reproducible triggers for bugs without a known cause, at which point
debugging tools can be brought to bear on the problem. This article aims to highlight
some of these facilities and provide some techniques for using them.
6 FreeBSD Journal
Triggering Kernel Bugs
Unexpected kernel panics are never pleasant. Almost as bad, though, is trying frantically to replicate someone
else's kernel panic and failing. Usually this happens when the bug is timing-dependent, as in a race condition
or a use-after-free. There is no silver bullet for these situations, but the FreeBSD kernel provides several facili-
ties that may prove useful.
INVARIANTS Kernels
INVARIANTS is a kernel option that is configured in development builds of the FreeBSD kernel—if you
are not running a FreeBSD release, it is quite likely that you are using GENERIC with INVARIANTS configured.
In this case, congratulations! Your kernel has thousands of integrity checks enabled, constantly testing the
assumptions of the programmers who wrote it. These checks impose runtime overhead, making INVARIANTS
unsuitable for many production workloads, but they should be enabled whenever it is feasible. If you report a
FreeBSD kernel panic to the mailing lists, it is quite likely that you will simply be asked to test with an INVARI-
ANTS kernel if you are not already doing so. Other kernel options, such as DIAGNOSTIC and
DEBUG_VFS_LOCKS, enable further rounds of more expensive integrity checking that can catch many bugs.
Memory Trashing
The FreeBSD kernel has an implementation of malloc(9) very similar to that used in regular C applications.
Additionally, a memory subsystem called UMA ("Universal Memory Allocator") provides a slab allocation API
for high-performance code paths. When INVARIANTS is configured in the kernel, memory freed using either
of these APIs is "trashed," meaning that its contents is overwritten with a specific pattern. Any time trashed
memory is reused, its contents are verified against that pattern. If verification fails, it means that the memory
was written to after it was freed, and the kernel panics.
This is a very standard technique and is implemented in most full-featured memory allocators. On FreeBSD, the
trash pattern is 0xdeadc0de so that, while the allocator itself will catch a write-after-free, anything that reads a
trashed memory location (i.e., a read-after-free) is also likely to crash if it treats the value as a pointer. When
debugging or triaging a kernel panic, keep an eye out for 0xdeadc0de —it very often signifies a use-after-free.
Memory trashing helps use-after-frees stand out by making it clear what the use-after-free did. Consider
the following memory dump:
0xc00000037d3c0ba0: 0 fffffe00017f2530
0xc00000037d3c0bd0: fffff80002268e80 ffffffff80b35c58
0xc00000037d3c0be0: deadc0dedeadc0de deadc0dedeadc0de
0xc00000037d3c0bf0: deadc0dedeadc0de deadc0dedeadc0de
0xc00000037d3c0c00: 0 deadc0dedeadc0de
0xc00000037d3c0c10: deadc0dedeadc0de deadc0dedeadc0de
0xc00000037d3c0c20: deadc0dedeadc0de deadc0dedeadc0de
0xc00000037d3c0c30: deadc0dedeadc0de deadc0dedeadc0de
Here we can see that the write-after-free wrote 8 bytes of zeros at a specific offset within a structure; that
can be a helpful clue for identifying the responsible code.
A use-after-free is an especially nasty type of bug since the cause and effect of the bug may be far
removed from each other. Memory trashing helps identify that a use-after-free occurred but can't catch the
offending code in the act. This is where MemGuard comes in.
MemGuard
MemGuard is a memory subsystem feature which proactively detects use-after-frees for a particular memory type.
A memory type is defined by either the type argument to malloc(9), or a UMA zone. For example, the line:
The M_KQUEUE malloc(9) type is thus named "kqueue". Statistics for each malloc(9) type can be
listed with: vmstat -m, and similar statistics for UMA zones can be listed with vmstat -z.
MemGuard is not present in FreeBSD kernels by default and must be compiled in by adding the
Nov/Dec 2018 7
DEBUG_MEMGUARD option to the kernel configuration. Then, to configure MemGuard for a particular mem-
ory type, set vm.memguard.desc to the name of the malloc(9) type or UMA zone, either by adding an
entry to /boot/loader.conf or by setting the sysctl. Once MemGuard is configured, it hooks all alloca-
tions and frees of the memory type. Each allocation is padded to a full page, and when a piece of memory
is freed, the entire range of allocated memory is unmapped. This means that an attempt to access the mem-
ory after it has been freed will likely result in a page fault which crashes the kernel, catching the use-after-
free as it happens.
The MemGuard facility is quite powerful but is correspondingly heavy-handed. Since each memory alloca-
tion must be padded to a full page, it will waste a large amount of memory if the memory type is used for
small allocations. For instance, enabling MemGuard on the mbuf zone causes 4096 - 256 = 3840 bytes to
be wasted per mbuf. The MemGuard allocation and free hooks will also severely impact the performance of
the allocation: unmapping kernel memory is an expensive operation that must be serialized across all CPUs
in the system. Thus, on a system with many CPUs and at a high allocation rate, MemGuard can easily
become a major bottleneck. If the use-after-free is timing-dependent as is often the case, MemGuard's over-
head can be such that the bug stops occurring. Nonetheless, it is quite effective in practice.
Fail Points
Many bugs, both in the kernel and userspace, are the result of incorrect or insufficient error handling. These
bugs lay dormant until the corresponding error occurs, so the commonly executed "happy path" is unaffected.
When the error does occur, it may be the result of a hardware or firmware malfunction, or some other seem-
ingly unlikely event. Suppose you are in the position of having to debug the error-handling problem, and
against all odds, you think that you have spotted the bug and want to verify your theory. How do you go about
this? How can you write a regression test to ensure that your fix remains valid into the foreseeable future?
FreeBSD's fail point subsystem (documented in fail(9)) is a useful tool here. Fail points are a mecha-
nism that allows programmers to inject errors and modify behavior at pre-defined locations in the kernel
code. Fail points do not have any effect by default, but can be activated using sysctl(8): each fail point
is controlled by a unique sysctl, usually under debug.fail_point. Fail points can be used to modify stack
variables (usually to inject an error), change control flow (by following a goto or returning early from a
function), or sleep or pause (useful for triggering race conditions).
The fail point interface consists of a set of macros, most of which are convenience wrappers for
KFAIL_POINT_CODE. For example, we might try using a fail point to simulate random failures from
malloc(9). The following fail point at the end of malloc() would do the trick:
void *
malloc(size_t size, struct malloc_type *mtp, int flags)
{
...
if ((flags & M_WAITOK) == 0)
/* M_WAITOK allocations are not allowed to fail. */
KFAIL_POINT_CODE(DEBUG_FP, malloc, {
free(va, mtp);
va = 0;
});
return ((void *)va);
}
void *
malloc(size_t size, struct malloc_type *mtp, int flags)
{
...
KFAIL_POINT_CODE_COND(DEBUG_FP, malloc, (flags & M_WAITOK) == 0, 0, {
free(va, mtp);
va = 0;
});
return ((void *)va);
}
8 FreeBSD Journal
Either way, we must take care to free the newly allocated memory ourselves before simulating an allocation
failure: otherwise that memory would be permanently leaked. For ad-hoc debugging, this might be accept-
able, but in general, one must take care to ensure that a fail point does not introduce a new bug.
With this fail point defined, we get a sysctl named debug.fail_point.malloc. Fail points are activat-
ed by setting the corresponding sysctl to a string which describes the action to take when a thread executes
the fail point. The default action is "off," meaning that the fail point has no effect. The "return" action caus-
es the fail point's code to be executed; other actions are defined in the manual page. The kernel would not
survive for long if all malloc(M_NOWAIT) allocations failed, so fail points can be configured to execute with
a specified probability. The setting
# sysctl debug.fail_point.malloc='1%*return'
would cause the fail point's code to be executed roughly once for every 100 malloc(M_NOWAIT) calls.
Fail points are particularly useful for simulating disk errors; such errors are hopefully quite rare, but filesys-
tems, among other things, must handle them robustly. When properly integrated, fail points can be an impor-
tant part of the regression testing regimen for such code. For example, FreeBSD's RAID1 implementation gmir-
ror contains fail points used to simulate errors from the underlying disks. Over time, a number of bugs in
gmirror's error handling have been fixed, and fail points are now used in several automated {gmirror} tests.
Kernel Stack Tracing
Occasionally a kernel bug will manifest with a thread entering an infinite loop in the kernel. No crash
occurs, but a keen observer might report the issue as a task in top(1) consuming 100% of a CPU for no
obvious reason. If the system is not being used in production, it might be reasonable to drop in to a kernel
debugger to interrogate the task in question, but this is not always possible.
FreeBSD has the ability to print kernel stacks without pausing the system using the procstat(1) utility's
-k option. procstat -kk <pid> prints kernel stacks, and procstat -kka prints kernel stacks for every
thread in the system. While this does not directly help in finding a trigger for the bug, it provides an accessi-
ble starting point both for tasks that appear "stuck" and are unresponsive, and for tasks that appear to be
spinning forever. When procstat(1) is asked to capture the stack for a given thread and the thread is cur-
rently running on a CPU, procstat(1) will raise an interrupt on that CPU and capture the thread's stack
frames from the interrupt handler. While this may not provide sufficient information to root-cause a bug on
the spot, it is often a useful clue, and the fact that it can be used without bringing down the entire system
makes it a handy place to begin an investigation.
Nov/Dec 2018 9
to a filesystem! The simplest approach is to save the dump on the system's swap partition: the contents of
swap are, by definition, useless after a reboot, so there is no harm in it being overwritten. This behavior can
be configured by adding the line
dumpdev="AUTO"
to /etc/rc.conf —if a swap device is configured in /etc/fstab, it will then be used for kernel
dumps. Otherwise, to save dumps to a local disk, the name of the disk device must be specified instead of
"AUTO."
When configuring a dump device, it's a good idea to test it by triggering a panic and verifying that a
vmcore file appears in /var/crash after the reboot:
# sysctl debug.kdb.panic=1
The main difficulty with a local dump device is storage space. The size of a kernel dump is roughly pro-
portional to the amount of RAM in the system. While minidumps greatly reduce the need for space, they
can still easily be in the tens of gigabytes on a system with 128GB of RAM, larger than a reasonably sized
swap partition. A system administrator would be forced to either waste disk space or potentially lose the
ability to save kernel dumps. Fortunately, FreeBSD 12.0 comes with several new features to help alleviate
this problem. First, the code which saves a kernel dump can now compress the data before writing it,
greatly reducing space requirements. Currently, the zlib and newfangled Zstandard algorithms are support-
ed for this purpose. Zstandard is both faster and better at compressing kernel dumps, and we frequently
see compression ratios of 8:1 or better. Either algorithm can be configured by setting dumpon_flags in
/etc/rc.d; see the dumpon(8) manual page for details. Zstandard compression can thus be easily con-
figured:
dumpdev="AUTO"
dumpon_flags="-Z"
Compression staves off the storage requirement problem for now, but the fundamental problem
remains. Moreover, FreeBSD is used on many systems that do not have a suitable dump device to begin
with. For example, many embedded devices—such as routers—boot from read-only media and do not have
any dynamic storage. Virtual machines are often provisioned without swap space, and diskless servers boot
from an NFS mount and do not have any local storage at all. To address these scenarios, FreeBSD 12.0 will
come with the ability to transmit kernel dumps over a network interface using an IPv4 UDP-based protocol
called Netdump.
Netdump
Netdump provides flexibility to system administrators and developers by allowing them to use a remote
server to receive kernel dumps from clients (panicking kernels). Netdump is integrated with the existing ker-
nel dump facilities and is transparent with respect to the kernel dump features described earlier; it is possi-
ble to use kernel dump compression or encryption with Netdump, just as with dumps to a local disk. From
an adminstrator's perspective, Netdump simply requires additional configuration.
Netdump clients use dumpon(8) to configure client parameters. Instead of specifying a block device,
specify the name of the network interface to be used. This should preferably be a low-traffic management
interface rather than one of the system's data ports: Netdump will be less reliable if the interface that it
uses was busy at the time of the panic. Netdump does not automatically support every network adapter,
but many commonly encountered adapters are, indeed, supported, and a list of supported drivers can be
found in the netdump(4) manual page. Of course, the network interface name is not sufficient: the
Netdump client code needs to know how to contact the server! Three network address parameters must be
specified:
10 FreeBSD Journal
These parameters can all be specified in /etc/rc.conf:
dumpdev="em0"
dumpon_flags="-s 192.168.1.34 -c 192.168.2.154 -g 192.168.2.1"
At the time of writing, only IPv4 parameters are supported. We hope to support IPv6 in future releases of
FreeBSD.
Because the Netdump client code runs only after the system has panicked, it cannot use any of the kernel
code which normally does the work of building IPv4 packets and sending them out onto the wire. In particu-
lar, it cannot make use of the kernel's ARP cache, so as a first step, it must resolve the Ethernet address of
the next hop. In fact, the Netdump client always attempts to resolve the server directly before trying to
resolve the gateway. This way, if both the client and server happen to be on the same link, they can commu-
nicate directly. Otherwise, the client relies on the gateway to forward all Netdump packets to the server.
The Netdump server program is called netdumpd and is available as a package for FreeBSD 11 and 12. It
can be configured via /etc/rc.conf; its main parameter specifies the root directory under which it will
save all kernel dumps from clients. netdumpd listens on UDP port 20023 for HERALD messages from a
client (described below). Using an ephemeral port, it replies to all client messages with an ACK message.
When a client receives the ACK for its initial HERALD, it sends all subsequent messages to the UDP port from
which the ACK originated. When a Netdump session completes successfully, the server saves two files under
its root directory: a vmcore file, identical to those created by savecore(8), and a text file (info) contain-
ing metadata describing the kernel dump. It may optionally be configured to execute a user-defined script
upon completion, successful or otherwise. This can be used, for example, to send an email to the owner of a
system when that system panics and transmits a kernel dump.
The Netdump protocol is extremely simple by design and closely resembles the TFTP protocol. All messages
consist of a fixed-size header. Client messages begin with:
struct netdump_msg_hdr {
uint32_t mh_type;
uint32_t mh_seqno;
uint64_t mh_offset;
uint32_t mh_len;
uint32_t mh__pad;
};
struct netdump_ack {
uint32_t na_seqno;
};
Server messages are very simple: they acknowledge receipt of a client message by including the correspon-
ding sequence number.
Nov/Dec 2018 11
kgdb
After going through all the work to trigger a panic and obtain a kernel dump, it's time to do some actual
debugging. With kgdb, opening the kernel dump is just like opening a userland core dump using gdb: you
simply point kgdb at the kernel binary and vmcore files:
As with gdb, debug info is needed in order for kgdb to make sense of a kernel dump's contents. In
recent versions of FreeBSD, these symbols are stored externally, in /usr/lib/debug/boot/kernel,
assuming that the kernel being debugged is in /boot/kernel. When sending a kernel dump to a devel-
oper, it is important to include the contents of both /boot/kernel and /usr/lib/debug/boot/kernel.
kgdb is smart enough to unwind through trap frames, printed above as <signal handler called>.
These occur when the CPU interrupted the running thread, saving its state on the stack. In the example, we
have two such frames: first, a timer interrupt interrupted an idle CPU, and, while processing pending call-
outs, that CPU took a protection fault caused by a corrupted pointer:
(kgdb) frame 11
#11 0xffffffff808594cf in callout_process (now=1960148907297521)
510 LIST_REMOVE(tmp, c_links.le);
(kgdb) p tmp
$1 = (struct callout *) 0x11777be9162acbc1
(kgdb) x tmp
0x11777be9162acbc1: Cannot access memory at address 0x11777be9162acbc1
Most of the usual gdb commands work in kgdb. For instance, info threads will enumerate all of
the system's threads, including kernel-only threads, and one can switch between threads. As an extension,
12 FreeBSD Journal
kgdb can identify threads based on FreeBSD thread ID. This is handy when a data structure points to a partic-
ular thread and you want to switch to that thread; for example, when the current thread is blocked on a lock,
we may wish to look at the thread holding that lock:
(kgdb) p kq
$5 = (struct kqueue *) 0xfffff801cb5a4a00
(kgdb) p/x kq->kq_lock
$6 = {
lock_object = {
lo_name = 0xffffffff814e8f66,
lo_flags = 0x1430000,
lo_data = 0x0,
lo_witness = 0x0
},
mtx_lock = 0xfffff801e3b34000
}
(kgdb) p ((struct thread *)kq->kq_lock.mtx_lock)->td_tid
$7 = 101322
(kgdb) tid 101322 # Switch to the thread with TID 101322.
(kgdb)
kgdb offers scripting facilities: GDB script, which is the native GDB command language with some stan-
dard programming constructs like if statements and while loops, and a Python API. The GDB scripting lan-
guage is rather primitive and limited, so we will focus on the more powerful Python integration and provide
several examples. The API documentation is quite thorough, so the examples merely portray some possible
uses for the API without explaining each detail of how they work.
The kernel's struct thread contains many fields specific to one subsystem; kernel developers effectively
allocate thread-local storage by adding fields to this rather large structure. When debugging, a common need is
to find the pointer to the currently selected thread. This can often be done in an ad-hoc way, by searching vari-
ous frames of the stack for a copy of the pointer, but this is tedious and not scriptable. Instead, let's use Python
to add a "convenience function" to behave like the magic curthread variable in the kernel:
import gdb
allproc = gdb.lookup_global_symbol("allproc").value()
for p in list_foreach(allproc, "p_list"):
if pid != -1 and pid != p['p_pid']:
continue
for td in tailq_foreach(p['p_threads'], "td_plist"):
ntid = td['td_tid']
tdfind.cached_threads[int(ntid)] = td
if ntid == tid:
class curthread(gdb.Function):
def __init__(self):
super(curthread, self).__init__("curthread")
def invoke(self):
return tdfind(gdb.selected_thread().ptid[2])
curthread() # Register the function with kgdb.
The API provides a way to get the selected thread's thread ID, but we must traverse the kernel's data
structures ourselves to get the corresponding struct thread pointer. This is done by the tdfind() rou-
tine, which emulates a kernel function of the same name to search the kernel's global list of processes
(allproc) and find the thread with the specified thread ID. The global list can be quite large, so we cache
all threads encountered during the search to speed up future lookups.
The curthread class provides the requisite glue: when $curthread() is invoked it simply returns the
result of calling tdfind() on the selected thread's thread ID:
(kgdb) p $curthread()
$1 = (struct thread *) 0xfffff801e3b34000
Now we need not grovel through the stack to find the current thread pointer, and the lookup code can be
reused for other purposes.
With the addition of VIMAGE to the standard kernel configuration, many global data structures in the net-
work are virtualized: every reference to such a global structure is actually relative to the current vnet, so
there is some implicit context to each reference. The kernel uses a set of macros in vnet.h to handle this,
but kgdb doesn't know about them; the upshot of all this is that the following doesn't work:
We can get close, though, by providing a convenience function to resolve symbols in the context of the
current vnet, or in a user-specified vnet if that happens to be useful:
class vimage(gdb.Function):
def __init__(self):
super(vimage, self).__init__("V")
if vnet is None:
vnet = tdfind(gdb.selected_thread().ptid[2])['td_vnet']
if not vnet:
# If curthread->td_vnet == NULL, vnet0 is the current vnet.
vnet = gdb.lookup_global_symbol("vnet0").value()
base = vnet['vnet_data_base']
entry = gdb.lookup_global_symbol("vnet_entry_" + sym).value()
entry_addr = entry.address.cast(gdb.lookup_type("uintptr_t"))
ptr = gdb.Value(base + entry_addr).cast(entry.type.pointer())
return ptr.dereference()
vimage() # Register the function with kgdb.
14 FreeBSD Journal
This snippet effectively reimplements what the real VIMAGE implementation does: get a pointer to the
current vnet from the current thread and do a bit of math to find that vnet's copy of the desired structure.
It also tries to provide sane behavior if the kernel does not support VIMAGE by simply returning the global
structure. Additionally, the desired vnet may be explicitly specified if needed. While we cannot simply print
V_tcbinfo, we can now get pretty close:
(kgdb) p $V("tcbinfo")
$1 = {
ipi_lock = {
lock_object = {
lo_name = 0xffffffff8125b55e "tcp",
...
As a final example, we will implement the acttrace command from DDB, which prints the backtraces
from all threads that were actively executing on a CPU at the time of the panic. This is often a very important
piece of the picture, as many bugs are the result of errant interactions between multiple threads. With luck,
the threads that caused the panic will have been on-CPU at the time of the panic.
This example adds a new top-level command, mimicking DDB's interface. It uses the per-CPU (pcpu)
structures which store information about the execution state of each CPU in the system.
import math
def cpu_foreach():
all_cpus = gdb.lookup_global_symbol("all_cpus").value()
bitsz = gdb.lookup_type("long").sizeof * 8
maxid = gdb.lookup_global_symbol("mp_maxid").value()
cpu = 0
while cpu <= maxid:
upper = cpu >> int(math.log(bitsz, 2))
lower = 1 << (cpu & (bitsz - 1))
if (all_cpus['__bits'][upper] & lower) != 0:
yield cpu
cpu = cpu + 1
class acttrace(gdb.Command):
def __init__(self):
super(acttrace, self).__init__("acttrace", gdb.COMMAND_USER)
Nov/Dec 2018 15
Continued
The idea is to iterate over all online CPUs in the system, looking up the per-CPU structure and using that
to find the thread that is on the CPU. We use the thread's ID to look up kgdb's representation of that
thread, switch to it, and print the backtrace. Once this is all done, we switch back to the starting thread to
avoid confusing the user.
(kgdb) acttrace
Tracing command idle pid 10 tid 100002 (CPU 0)
#0 cpustop_handler ()
#1 0xffffffff80928074 in ipi_nmi_handler ()
#2 0xffffffff808c0e19 in trap (frame=0xffffffff8104b160 <nmi0_stack+3888>)
#3 <signal handler called>
#4 acpi_cpu_idle_mwait (mwait_hint=0)
...
Conclusion
Hopefully this article has helped illuminate several
stages of the typical FreeBSD kernel developer's Mark Johnston is a
debugging workflow. Of course, experience and freelance FreeBSD devel-
intuition can get you pretty far, but good tooling is oper living in Toronto,
tremendously helpful in tracking down the more Canada. In his spare
slippery bugs that arise, especially when they lie time he enjoys attempt-
outside one's domain of expertise. Debuggers also ing to make nice sounds
help attune one to various patterns and details that on a cello, running, trav-
underlie the core functionality of the kernel, so elling, and playing
with experience it becomes easier to spot things dodgeball on a team
that "just don't look right": clues which might lead with his friends.
to a root cause. Happy debugging! •
®
TM
TM
®
JOURNAL
Wishing You All A Very
16 FreeBSD Journal
Nov/Dec 2018 • 17
SEE
TEXT
O N LY
I
n March of this year (2018), Alexander Motin records the TXG during which it was created,
(mav@freebsd.org) ported the Pool Checkpoint called its birth TXG. Birth TXGs are important for
feature of OpenZFS from Illumos to FreeBSD. A the pool checkpoint feature because they help us
pool checkpoint can be thought of as a “pool- differentiate between blocks created before and
wide snapshot” that captures the entire state of after a checkpoint. Finally, a TXG’s last step when
the pool at the time the checkpoint is created, syncing to disk is writing the new uberblock to the
allowing the user to revert the entire pool back to beginning and end of each disk in an area called
that state or discard it. The generic use cases are the ZFS label, which is different from the EFI label.
administrative tasks, like OS upgrades, that involve Each uberblock is the root of the tree of the entire
actions that change or destroy ZFS state and meta- state of the pool for that TXG. Uberblocks are
data. Examples of such actions are: enabling new used during pool import as a starting point to find
pool features, changing properties of datasets, or the latest version of all of the pool’s data and
destroying snapshots and filesystems. Before metadata.
undertaking such actions, administrators can cre- Whenever an administrator checkpoints a pool
ate a checkpoint of their pool and then apply their (with the “zpool checkpoint” command), ZFS
changes. If something goes wrong with the copies the uberblock of the current TXG to an area
upgrade, the administrator can then rewind back within the pool’s state called the Meta-Object Set
to the checkpoint as if the actions had never been (MOS). When this change is synced to disk, the
taken. In the same way a snapshot can help you uberblocks from subsequent TXGs reference the
return user data to a previous state, the check- “checkpointed” uberblock through the MOS.
point can help you return the ZFS pool to a previ-
ous state. The Lifetime of a Block in a
There are already two tutorials online demon- Checkpointed Pool
strating how to use this feature, and a block com- When a checkpoint exists, the process of allocating
ment in the source code that gives a high-level new blocks stays the same, but the process of
overview of its implementation (see the References freeing blocks is different. We can’t free blocks
box). This article lies somewhere in the middle, giv- that belong to the checkpoint, because we want
ing a high-level description of what happens under to be able to rewind back to that point in time. At
the hood during each administrative operation. the same time, we can’t stop freeing blocks entire-
ly, because that would fill up the pool. Thus, every
Checkpointing a Pool time we are about to free a block, we look at its
In ZFS we keep track of changes over time with birth TXG and compare it to the TXG of the
Transaction Groups (aka TXGs). During each TXG, “checkpointed” uberblock that we saved in the
ZFS accumulates changes in memory and, when MOS. If the block was born at or before the TXG
certain conditions are met, it syncs those changes of the checkpointed uberblock, it means that the
to disk, then opens the next TXG. Each data block block is part of the checkpoint (i.e., it is referenced
18 FreeBSD Journal
by the checkpointed uberblock). These “checkpoint- (“zpool checkpoint --discard”). The com-
ed” blocks are never actually freed. Instead, we add mand instructs ZFS to get rid of the checkpointed
their ranges to lists on-disk we call checkpoint uberblock from the MOS. At that point, the pool is
spacemaps (there is one per vdev) and leave their considered to no longer have a checkpoint, which
segments marked as allocated so they are not allows blocks to be freed normally regardless of their
reused by the pool. Blocks whose birth TXGs are birth TXG. ZFS will also free all the previously record-
after the checkpoint’s TXG, are not part of the ed ranges from the checkpoint spacemaps—the
checkpointed state and can be freed normally. blocks that we couldn’t actually free because they
were referenced by the checkpointed uberblock.
Rewinding to the Checkpoint The number of these blocks can be quite large
If administrators want to rewind back to the check- depending on how long the checkpoint has existed
point, all they need to do is to export and then re- and how many changes have been made to the
import the pool with the rewind option pool. Freeing them all in a single TXG would be
(“zpool import --rewind-to-check- expensive. Instead, ZFS spawns a thread that frees
point”). In this case, the import process takes them over the course of multiple TXGs by prefetch-
place as usual, but with one additional step. ZFS first ing them into memory in chunks and freeing one
looks at the current uberblock and from that it finds chunk per TXG.
the checkpointed uberblock in the MOS. Then it
uses the checkpointed uberblock instead of the cur- Acknowledgments
rent uberblock for the import process, effectively The development of this feature would not have
rewinding the pool back to the checkpointed state. been possible without the help of my colleagues
Once the import process is done, the rewind is com- from Delphix, especially Dan Kimmel, who started
plete. The checkpoint spacemaps no longer exist as the initial prototype with me, and Matt Ahrens for
they were created after the checkpointed uberblock guiding me every step of the way. I’d also like to
(which is now the current uberblock). For the same thank Alexander Motin for porting the feature to
reason, there is no checkpoint uberblock in the FreeBSD, and Marius Zaborski for promoting it. •
MOS. This means that after rewinding there is no
additional cleanup and the pool no longer has a .
checkpoint.
Serapheim Dimitropoulos is a
It is also possible to import a checkpoint read-only
to access the pool in the state it existed at the time software engineer working on
of the checkpoint without actually undoing all of ZFS at Delphix. His main
the changes that have happened since the check- contributions to the project
point was created. This can allow the administrator are the Log Spacemap and Pool
to recover specific data or a filesystem that was Checkpoint features. When not
destroyed without rolling back the entire pool. programming, Serapheim spends
his time playing soccer,
Discarding the Checkpoint salsa dancing, and playing
If administrators decide to get rid of the checkpoint, classical guitar.
they run the discard command
REFERENCES
Serapheim's ZPool Checkpoint tutorial. http://sdimitro.github.io/post/zpool-checkpoint/
Nov/Dec 2018 19
SEE
TEXT
O N LY
FreeBSD 12.0
Toolchain
Update
By John Baldwin & Ed Maste
reeBSD 12.0 continues the trend in recent
20 FreeBSD Journal
Example: Building 32-bit MIPS World with External GCC: As a result, developers
# pkg install mips-xtoolchain-gcc have now begun to focus
# cd /path/to/src on using (and in some
# make buildworld TARGET_ARCH=mips CROSS_TOOLCHAIN=mips-gcc cases requiring) features
only supported by modern
The base system packages consist of two packages: toolchains.
freebsd-binutils and freebsd-gcc. These A prominent example of this is the use of indirect
packages are built from the base/binutils and functions on x86 kernels. Indirect functions are a tool-
base/gcc ports. Unlike the additional toolchain chain feature that permit a linker to invoke a function
packages, these packages replace components in the when resolving a symbol to determine what address
base system toolchain such as /usr/bin/cc and the symbol should resolve to. This is commonly used
/usr/bin/ld. The ports for these packages (along to provide routines optimized for different processors.
with pkg(8) itself) can be cross-built from a non- For example, a C runtime library might provide ver-
native host. This will permit the Project to provide tool- sions of string functions that use AVX or SSE instruc-
chain packages even on architectures for which the tions and choose the optimal version for the current
Project does not provide full package repositories. CPU. FreeBSD 12.0 kernels for x86 architectures make
Even when using a GNU toolchain, many toolchain use of this feature to provide optimized routines for
components are still provided from other sources. For memory copies, TLB flushes, and FPU state manage-
example, all FreeBSD architectures with a modern ment. FreeBSD amd64 kernels also use indirect func-
toolchain use libc++ from LLVM as the C++ runtime tions to support the Supervisor Mode Access
library. Utilities such as strip(8) and objcopy(8) Prevention (SMAP) security feature. Looking forward,
are provided by the ELF Tool Chain project. the FreeBSD 13 development branch has already
FreeBSD 11 included support for additional tool- begun using indirect functions in the userland C run-
chain packages and CROSS_TOOLCHAIN. During the time library to provide optimized routines for memory
FreeBSD 12 development cycle, work has focused on clearing and memory copies. The use of indirect func-
further refining this support. For example, the support tions will continue to expand in the future in both
for the --sysroot flag has been improved by both userland and the kernel.
patches and configuration changes to the toolchain Conclusion
packages. In addition, the build system was updated FreeBSD 12.0 marks another milestone in toolchain
to be more friendly to external toolchains with development. The ARM and x86 architectures now
changes such as using the compiler driver to link use modern, permissively licensed compilers and link-
binaries whenever possible and supporting different ers. Support for external GCC toolchains is maturing.
MIPS ABIs such as N32. FreeBSD 13 will no longer use GPLv2 bits in the base
The base system toolchain packages have also been system toolchain on any architectures. As a result,
under active development over the past two years. FreeBSD developers will accelerate the adoption of
Support has been added for the MIPS and x86 architec- new toolchain features in the future. This will range
tures. The same fixes for --sysroot support applica- from expanding the use of indirect functions, to
ble to the additional toolchain packages also fixed simi- enabling new features such as link-time optimization
lar issues with the base system packages. While they are (LTO), build identifiers, compressed debug informa-
not yet in a state to replace the legacy GPLv2 toolchain tion, and more. •
for any architectures in FreeBSD 12.0, developers have
been able to build and boot a self-hosted world and JOHN BALDWIN is a systems software developer. He has
kernel on 32-bit MIPS. directly committed changes to the FreeBSD operating system for
Using Modern Toolchain Features 19 years across various parts of the kernel (including x86 plat-
form support, SMP, various device drivers, and the virtual mem-
One of the benefits of moving to modern toolchains is ory subsystem) and userspace programs. In addition to writing
the ability to use new toolchain features in the base sys- code, John has served on the FreeBSD core and release engi-
tem. Much of the work on toolchains prior to FreeBSD neering teams. He has also contributed to the GDB debugger
12 focused on bringing on supporting a permissively- and LLVM. John lives in Concord, California, with his wife,
licensed toolchain on x86 architectures as well as sup- Kimberly, and three children: Janelle, Evan, and Bella.
porting new architectures such as 64-bit ARM. However,
FreeBSD was still treating the legacy GPLv2 toolchain as ED MASTE manages project development for the FreeBSD
the lowest-common-denominator for deciding which Foundation and works in an engineering support role with the
toolchain features the base system used. University of Cambridge Computer Laboratory. He is also a mem-
Toward the end of the FreeBSD 12 development ber of the elected FreeBSD core team. Aside from FreeBSD and
LLVM projects, he is a contributor to several other open-source
cycle this focus has shifted. As LLD has matured,
projects. He lives in Kitchener, Canada, with his wife, Anna, and
FreeBSD has achieved the goal of a permissively- sons, Pieter and Daniel.
licensed toolchain on the ARM and x86 architectures.
Nov/Dec 2018 21
SEE
TEXT
O N LY
letters@
WeGetletters
freebsdjournal.org
by Michael W Lucas
Dear ed(1), thing much like it. Come looking for an interest-
I keep hearing about FreeBSD conferences. ing problem to spend some time on? There’s a
Should I go? whole bunch of people with problems, and some
of them you can solve. And these folks keep up
Sincerely
on technology. You’ll get offered current ideas,
Random Sysadmin along with older ideas that still work really well—
much like FreeBSD itself.
Each time I go to a BSD conference, I leave
Dear Random, with more ideas than I can fail to deploy in the
Yes, you should go. Go, now. Get out of here next year.
and go! Conferences also give you a chance to interact
(time passes) directly with a bunch of FreeBSD developers. You
You’re still here? shouldn’t barge up to every developer in the
Sigh. Fine. Let’s talk about conferences. place with your pet annoyance, but it’s really
You’ll find FreeBSD content at all sorts of tech- easy to ask a question over lunch and see what
nical conferences. Even most Linux events have happens. People code FreeBSD either to a specifi-
one or two FreeBSD talks. Ohio LinuxFest has so cation (for Foundation-sponsored work) or to
much, they might as well declare an official solve their own problems.
FreeBSD track. You’ll find FreeBSD content at A developer might be able to point you at a
commercial conferences like Usenix and at a solution, suggest a different tool, or give you
whole slew of security conferences. some pointers on developing your own solution
But really, you want to hit a dedicated BSD and contributing it back to the community. Don’t
event: BSDCan, AsiaBSDCon, or EuroBSDCon. say you can’t become a committer—at one
The smaller, less regular conferences like point, every single committer was just as clueless
NYCBSDCon, MeetBSD, vBSDCon are all pretty as you.
cool as well. If you speak Portuguese there’s These conferences are important enough to
Brazil’s BSDDay. the FreeBSD community that the FreeBSD
BSDCan is at the University of Ottawa, Foundation sponsors them. A two-day FreeBSD
Canada, usually in June. AsiaBSDCon is at the developer summit precedes each conference.
University of Tokyo, Japan, in March. Committers show up from all over the world to
EuroBSDCon, in September, moves each year— discuss FreeBSD improvements, works in
not only to a different city, but a different coun- progress, and what comes next. Sometimes five
try. I guess Europeans are easily bored? All of minutes with a whiteboard can cut out months
these conferences have massive amounts of of emails. By subscribing to this magazine, you’re
excellent FreeBSD content. You’ll also find all supporting those meetings.
sorts of stuff from other BSDs as well. You can There’s a last reason for going to a BSD con-
binge on FreeBSD. Drown in it. You can even ference, though:
spread FreeBSD on the floor of your hotel room They’re a whole lot of fun.
and roll around in it but do have a shower after- The fun is built into the programming.
wards. Food is fun. EuroBSDCon takes advantage of
While the talks are overwhelmingly excellent the locale to host spectacular dinners. You
(except when they give that Lucas guy a plat- haven’t lived until you’ve eaten spectacular food
form) the part I most appreciate is the hallway in a Parisian catacomb or on the only sand beach
track. When you get together all these really on Malta. Everyone tells me Japan is magnificent,
smart programmers, sysadmins, users, and hang- what with the geeky shopping and amazing
ers-on, it’s like spending a weekend inside the seafood. And poutine is sufficient reason to visit
Pinball Machine of Knowledge. Show up to a Canada. All of these conferences are held in
BSD conference with a problem? You can find places that have a bunch to interest we geeky
people who have solved your problem, or some- sorts, from museums to mints. Bring the family.
22 FreeBSD Journal
They can go out and do cool stuff while you nerd Bringing talented folks to a FreeBSD devsummit, let-
out. ting them meet people and learn about existing
If all that doesn’t sell you on attending a confer- issues they could help with, improves the chances
ence, I can’t help you. that they’ll pitch in to help FreeBSD.
Pick a conference on your continent and go!
Dear ed(1),
Okay, I want to go. But I can’t afford it. Dear ed(1),
Any suggestions? There’s no conference on my continent.
—Random Sysadmin —RS
Write
For Us! Contact Jim Maurer
(jmaurer@freebsdjournal.com)
with your article ideas.
Nov/Dec 2018 23
SEE
TEXT
O N LY
®
24 FreeBSD Journal
information on networks, subnets, and static routes, problem and he would look into porting it to see if it
which are a prerequisite to understanding BGP and fits his workflow. I think it will, as I use it every day
its important role on the Internet. BGP stands for and it does a good job of removing a lot of the com-
Border Gateway Protocol and it was designed to plexity associated with media management as well as
allow AS routers—known as autonomous systems— providing a stable interface for working with media.
to share routing tables. It basically allows computers In the talk by Ingo Schwarze on some of the
on the Internet to find each other in order to con- headaches related to working with Markdown,
nect. I had a great time learning about BGP, sharing he asked for help with a CSS issue in the main
routing tables with other participants and learning mandoc.css file that controls the presentation of
about the importance of filtering the routes man pages on the web for OpenBSD. I'm familiar
announced by peers. Hessler asked participants to with CSS and offered to take a look at the issue to
share routes and then demonstrated what can hap- see if I could help. The moment I saw the code, I
pen when you don’t use proper filtering. A peer can recognized the issue and knew what it would take
just announce every address—yes, every single to make the fix. I just needed to find the time to sit
address—and make the processing on your machine down and write the code. Ingo assured me that I
go crazy. Everyone was pleasantly entertained and could take my time and contact him at my leisure.
had a great time. I highly recommend this tutorial to Not only are there great opportunities to take in
anyone wanting to learn about BGP. information, but also opportunities to be a part of
After the all-day tutorial, I headed over to the the public conversation.
Devsummit dinner. I sat next to Marshall Kirk The next talk was “What’s in Store for NetBSD
McKusick and ended up talking to him all night. In 9.0,” in which Sevan Janiyan shared the milestones
fact, his husband actually told me at the end of the reached in the NetBSD project, including details
night that I had monopolized Kirk, and so I was about what will be in the next release. I highly rec-
responsible for making sure he made it back to the ommend checking out his slides. All conference
hotel safely. Oops! I guess I deserved that. I couldn’t slides are available for download via
help it! Sitting next to this revolutionary FreeBSD the EuroBSDCon.org website. The second keynote,
committer was my opportunity to ask a lot of ques- “Some Computing and Networking Historical
tions. What an experience that was. Perspectives,” by Ron Broersma, was a real treat for
all in the room. Ron brought along various historical
SATURDAY artifacts from his early career, including his work on
The morning announcements were given by Michai Arpanet. The entire auditorium loved the talk and
Stanek and then he introduced his colleague, Costin you could feel the interest growing in the room
Raiciu, from the Politehnica University of Bucharest. each minute. Ron would periodically pull out an
Costin gave the morning keynote on “Lightweight artifact from under the podium—for example, an
Virtualization.” He presented some interesting old boot card that used to come with the big, mil-
research on the topic and an example of a very tiny lion-dollar computers in use at the time. You actual-
Python environment running on one of the ly needed to use the card to know where in memo-
machines. ry to set the program to be able to boot the thing.
Next, “Hacking Together a FreeBSD Presentation You had to manually type in the codes to get every-
Streaming Box for as Little as Possible,” by Tom thing lined up so that boot would function proper-
Jones, provided some useful examples on how one ly! After the talk, Ron welcomed people to come up
might record and stream BSD conference slides along and look at the items he had brought with him.
with the speaker’s voice. The conference was
designed to provide a lot of varied information in a SUNDAY
short time, and so I was then on to Kristaps Dz’s On the last day of the conference, I was running a
“OpenBSD and Diving.” Not only humorous and full little late but hurried over to the venue in time to
of great pictures, this presentation was also full of catch “Pledge and Unveil in OpenBSD,” by Bob
great facts on the use of OpenBSD for managing and Beck. Bob understands the guts of operating system
editing media. All the media transfer, editing, and development and presented some best practices
color correcting was done on OpenBSD. Kristaps and various ways to write safe programs on
asked if anyone had suggestions for good media OpenBSD with pledge and unveil.
management applications, and I recommended Plex. I was totally lost at “Integrate libFuzzer with the
I told him I’d only seen it available in the FreeBSD NetBSD Userland,” a very advanced and technical
ports collection, but he assured me that was not a talk by Yang Zheng. Perhaps one day I will get to
Nov/Dec 2018 25
conference report • EuroBSDcon 2018
the point where I can grok this topic. Despite my nervous reaction to public speaking, the
“DeforaOS, NetBSD, Future Internet,” by Pierre talk went quite well. The audience was engaged
Pronchery, was a very intellectual talk. Pierre is a and asked questions. I shared information on my
very thoughtful developer who pushes the limits experiences with BSD technologies and how teach-
and has a fantastic understanding of the underlying ing young scientists about the technology is a great
technology. I learned a lot about why things are the experience. Learning about all the cool things you
way they are and was given a challenge to make can do with the BSDs and about the amazing com-
things better. munity is an experience I enjoy bringing to others. I
“Debugging Lessons Learned as a Newbie Fixing had some great feedback from Allan Jude, Kirk
NetBSD,” by Maya Rashish, followed a newbie McKusick, and others. Although I was initially a lit-
through the process of contributing to NetBSD tle concerned about the 45-minute block I needed
and detailed some of the things a person new to to fill, once I began talking, my passion took over
the Project should be aware of. After Maya’s talk, I and I was able to fill the time slot with no problem.
caught the tail end of “FreeBSD Graphics,” by There were 181 attendees from 37 countries.
Niclas Zeising, where I asked a question about a Groff was brought to the conference by Sevan
hiccough in the process of compiling drm-stable- Janiyan and put in the care of Deb Goodkin at the
kmod. There is an ignore flag in the Makefile that closing session. •
needs to be removed if one wants to compile the
port on FreeBSD 11.2-RELEASE, which is what I’m
using. Niclas said it must have been an oversight
and that it should work fine on 11.2. I let him Roller Angel is an avid BSD user who enjoys all
know I removed the ignore flag from the Make- the amazing things that can be done with BSD
file and compiled drm-stable-kmod on my 11.2- technology. He has taught programming work-
RELEASE laptop and it has been working like a shops based on FreeBSD and is working on
charm. building an online training platform for teaching
I was up next with my talk, “Being a BSD User.” BSD and related technologies. See BSD.pw for
Minutes before I started, the adrenaline kicked in. more information.
26 FreeBSD Journal
SEE
TEXT
O N LY
new faces
of FreeBSD BY DRU LAVIGNE
Tell us a bit about yourself, your background, How did you first learn about FreeBSD and what
and your interests. about FreeBSD interested you?
• Sergey: My name is Sergey Kozlov • Sergey: When I was a beginner sysadmin, I
and I'm a FreeBSD addict. I was born worked mostly on Windows Server, but wanted to
and spent most of my life in the learn UNIX-like operating systems, as I had heard
Ukraine, Kyiv, but since 2015, I've so much about them. I started with the most popu-
been living in Gdansk, Poland. I'm a lar, Linux, and I was trying to learn it for quite a
Python programmer, sysadmin, and while. I installed different distributions, versions,
tester. I’m working at Intel on auto- flavors, all the while looking on the Internet for
matic testing of Intel’s FreeBSD wired books and articles related to Linux but couldn't
Ethernet drivers (em, igb, ix, ixl, etc.). learn a thing. I just felt that I hadn’t started at the
• Vinícius: I was born and raised in right place and didn’t see a clear path for master-
Fortaleza, with roots in Rio de Janeiro ing my Linux skills. I was very surprised that all
and Minas Gerais. My family’s name those operating systems were even called Linux,
comes originally from Italy and was because I could hardly find anything common in
misspelled during the Italian migration them.
to Brazil, so I'm also the inheritor of Later, I heard about FreeBSD from one of my
the tasty tradition of dipping bread in friends. He said that somewhere out there, in the
coffee. The original family’s name is land of heavily loaded web servers and routers,
Zavani. Now I’m living in Germany there’s a caste of old sysadmins with long beards
(and “crazy” homesick, especially in winter) work- who prefer tidiness and order. Those guys know
ing for cleverbridge, which employs a massive use everything about how their tools work and the OS
of FreeBSD. they’re using is called FreeBSD. And that’s exactly
As a kid, I got into computers after encounter- what I was looking for! I downloaded the ISO and
ing a TK 85 and later graduated in computer engi- got a copy of Michael W Lucas’s Absolute FreeBSD
neering. I enjoy learning different languages. I’m and that’s where the world of UNIX opened to
engaged and motivated by the use and benefits of me. One thing at a time, step by step, I was get-
free and open-source software (FOSS), especially ting more and more knowledge about what I was
the ones under the BSD license, and have made really doing and how the whole thing worked
contributions and donations to different projects “under the hood.” This feeling of order and clarity
over time. In the last few years I have been work- is what got me hooked on FreeBSD.
ing to get more privacy-enhancing technologies • Vinícius: My first steps into FreeBSD were
(PET) support working on *BSD operating systems. about 15 years ago while I was working for an ISP
These efforts brought me to the TorBSD Diversity in my hometown. The person who introduced me
Project (TDP, https://torbsd.org/2016/12/17/wel- to FreeBSD was a good friend I met on the
come-aboard-vinicius.html) where I became a core BRASnet Internet Relay Chat (IRC) network. Sadly,
team member (https://www.torproject.org/ I was the only one at the ISP who really got into
about/corepeople.html.en#egypcio). *BSD and I was even running it on my worksta-
tions. After moving forward, I didn't abandon
28 FreeBSD Journal
Beastie and kept using FreeBSD for everything I • Vinícius: Well, time flies very fast! It seems like
could, even if others said I was just wasting my only yesterday when I ran sysinstall for the first
time—which I heard a lot. time and upgraded my machines using cvsup,
It’s still not easy for me to describe exactly what when I was subscribed to the FreeBSD Users Group
brought me to FreeBSD. I cannot just pick one sin- Brazil mailing list. At that time, I was giving talks,
gle feature and say it was that. But there wasn't a facilitating tutorials, writing articles about FreeBSD,
single time that running it on my laptop or on any and getting people familiar with its features and
server got me mad or disappointed. The OS is rock what it’s capable of. Sometime in between, I
solid, the Project and its documentation are well engaged in a great effort to translate the FreeBSD
maintained, the developers care about the commu- Handbook to Portuguese, and then I helped review
nity and are in touch to get new features and/or and update a book chapter about network and
hardware support working. And I like the license, Internet services powered by FreeBSD (FreeBSD: O
the great companies that adopt FreeBSD, the Poder dos Servidores em Suas Mãos). One day in
research/products/solutions it empowers, the stan- 2015, I found myself organizing the very first inter-
dards it follows, the certifications it's compliant to. national BSD conference in Brazil!
When FreeBSD landed on hardware like Finally, a couple months ago I attended
RaspberryPi and BeagleBone, I got very little sleep EuroBSDcon 2018, and there I met some other
but had lots of fun. I got familiar with Poudriere committers. During one of the coffee breaks and
and presented it together with a cross-compiling an off-the-record chat about PET support and alter-
setup to the Institute of Technology (where I had natives to freebsd-update and what's going on
graduated) during one of the Compilers 101 class- regarding pkg-base and bhyve development,
es. At the very same Institute (Instituto Federal de araujo@, beat@, and rene@ decided to punish me
Educação, Ciência e Tecnologia do Ceará—IFCE), I with a commit bit.
talked about KAME/IPv6 on FreeBSD and recom-
mended the net/sysadmins use it to handle packet How has your experience been since joining the
filtering. At the lab where I did my research (things FreeBSD Project? Do you have any advice for
like playing with Docker+Jails or developing an readers who may be interested in also becoming
embedded system), I was also the net/sysadmin. a FreeBSD committer?
Most of the servers were running FreeBSD, and one • Sergey: Before joining, I was eagerly searching
of the official FreeBSD Brazilian mirrors was run- for any news about the Project. I checked out every
ning at that lab (it was the only one in Brazil serv- change on the wiki and looked forward to every
ing the /snapshots folder, by the way). quarterly status report. All the people working on
the system were nothing less than rock stars for
How did you end up becoming a committer? me. And then, all of a sudden, I get my commit bit
• Sergey: It all started when I found two pieces of and go to DevSummit 2018 in Bucharest. I’m sit-
software: p910nd and mjpg-streamer. I needed ting in the same room with all those stars, dis-
them, but I was forced to use Linux because they cussing the future of the Project and being treated
weren't available in the ports collection. This made like one of their own. I just couldn't believe it. In
me angry—“Why can they use it and we can’t? addition to all the excitement I had, lots of people
How hard could it be to make such a small app wanted to help with my “new committer steps,”
work on FreeBSD?” This is when I decided to make like setting up mail, adding myself to docs, etc.;
my own port. I made several—every time someone des@ even accidentally reset his password while
at my workplace approached me and said, “Hey, helping me with SVN! It just really felt like a family.
we need <app_name>, but there’s no version for For anyone who wants to become a committer, I
FreeBSD on the website,” I would just smile and can suggest three things: work, socialize, behave.
say, “Don't worry, I’ll port it by tomorrow.” First of all, make sure you’re contributing to the
After a while I joined Intel where I was able to Project on a regular basis. It may not be every day
(finally!) make working on ports part of my job or week, but it has to be regular. Also, make sure
duties. Many Phabricator reviews later, sbruno@ to join one of the IRC channels and introduce your-
apparently got tired of committing my code and self, start a conversation with someone. To be hon-
decided that it was time for me to start breaking est, I wasn’t doing that, as I was too shy, and only
stuff myself. after I joined the Project, did I realize how valuable
Nov/Dec 2018 29
new faces of FreeBSD
this can be. And last but not least, respect people tions or embedded devices.
around you, their work, time, and opinion. Make Before I officially got the title of ports committer,
sure you fit into the family. I was already maintaining a few ports and adopting
• Vinícius: I have gotten great input from many others slowly. It is a great journey and a lot of fun!
developers involved with the Project and learned What else can I say to others wanting to con-
more about FreeBSD internals over these last tribute? Just do your thing and have fun doing it.
months—and that's amazing! There is really good Contribute small changes, submit your patches, get
stuff going on. In the past I just chatted, mostly stuff done. It’s not a race, and it may take time.
with Brazilians, on IRC, and shared thoughts or Chill. Help others and you will also help yourself.
experiences regarding all kinds of scenarios where
FreeBSD is involved. Now, I can make a direct
impact and contribute to solving other users’ or DRU LAVIGNE is a FreeBSD doc committer and
even other developers’ issues—especially when we the author of BSD Hacks and The Best of FreeBSD
all use it every single day on servers and worksta- Basics.
“Congratulations to the RE team and all the contributors for the release
of 12.0! We’d also like to thank our community of donors whose
support helps us fund a full-time staff member dedicated to leading the
Release Engineering team and overseeing the release process.”
30 FreeBSD Journal
JAN., FEB., & MARCH BY DRU LAVIGNE
Nov/Dec 2018 31
SEE
TEXT
O N LY
svn UPDATE
by Steven Kreuzer
Very recently something exciting happened to me. One night I ran svn up in /usr/src,
kicked off a build, and went to bed. When I woke up in the morning, I found my laptop
sitting at a prompt asking me to log in to my FreeBSD 13.0 workstation. That can only
mean one thing. We are currently in the middle of the FreeBSD 12.0 release cycle!
For a while there was not much action in HEAD, as the release engineering team had
frozen the tree to create a branch of what will become 12-STABLE. That freeze has been
lifted and we once again are seeing lots of new features and exciting changes. Can you
think of a better way to end the year?
Add macros for reading performance counter Add new rc keywords: enable, disable,
CSRs on RISC-V— https://svnweb.freebsd.org/ delete— https://svnweb.freebsd.org/
changeset/base/340399 changeset/base/339971
32 FreeBSD Journal
Allow configuration of several ipsec interfaces and installed along with the kernel.
with the same tunnel endpoints — https:// This means that a port that installs a kernel module
svnweb.freebsd.org/changeset/base/340477 can choose to install its source along with a suitable
modules can be built along with a kernel by defining STEVEN KREUZER is a FreeBSD Developer and
LOCAL_MODULES to the list of modules. Each is Unix Systems Administrator with an interest in
assumed to be a subdirectory containing a valid retro-computing and air-cooled Volkswagens.
Makefile. If LOCAL_MODULES is not specified, all of the He lives in Queens, New York, with his wife,
modules present in LOCALBASE/sys/modules are built daughter, and dog.
Already involved?
Dont forget to check out the latest
grant opportunities at
freebsdfoundation.org
Nov/Dec 2018 33