Home - Waterfall Grid T-Grid Console Builders Recent Builds Buildslaves Changesources - JSON API - About

Console View


Tags: Architectures Platforms default
Legend:   Passed Failed Warnings Failed Again Running Exception Offline No data

Architectures Platforms default
Ryan Moeller
arc_summary: Make get_descriptions per platform

Linux uses modinfo to get tunables descriptions, FreeBSD has to use
sysctl.

Move the existing function definition so it is defined that way on
Linux, and add a definition in terms of sysctl for FreeBSD.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Ryan Moeller <ryan@iXsystems.com>
Closes #10062
Ryan Moeller
pyzfs: Add constants for platform-specific errnos

FreeBSD doesn't have EBADE, ECHRNG, or ETIME.

Add constants for these and set them appropriately for the platform.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Ryan Moeller <ryan@iXsystems.com>
Closes #10061
Matthew Macy
Consolidate arc_buf allocation checks

The following check currently occurs in three separate locations
in dbuf.c.  This change consolidates those checks in to the
dbuf_alloc_arcbuf_from_arcbuf() function.

if (arc_is_encrypted(data)) {
...
} else if (compress_type != ZIO_COMPRESS_OFF) {
...
} else {
...
}

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Matt Macy <mmacy@FreeBSD.org>
Closes #10057
Matthew Macy
Don't open zfs control device exclusively

With the FreeBSD platform changes that were made for #10073
it is no longer necessary on FreeBSD to open the control device
exclusively to get onexit callbacks invoked.

Signed-off-by: Matt Macy <mmacy@FreeBSD.org>

Pull-request: #10076 part 1/1
Matthew Macy
Don't open zfs control device exclusively

With the FreeBSD platform changes that were made for #10073
it is no longer necessary on FreeBSD to open the control device
exclusively to get onexit callbacks invoked.

Signed-off-by: Matt Macy <mmacy@FreeBSD.org>

Pull-request: #10076 part 1/1
Matthew Macy
Don't call zrele on passed zp in zfs_xattr_owner_unlinked on FreeBSD

FreeBSD has a somewhat more cumbersome locking and refcounting
protocol for the platform counterpart to znode. We need to not call
zrele on the passed zp, but do need to do so on any intermediate zp.

Signed-off-by: Matt Macy <mmacy@FreeBSD.org>

Pull-request: #10075 part 1/1
Matthew Macy
Don't call zrele on passed zp in zfs_xattr_owner_unlinked on FreeBSD

FreeBSD has a somewhat more cumbersome locking and refcounting
protocol for the platform counterpart to znode. We need to not call
zrele on the passed zp, but do need to do so on any intermediate zp.

Signed-off-by: Matt Macy <mmacy@FreeBSD.org>

Pull-request: #10075 part 1/1
Matthew Macy
Re-share zfsdev_getminor and zfs_onexit_fd_hold

By adding a zfs_file_private accessor to the common
interfaces and some extensions to FreeBSD platform
code it is now possible to share the implementations
for the aforementioned functions.

Signed-off-by: Matt Macy <mmacy@FreeBSD.org>

Pull-request: #10073 part 1/1
Brian Behlendorf
Linux 5.5 compat: blkg_tryget()

Commit https://github.com/torvalds/linux/commit/9e8d42a0f accidentally
converted the static inline function blkg_tryget() to GPL-only for
kernels built with CONFIG_PREEMPT_RCU=y and CONFIG_BLK_CGROUP=y.

Resolve the build issue by providing our own equivilant functionality
when needed which uses rcu_read_lock_sched() internally as before.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue #9745

Pull-request: #10072 part 1/1
Brian Behlendorf
Linux 5.4 compat: blkg_tryget()

Commit https://github.com/torvalds/linux/commit/9e8d42a0f accidentally
converted the static inline function blkg_tryget() to GPL-only for
kernels built with CONFIG_PREEMPT_RCU=y and CONFIG_BLK_CGROUP=y.

Resolve the build issue by providing our own equivilant functionality
when needed which uses rcu_read_lock_sched() internally as before.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue #9745

Pull-request: #10072 part 1/1
  • Amazon 2 x86_64 (BUILD): cloning zfs -  stdio
  • Debian 8 arm (BUILD): cloning zfs -  stdio
  • Debian 8 ppc64 (BUILD): cloning zfs -  stdio
  • Debian 8 ppc (BUILD): cloning zfs -  stdio
  • Kernel.org Built-in x86_64 (BUILD): cloning zfs -  stdio
  • Ubuntu 18.04 x86_64 (STYLE): cloning zfs -  stdio
John Gallagher
Add trim support to zpool wait

Manual trims fall into the category of long-running pool activities
which people might want to wait synchronously for. This change adds
support to 'zpool wait' for waiting for manual trim operations to
complete. It also adds a '-w' flag to 'zpool trim' which can be used to
turn 'zpool trim' into a synchronous operation.

Signed-off-by: John Gallagher <john.gallagher@delphix.com>

Pull-request: #10071 part 1/1
John Gallagher
Add trim support to zpool wait

Manual trims fall into the category of long-running pool activities
which people might want to wait synchronously for. This change adds
support to 'zpool wait' for waiting for manual trim operations to
complete. It also adds a '-w' flag to 'zpool trim' which can be used to
turn 'zpool trim' into a synchronous operation.

Signed-off-by: John Gallagher <john.gallagher@delphix.com>

Pull-request: #10071 part 1/1
John Gallagher
Add trim support to zpool wait

Manual trims fall into the category of long-running pool activities
which people might want to wait synchronously for. This change adds
support to 'zpool wait' for waiting for manual trim operations to
complete. It also adds a '-w' flag to 'zpool trim' which can be used to
turn 'zpool trim' into a synchronous operation.

Signed-off-by: John Gallagher <john.gallagher@delphix.com>

Pull-request: #10071 part 1/1
Matthew Ahrens
Improve performance of zio_taskq_member

__zio_execute() calls zio_taskq_member() to determine if we are running
in a zio interrupt taskq, in which case we may need to switch to
processing this zio in a zio issue taskq.  The call to
zio_taskq_member() can become a performance bottleneck when we are
processing a high rate of zio's.

zio_taskq_member() calls taskq_member() on each of the zio interrupt
taskqs, of which there are 21.  This is slow because each call to
taskq_member() does tsd_get(taskq_tsd), which on Linux is relatively
slow.

This commit improves the performance of zio_taskq_member() by having it
cache the value of tsd_get(taskq_tsd), reducing the number of those
calls to 1/21th of the current behavior.

In a test case running `zfs send -c >/dev/null` of a filesystem with
small blocks (average 2.5KB/block), zio_taskq_member() was using 6.7% of
one CPU, and with this change it is reduced to 1.3%.  Overall time to
perform the `zfs send` reduced by 10% (~150,000 block/sec to ~165,000
blocks/sec).

Signed-off-by: Matthew Ahrens <mahrens@delphix.com>

Pull-request: #10070 part 1/1
Brian
Multi ABD Type

Adding the mutli abd type, which allows for
linear and scatter abd's to be chained together
into a single abd.

Signed-off-by: Brian <batkinson.lanl.gov>
Authored-by: Mark Maybee <mmaybee@cray.com>

Pull-request: #10069 part 1/1
Brian
Multi ABD Type

Adding the mutli abd type, which allows for
linear and scatter abd's to be chained together
into a single abd.

Signed-off-by: Brian <batkinson.lanl.gov>
Authored-by: Mark Maybee <mmaybee@cray.com>

Just fixing inline problem...

Pull-request: #10069 part 1/1
Brian
Multi ABD Type

Adding the mutli abd type allows for linear and
scatter abd's to be chained together into a
single abd.

Signed-off-by: Brian <bwa@clemson.edu>
Authored-by: Mark Maybee <mmaybee@cray.com>

Pull-request: #10069 part 1/1
Ryan Moeller
ZTS: Misc fixes for FreeBSD

* Set geom debug flags in corrupt_blocks_at_level
* Use the right time zone for history tests
* Add missing commands.cfg entry for diskinfo
* Rewrite get_last_txg_synced to use zdb
* Don't check ulimits for sparse files
* Suspend removal before removing a vdev, not after

Reviewed-by: John Kennedy <john.kennedy@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Ryan Moeller <ryan@iXsystems.com>
Closes #10054
Matthew Ahrens
ZTS: Fix zfs_receive_004_neg

`zfs recv` of an incremental stream that already exists is ignored, with
a message like:

    receiving incremental stream of pool/fs@incsnap into pool/fs@incsnap
    snap testpool/testfs@incsnap already exists; ignoring

And the command exits successfully (exit code 0).

The zfs_receive_004_neg test is expecting that a this case will fail,
with nonzero exit code.

The fix is to remove this specific command from the test case.  This
lets us check that the remaining commands do in fact fail.

Reviewed-by: Ryan Moeller <ryan@iXsystems.com>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Matthew Ahrens <mahrens@delphix.com>
Closes #10055
Brian Behlendorf
Linux 5.6 compat: time_t

As part of the Linux kernel's y2038 changes the time_t type has been
fully retired.  Callers are now required to use the time64_t type.

Rather than move to the new type, I've removed the few remaining
places where a time_t is used in the kernel code.  They've been
replaced with a uint64_t which is already how ZFS internally
handled these values.

Going forward we should work towards updating the remaining user
space time_t consumers to the 64-bit interfaces.

Reviewed-by: Matthew Macy <mmacy@freebsd.org>
Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #10052
Closes #10064
Brian Behlendorf
Linux 5.6 compat: ktime_get_raw_ts64()

The getrawmonotonic() and getrawmonotonic64() interfaces have been
fully retired.  Update gethrtime() to use the replacement interface
ktime_get_raw_ts64() which was introduced in the 4.18 kernel.

Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #10052
Closes #10064
Matthew Ahrens
Improve zfs send performance by bypassing the ARC

When doing a zfs send on a dataset with small recordsize (e.g. 8K),
performance is dominated by the per-block overheads.  This is especially
true with `zfs send --compressed`, which further reduces the amount of
data sent, for the same number of blocks.  Several threads are involved,
but the limiting factor is the `send_prefetch` thread, which is 100% on
CPU.

The main job of the `send_prefetch` thread is to issue zio's for the
data that will be needed by the main thread.  It does this by calling
`arc_read(ARC_FLAG_PREFETCH)`.  This has an immediate cost of creating
an arc_hdr, which takes around 14% of one CPU.  It also induces later
costs by other threads:

* Since the data was only prefetched, dmu_send()->dmu_dump_write() will
  need to call arc_read() again to get the data.  This will have to
  look up the arc_hdr in the hash table and copy the data from the
  scatter ABD in the arc_hdr to a linear ABD in arc_buf.  This takes
  27% of one CPU.

* dmu_dump_write() needs to arc_buf_destroy()  This takes 11% of one CPU.

* arc_adjust() will need to evict this arc_hdr, taking about 50% of one
  CPU.

All of these costs can be avoided by bypassing the ARC if the data is
not already cached.  This commit changes `zfs send` to check for the
data in the ARC, and if it is not found then we directly call
`zio_read()`, reading the data into a linear ABD which is used by
dmu_dump_write() directly.

The performance improvement is best expressed in terms of how many
blocks can be processed by `zfs send` in one second.  This change
increases the metric by 50%, from ~100,000 to ~150,000.  When the amount
of data per block is small (e.g. 2KB), there is a corresponding
reduction in the elapsed time of `zfs send >/dev/null` (from 86 minutes
to 58 minutes in this test case).

In addition to improving the performance of `zfs send`, this change
makes `zfs send` not pollute the ARC cache.  In most cases the data will
not be reused, so this allows us to keep caching useful data in the MRU
(hit-once) part of the ARC.

Signed-off-by: Matthew Ahrens <mahrens@delphix.com>

Pull-request: #10067 part 1/1
Matthew Macy
Refactor dnode dirty context from dbuf_dirty

* Add dedicated donde_set_dirtyctx routine.
* Add empty dirty record on destroy assertion.
* Make much more extensive use of the SET_ERROR macro.

Reviewed-by: Will Andrews <wca@FreeBSD.org>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Matthew Ahrens <mahrens@delphix.com>
Signed-off-by: Matt Macy <mmacy@FreeBSD.org>
Closes #9924
Ryan Moeller
ZTS: Fix zfs_copies_002_pos

The function `get_used_prop` does not exist.

Use `get_prop used` instead.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: John Kennedy <john.kennedy@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Ryan Moeller <ryan@iXsystems.com>
Closes #10059
Brian Behlendorf
Linux 5.6 compat: time_t

As part of the Linux kernel's y2038 changes the time_t type has been
fully retired.  Callers are now required to use the time64_t type.

Rather than move to the new type, I've removed the few remaining
places where a time_t is used in the kernel code.  They've been
replaced with a uint64_t which is already how ZFS internally
handled these values.

Going forward we should work towards updating the remaining user
space time_t consumers to the 64-bit interfaces.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>

Pull-request: #10064 part 2/2
Brian Behlendorf
Linux 5.6 compat: ktime_get_raw_ts64()

The getrawmonotonic() and getrawmonotonic64() interfaces have been
fully retired.  Update gethrtime() to use the replacement interface
ktime_get_raw_ts64() which was introduced in the 4.18 kernel.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>

Pull-request: #10064 part 1/2
Brian Behlendorf
Fix CONFIG_MODULES=no Linux kernel config

When configuring as builtin (--enable-linux-builtin) for kernels
without loadable module support (CONFIG_MODULES=n) only the object
file is created.  Never a loadable kmod.

Update ZFS_LINUX_TRY_COMPILE to handle this in a manor similar to
the ZFS_LINUX_TEST_COMPILE_ALL macro.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue #9887

Pull-request: #10063 part 1/1
Matthew Macy
Consolidate arc_buf allocation checks

The following check currently occurs in three
separate locations in dbuf.c. This change
consolidates those checks in to dbuf_alloc_arcbuf

if (arc_is_encrypted(data)) {
...
} else if (compress_type != ZIO_COMPRESS_OFF) {
...
} else {
...
}

Signed-off-by: Matt Macy <mmacy@FreeBSD.org>

Pull-request: #10057 part 1/1
Ryan Moeller
ZTS: Fixup shebang in rsend_016, add to common.run

All other ksh scripts use /bin/ksh in the shebang.

Make rsend_016_neg consistent with the rest of the suite.

The test also was absent from any runfiles. Add it to common.run.

Signed-off-by: Ryan Moeller <ryan@iXsystems.com>

Pull-request: #10051 part 1/1
Brian
WIP Direct IO ZoL

This current state of adding Direct IO
support to ZFS on Linux rebased on ZoL master.

The current work still remaining is:
1. Handle issues related to Direct IO
  requests for dbuf's with multiple holds.
2. Create ZTS tests
3. Further debugging

At the moment, tests have been run using FIO and
XDD to resolve all failed VERIFY and ASSERT statements.

Signed-off-by: Brian <batkinson@lanl.gov>
Co-authored-by: Mark Maybee <mmaybee@cray.com>
Co-authored-by: Brian Atkinson <batkinson@lanl.gov>

Pull-request: #10018 part 1/1
Brian
WIP Direct IO ZoL

This current state of adding Direct IO
support to ZFS on Linux rebased on ZoL master.

The current work still remaining is:
1. Handle issues related to Direct IO
  requests for dbuf's with multiple holds.
2. Create ZTS tests
3. Further debugging

At the moment, tests have been run using FIO and
XDD to resolve all failed VERIFY and ASSERT statements.

Signed-off-by: Brian <batkinson@lanl.gov>
Co-authored-by: Mark Maybee <mmaybee@cray.com>
Co-authored-by: Brian Atkinson <batkinson@lanl.gov>

Pull-request: #10018 part 1/1
Brian
WIP Direct IO ZoL

This current state of adding Direct IO
support to ZFS on Linux rebased on ZoL master.

The current work still remaining is:
1. Handle issues related to Direct IO
  requests for dbuf's with multiple holds.
2. Create ZTS tests
3. Further debugging

At the moment, tests have been run using FIO and
XDD to resolve all failed VERIFY and ASSERT statements.

Signed-off-by: Brian <batkinson@lanl.gov>
Co-authored-by: Mark Maybee <mmaybee@cray.com>
Co-authored-by: Brian Atkinson <batkinson@lanl.gov>

Pull-request: #10018 part 1/1
Matthew Macy
Avoid COW faults on DB fills

This is the final piece of COW fault avoidance. We track
partial buffer fills and skip the read from disk if the whole
buffer is overwritten before the txg happens.

async COW part 6

Adapted from work in 2011 by Will Andrews at SpectraLogic.

Signed-off-by: Matt Macy <mmacy@FreeBSD.org>
Authored-by: Will Andrews <wca@FreeBSD.org>

Pull-request: #9954 part 2/2
Matthew Macy
refactor body of dbuf_free_range in to multiple functions

async COW part 5

Signed-off-by: Matt Macy <mmacy@FreeBSD.org>

Pull-request: #9936 part 1/1
George Amanakis
Persistent L2ARC

This commit makes the L2ARC persistent across reboots. It is largely
based on issue 3525 in Illumos. This feature implements a light-weight
persistent L2ARC metadata structure that allows L2ARC contents to be
recovered after a reboot. This significantly eases the impact a reboot
has on read performance on systems with large caches.

Co-authored-by: Saso Kiselkov <skiselkov@gmail.com>
Co-authored-by: Jorgen Lundman <lundman@lundman.net>
Co-authored-by: George Amanakis <gamanakis@gmail.com>
Ported-by: Yuxuan Shui <yshuiv7@gmail.com>
Signed-off-by: George Amanakis <gamanakis@gmail.com>

Pull-request: #9582 part 1/1
George Amanakis
Persistent L2ARC

This commit makes the L2ARC persistent across reboots. It is largely
based on issue 3525 in Illumos. This feature implements a light-weight
persistent L2ARC metadata structure that allows L2ARC contents to be
recovered after a reboot. This significantly eases the impact a reboot
has on read performance on systems with large caches.

Co-authored-by: Saso Kiselkov <skiselkov@gmail.com>
Co-authored-by: Jorgen Lundman <lundman@lundman.net>
Co-authored-by: George Amanakis <gamanakis@gmail.com>
Ported-by: Yuxuan Shui <yshuiv7@gmail.com>
Signed-off-by: George Amanakis <gamanakis@gmail.com>

Pull-request: #9582 part 1/1
George Amanakis
Persistent L2ARC

This commit makes the L2ARC persistent across reboots. It is largely
based on issue 3525 in Illumos. This feature implements a light-weight
persistent L2ARC metadata structure that allows L2ARC contents to be
recovered after a reboot. This significantly eases the impact a reboot
has on read performance on systems with large caches.

Co-authored-by: Saso Kiselkov <skiselkov@gmail.com>
Co-authored-by: Jorgen Lundman <lundman@lundman.net>
Co-authored-by: George Amanakis <gamanakis@gmail.com>
Ported-by: Yuxuan Shui <yshuiv7@gmail.com>
Signed-off-by: George Amanakis <gamanakis@gmail.com>

Pull-request: #9582 part 1/1
George Amanakis
Persistent L2ARC

This commit makes the L2ARC persistent across reboots. It is largely
based on issue 3525 in Illumos. This feature implements a light-weight
persistent L2ARC metadata structure that allows L2ARC contents to be
recovered after a reboot. This significantly eases the impact a reboot
has on read performance on systems with large caches.

Co-authored-by: Saso Kiselkov <skiselkov@gmail.com>
Co-authored-by: Jorgen Lundman <lundman@lundman.net>
Co-authored-by: George Amanakis <gamanakis@gmail.com>
Ported-by: Yuxuan Shui <yshuiv7@gmail.com>
Signed-off-by: George Amanakis <gamanakis@gmail.com>

Pull-request: #9582 part 1/1
Ryan Moeller
Add FreeBSD support to ZoL

- Import FreeBSD specific parts

Signed-off-by: Matt Macy <mmacy@FreeBSD.org>
Co-authored-by: Ryan Moeller <ryan@iXsystems.com>
Signed-off-by: Ryan Moeller <ryan@iXsystems.com>

Requires-builders: amazon2, debian10, freebsd12, freebsd13, style

TEST_ZTEST_SKIP="yes"

Pull-request: #8987 part 1/1