Home - Waterfall Grid T-Grid Console Builders Recent Builds Buildslaves Changesources - JSON API - About

Console View


Tags: Architectures Platforms default
Legend:   Passed Failed Warnings Failed Again Running Exception Offline No data

Architectures Platforms default
Ryan Moeller
Let default arc_c_max be platform dependent

Linux changed the default max ARC size to 1/2 of physical memory to
deal with shortcomings of the Linux SLUB allocator.  Other platforms
do not require the same logic.

Implement an arc_default_max() function to determine a default max ARC
size in platform code.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Ryan Moeller <ryan@iXsystems.com>
Closes #10155
Matthew Ahrens
Compile cityhash code into libzfs

Make the cityhash code compile into libzfs, in preparation for the new
"zstream" command.

Reviewed-by: Paul Dagnelie <pcd@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Matthew Ahrens <mahrens@delphix.com>
Closes #10152
Ryan Moeller
Don't ignore zfs_arc_max below allmem/32

Fixes #10157

Set arc_c_min before arc_c_max so that when zfs_arc_min is set lower
than the default allmem/32 zfs_arc_max can also be set lower.

Add warning messages when tunables are being ignored.

Signed-off-by: Ryan Moeller <ryan@iXsystems.com>

Pull-request: #10158 part 1/1
Ryan Moeller
Don't ignore zfs_arc_max below allmem/32

Fixes #10157

Set arc_c_min before arc_c_max so that when zfs_arc_min is set lower
than the default allmem/32 zfs_arc_max can also be set lower.

Add warning messages when tunables are being ignored.

Signed-off-by: Ryan Moeller <ryan@iXsystems.com>

Pull-request: #10158 part 1/1
Ryan Moeller
Don't ignore zfs_arc_max below allmem/32

Fixes #10157

Set arc_c_min before arc_c_max so that when zfs_arc_min is set lower
than the default allmem/32 zfs_arc_max can also be set lower.

Add warning messages when tunables are being ignored.

Signed-off-by: Ryan Moeller <ryan@iXsystems.com>

Pull-request: #10158 part 1/1
Ryan Moeller
ZTS: Wait for free space between quota tests

And in removal tests, sync the specific pool we are waiting on.

Reviewed-by: John Kennedy <john.kennedy@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Ryan Moeller <ryan@iXsystems.com>
Closes #10146
Dirkjan Bussink
Remove checks for null out value in encryption paths

These paths are never exercised, as the parameters given are always
different cipher and plaintext `crypto_data_t` pointers.

Reviewed-by: Richard Laager <rlaager@wiktel.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Attila Fueloep <attila@fueloep.org>
Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
Closes #9661
Closes #10015
alex
zfs_get: change time format string from %k to %H

Issue #10090 reported that snapshots created between midnight and 1 AM
are missing a padded zero in the creation property

This change fixes the bug reported in issue #10090 where snapshots
created between midnight and 1 AM were missing a padded zero in the
creation timestamp output.

The leading zero was missing because the time format string used `%k`
which formats the hour as a decimal number from 0 to 23 where single
digits are preceded by blanks[0] and is fixed by changing it to `%H`
which formats the hour as 00-23.

The difference in output is as below

```
-Thu Mar 26  0:39 2020
+Thu Mar 26 00:39 2020
```

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Igor Kozhukhov <igor@dilos.org>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Signed-off-by: Alex John <alex@stty.io>
Closes #10090
Closes #10153
Matthew Ahrens
Add `zstream redup` command to convert deduplicated send streams

Deduplicated send and receive is deprecated.  To ease migration to the
new dedup-send-less world, the commit adds a `zstream redup` utility to
convert deduplicated send streams to normal streams, so that they can
continue to be received indefinitely.

The new `zstream` command also replaces the functionality of
`zstreamdump`, by way of the `zstream dump` subcommand.  The
`zstreamdump` command is replaced by a shell script which invokes
`zstream dump`.

The way that `zstream redup` works under the hood is that as we read the
send stream, we build up a hash table which maps from `<GUID, object,
offset> -> <file_offset>`.

Whenever we see a WRITE record, we add a new entry to the hash table,
which indicates where in the stream file to find the WRITE record for
this block. (The key is `drr_toguid, drr_object, drr_offset`.)

For entries other than WRITE_BYREF, we pass them through unchanged
(except for the running checksum, which is recalculated).

For WRITE_BYREF records, we change them to WRITE records.  We find the
referenced WRITE record by looking in the hash table (for the record
with key `drr_refguid, drr_refobject, drr_refoffset`), and then reading
the record header and payload from the specified offset in the stream
file.  This is why the stream can not be a pipe.  The found WRITE record
replaces the WRITE_BYREF record, with its `drr_toguid`, `drr_object`,
and `drr_offset` fields changed to be the same as the WRITE_BYREF's
(i.e. we are writing the same logical block, but with the data supplied
by the previous WRITE record).

This algorithm requires memory proportional to the number of WRITE
records (same as `zfs send -D`), but the size per WRITE record is
relatively low (40 bytes, vs. 72 for `zfs send -D`).  A 1TB send stream
with 8KB blocks (`recordsize=8k`) would use around 5GB of RAM to
"redup".

Signed-off-by: Matthew Ahrens <mahrens@delphix.com>

Pull-request: #10156 part 2/2
Matthew Ahrens
Compile cityhash code into libzfs

Make the cityhash code compile into libzfs, in preparation for the new
"zstream" command.

Signed-off-by: Matthew Ahrens <mahrens@delphix.com>

Pull-request: #10156 part 1/2
Ryan Moeller
Let default arc_c_max be platform dependent

Linux changed the default max ARC size to 1/2 of physical memory to
deal with shortcomings of the Linux SLUB allocator.  Other platforms
do not require the same logic.

Implement an arc_default_max() function to determine a default max ARC
size in platform code.

Signed-off-by: Ryan Moeller <ryan@iXsystems.com>

Pull-request: #10155 part 1/1
Ryan Moeller
Let default arc_c_max be platform dependent

Linux changed the default max ARC size to 1/2 of physical memory to
deal with shortcomings of the Linux SLUB allocator.  Other platforms
do not require the same logic.

Implement an arc_default_max() function to determine a default max ARC
size in platform code.

Signed-off-by: Ryan Moeller <ryan@iXsystems.com>

Pull-request: #10155 part 1/1
George Amanakis
Reset l2ad_hand and l2ad_first in l2arc_evict

Increasing l2arc_write_size or l2arc_write_boost can result in
l2arc_write_buffers() not having enough space to perform its writes and
panic zio_write_phys().

Instead of resetting l2ad_hand to l2ad_start at the end of
l2arc_write_buffers() and not taking into account a possible
user-mediated increase of l2arc_write_max, we do this in l2arc_evict(),
right after l2arc_write_size() has run. If there is not enough space to
evict (ie we will exceed l2ad_end) we evict to the end of the device,
reset l2ad_hand to l2ad_start, set l2ad_first to 0 and iterate
l2arc_evict(). We avoid infinite iteration of l2arc_evict() by making
sure in l2arc_write_size() that l2ad_start + size does not exceed
l2ad_end.

Signed-off-by: George Amanakis <gamanakis@gmail.com>

Pull-request: #10154 part 1/1
George Amanakis
Reset l2ad_hand and l2ad_first in l2arc_evict

Increasing l2arc_write_size or l2arc_write_boost can result in
l2arc_write_buffers() not having enough space to perform its writes and
panic zio_write_phys().

Instead of resetting l2ad_hand to l2ad_start at the end of
l2arc_write_buffers() and not taking into account a possible
user-mediated increase of l2arc_write_max, we do this in l2arc_evict(),
right after l2arc_write_size() has run. If there is not enough space to
evict (ie we will exceed l2ad_end) we evict to the end of the device,
reset l2ad_hand to l2ad_start, set l2ad_first to 0 and iterate
l2arc_evict(). We avoid infinite iteration of l2arc_evict() by making
sure in l2arc_write_size() that l2ad_start + size does not exceed
l2ad_end.

Signed-off-by: George Amanakis <gamanakis@gmail.com>

Pull-request: #10154 part 1/1
George Amanakis
Reset l2ad_hand and l2ad_first in l2arc_evict

Increasing l2arc_write_size or l2arc_write_boost can result in
l2arc_write_buffers() not having enough space to perform its writes and
panic zio_write_phys().

Instead of resetting l2ad_hand to l2ad_start at the end of
l2arc_write_buffers() and not taking into account a possible
user-mediated increase of l2arc_write_max, we do this in l2arc_evict(),
right after l2arc_write_size() has run. If there is not enough space to
evict (ie we will exceed l2ad_end) we evict to the end of the device,
reset l2ad_hand to l2ad_start, set l2ad_first to 0 and iterate
l2arc_evict(). We avoid infinite iteration of l2arc_evict() by making
sure in l2arc_write_size() that l2ad_start + size does not exceed
l2ad_end.

Signed-off-by: George Amanakis <gamanakis@gmail.com>

Pull-request: #10154 part 1/1
George Amanakis
Reset l2ad_hand and l2ad_first in l2arc_evict

Increasing l2arc_write_size or l2arc_write_boost can result in
l2arc_write_buffers() not having enough space to perform its writes and
panic zio_write_phys().

Instead of resetting l2ad_hand to l2ad_start at the end of
l2arc_write_buffers() and not taking into account a possible
user-mediated increase of l2arc_write_max, we do this in l2arc_evict(),
right after l2arc_write_size() has run. If there is not enough space to
evict (ie we will exceed l2ad_end) we evict to the end of the device,
reset l2ad_hand to l2ad_start, set l2ad_first to 0 and iterate
l2arc_evict(). We avoid infinite iteration of l2arc_evict() by making
sure in l2arc_write_size() that l2ad_start + size does not exceed
l2ad_end.

Signed-off-by: George Amanakis <gamanakis@gmail.com>

Pull-request: #10154 part 1/1
George Amanakis
Reset l2ad_hand and l2ad_first in l2arc_evict

Increasing l2arc_write_size or l2arc_write_boost can result in
l2arc_write_buffers() not having enough space to perform its writes and
panic zio_write_phys().

Instead of resetting l2ad_hand to l2ad_start at the end of
l2arc_write_buffers() and not taking into account a possible
user-mediated increase of l2arc_write_max, we do this in l2arc_evict(),
right after l2arc_write_size() has run. If there is not enough space to
evict (ie we will exceed l2ad_end) we evict to the end of the device,
and then reset l2ad_hand to l2ad_start, set l2ad_first to 0 and recurse
l2arc_evict(). We avoid infinite recursion of l2arc_evict() by making
sure in l2arc_write_size() that l2ad_start + size does not exceed
l2ad_end.

Signed-off-by: George Amanakis <gamanakis@gmail.com>

Pull-request: #10154 part 1/1
George Amanakis
Reset l2ad_hand and l2ad_first in l2arc_evict

Increasing l2arc_write_size or l2arc_write_boost can result in
l2arc_write_buffers() not having enough space to perform its writes and
panic zio_write_phys().

Instead of resetting l2ad_hand to l2ad_start at the end of
l2arc_write_buffers() and not taking into account a possible
user-mediated increase of l2arc_write_max, we do this in l2arc_evict(),
right after l2arc_write_size() has run. If there is not enough space to
evict (ie we will exceed l2ad_end) we evict to the end of the device,
and then reset l2ad_hand to l2ad_start, set l2ad_first to 0 and recurse
l2arc_evict(). We avoid infinite recursion of l2arc_evict() by making
sure in l2arc_write_size() that l2ad_start + size does not exceed
l2ad_end.

Signed-off-by: George Amanakis <gamanakis@gmail.com>

Pull-request: #10154 part 1/1
George Amanakis
Reset l2ad_hand and l2ad_first in l2arc_evict

Increasing l2arc_write_size or l2arc_write_boost can result in
l2arc_write_buffers() not having enough space to perform its writes and
panic zio_write_phys().

Instead of resetting l2ad_hand to l2ad_start at the end of
l2arc_write_buffers() and not taking into account a possible
user-mediated increase of l2arc_write_max, we do this in l2arc_evict(),
right after l2arc_write_size() has run. If there is not enough space to
evict (ie we will exceed l2ad_end) we evict to the end of the device,
and then reset l2ad_hand to l2ad_start, set l2ad_first to 0 and recurse
l2arc_evict(). We avoid infinite recursion of l2arc_evict() by making
sure in l2arc_write_size() that l2ad_start + size does not exceed
l2ad_end.

Signed-off-by: George Amanakis <gamanakis@gmail.com>

Pull-request: #10154 part 1/1
George Amanakis
Reset l2ad_hand and l2ad_first in l2arc_evict

Increasing l2arc_write_size or l2arc_write_boost can result in
l2arc_write_buffers() not having enough space to perform its writes and
panic zio_write_phys().

Instead of resetting l2ad_hand to l2ad_start at the end of
l2arc_write_buffers() and not taking into account a possible
user-mediated increase of l2arc_write_max, we do this in l2arc_evict(),
right after l2arc_write_size() has run. If there is not enough space to
evict (ie we will exceed l2ad_end) we evict to the end of the device,
and then reset l2ad_hand to l2ad_start, set l2ad_first to 0 and recurse
l2arc_evict(). We avoid infinite recursion of l2arc_evict() by making
sure in l2arc_write_size() that l2ad_start + size does not exceed
l2ad_end.

Signed-off-by: George Amanakis <gamanakis@gmail.com>

Pull-request: #10154 part 1/1
George Amanakis
Reset l2ad_hand and l2ad_first in l2arc_evict

Increasing l2arc_write_size or l2arc_write_boost can result in
l2arc_write_buffers() not having enough space to perform its writes and
panic zio_write_phys().

Instead of resetting l2ad_hand to l2ad_start at
the end of l2arc_write_buffers() and not taking into account a possible
user-mediated increase of l2arc_write_max, we do this in l2arc_evict(),
right after l2arc_write_size() has run. If there is not enough space to
evict (ie we will exceed l2ad_end) we evict to the end of the device,
and then reset l2ad_hand to l2ad_start, set l2ad_first to 0 and recurse
l2arc_evict(). We avoid infinite recursion of l2arc_evict() by making
sure in l2arc_write_size() that l2ad_start + size does not exceed
l2ad_end.

Signed-off-by: George Amanakis <gamanakis@gmail.com>

Pull-request: #10154 part 1/1
George Amanakis
Reset l2ad_hand and l2ad_first in l2arc_evict

Signed-off-by: George Amanakis <gamanakis@gmail.com>

Pull-request: #10154 part 1/1
George Amanakis
Reset l2ad_hand and l2ad_first in l2arc_evict

Signed-off-by: George Amanakis <gamanakis@gmail.com>

Pull-request: #10154 part 1/1
Alex John
zfs_get: change time format string from %k to %H

Issue #10090 reported that snapshots created between midnight and 1 AM
are missing a padded zero in the creation property

This change fixes the bug reported in issue #10090 where snapshots
created between midnight and 1 AM were missing a padded zero in the
creation timestamp output.

The leading zero was missing because the time format string used `%k`
which formats the hour as a decimal number from 0 to 23 where single
digits are preceded by blanks[0] and is fixed by changing it to `%H`
which formats the hour as 00-23.

The difference in output is as below

```
-Thu Mar 26  0:39 2020
+Thu Mar 26 00:39 2020
```

Signed-off-by: Alex John <alex@stty.io>

Pull-request: #10153 part 1/1
Matthew Ahrens
Compile cityhash code into libzfs

Make the cityhash code compile into libzfs, in preparation for the new
"zstream" command.

Signed-off-by: Matthew Ahrens <mahrens@delphix.com>

Pull-request: #10152 part 1/1
Fabio Scaccabarozzi
Fix missing renames

Signed-off-by: Fabio Scaccabarozzi <fsvm88@gmail.com>

Pull-request: #10148 part 6/6
Fabio Scaccabarozzi
Make 'make checkstyle' happy, and summarize the comment a bit as per request on PR.

Signed-off-by: Fabio Scaccabarozzi <fsvm88@gmail.com>

Pull-request: #10148 part 5/5
Fabio Scaccabarozzi
Make 'make checkstyle' happy

Signed-off-by: Fabio Scaccabarozzi <fsvm88@gmail.com>

Pull-request: #10148 part 4/5
Fabio Scaccabarozzi
Do not handle skip == iov->iov_len, because in the partial write case this is not reachable. Fix double-counting bug.

Signed-off-by: Fabio Scaccabarozzi <fsvm88@gmail.com>

Pull-request: #10148 part 3/5
Fabio Scaccabarozzi
Update pending bytes on partial copy in zfs_write() before continue-ing
the loop, and leave a lengthy comment explaining the reason why this is done.

Signed-off-by: Fabio Scaccabarozzi <fsvm88@gmail.com>

Pull-request: #10148 part 2/5
Fabio Scaccabarozzi
Account for partial copies and update uio struct before returning
EFAULT.

Signed-off-by: Fabio Scaccabarozzi <fsvm88@gmail.com>

Pull-request: #10148 part 1/5
Fabio Scaccabarozzi
Update pending bytes on partial copy in zfs_write() before continue-ing
the loop, and leave a lengthy comment explaining the reason why this is done.

Signed-off-by: Fabio Scaccabarozzi <fsvm88@gmail.com>

Pull-request: #10148 part 2/2
Fabio Scaccabarozzi
Account for partial copies and update uio struct before returning
EFAULT.

Signed-off-by: Fabio Scaccabarozzi <fsvm88@gmail.com>

Pull-request: #10148 part 1/2
Paul Dagnelie
more feedback

Signed-off-by: Paul Dagnelie <pcd@delphix.com>

Pull-request: #9707 part 9/9
George Amanakis
Persistent L2ARC

This commit makes the L2ARC persistent across reboots. We implement
a light-weight persistent L2ARC metadata structure that allows L2ARC
contents to be recovered after a reboot. This significantly eases the
impact a reboot has on read performance on systems with large caches.

Co-authored-by: Saso Kiselkov <skiselkov@gmail.com>
Co-authored-by: Jorgen Lundman <lundman@lundman.net>
Co-authored-by: George Amanakis <gamanakis@gmail.com>
Ported-by: Yuxuan Shui <yshuiv7@gmail.com>
Signed-off-by: George Amanakis <gamanakis@gmail.com>

Pull-request: #9582 part 1/1
GitHub
Merge branch 'master' into persist_l2arc_unified

Pull-request: #9582 part 2/2
  • Debian 8 arm (BUILD): cloning zfs -  stdio
  • Debian 8 ppc (BUILD): cloning zfs -  stdio
  • Ubuntu 16.04 aarch64 (BUILD): cloning zfs -  stdio
George Amanakis
Persistent L2ARC

This commit makes the L2ARC persistent across reboots. We implement
a light-weight persistent L2ARC metadata structure that allows L2ARC
contents to be recovered after a reboot. This significantly eases the
impact a reboot has on read performance on systems with large caches.

Co-authored-by: Saso Kiselkov <skiselkov@gmail.com>
Co-authored-by: Jorgen Lundman <lundman@lundman.net>
Co-authored-by: George Amanakis <gamanakis@gmail.com>
Ported-by: Yuxuan Shui <yshuiv7@gmail.com>
Signed-off-by: George Amanakis <gamanakis@gmail.com>

Pull-request: #9582 part 1/1
  • Ubuntu 16.04 i386 (BUILD): cloning zfs -  stdio
Ryan Moeller
Add FreeBSD support to OpenZFS

- Import FreeBSD specific parts

Signed-off-by: Matt Macy <mmacy@FreeBSD.org>
Co-authored-by: Ryan Moeller <ryan@iXsystems.com>
Signed-off-by: Ryan Moeller <ryan@iXsystems.com>

Requires-builders: amazon2, debian10, freebsd12, freebsd13, style

Pull-request: #8987 part 1/1
Ryan Moeller
Add FreeBSD support to OpenZFS

- Import FreeBSD specific parts

Signed-off-by: Matt Macy <mmacy@FreeBSD.org>
Co-authored-by: Ryan Moeller <ryan@iXsystems.com>
Signed-off-by: Ryan Moeller <ryan@iXsystems.com>

Requires-builders: amazon2, debian10, freebsd12, freebsd13, style

Pull-request: #8987 part 1/1