Creating a ZFS+Samba Share (Ampere Build Finale)

Today the final pieces of this server arrived, the SATA SSDs to go into the hotswap bays. I really wanted a 4x8TB setup for a few reasons:

• 8TB is the largest 2.5 SATA SSD that consumers can buy right now
• Even if cost was a concern, I can always start with a mirrored 2x8TB, and add in another set of mirrored 2x8TB for a total of 16TB, which is still larger than a RAIDZ1 4x4TB array

But I really urgently wanted to get a backup schedule of my primary NAS going, so I decided to compromise and grab the 4TBs, as I just was not able to find any 8TBs in stock.

I set up the 4 drives in a RAIDZ1 array so that one drive is used as a parity drive.

$ zpool status zfspool
  pool: zfspool
 state: ONLINE
config:

        NAME        STATE     READ WRITE CKSUM
        zfspool     ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            sda     ONLINE       0     0     0
            sdb     ONLINE       0     0     0
            sdc     ONLINE       0     0     0
            sdd     ONLINE       0     0     0

And my final available space is 10.6TB. This should be sufficient for backing up my critical UNAS Pro shares that I can't afford to lose.

$ zfs list
NAME      USED  AVAIL  REFER  MOUNTPOINT
zfspool   119G  10.6T   119G  /zfspool

I do wish I had double the capacity through 8TB drives... If they ever get back in stock, I'll do a full replacement with 8TB drives, and I'll move the 4TB drives to a new 2U chassis that I will be transplanting my current 3U gaming rig to.

As the last step, I set up a backup task in the UNAS Pro web console. One thing to note is that for IP Address, I had to explicitly type out the IP address of the host, as using the hostname kept throwing an error.

The backup seems to fail a bit, with logs being written to UNAS Pro's /var/log/error.

2025-04-02T05:15:05-07:00 UNAS-Pro rclone[2262]: ERROR : b2: failed to open directory "b2": open /srv/.unifi-drive/Immich/.data/b2: permission denied
2025-04-02T05:34:16-07:00 UNAS-Pro rclone[2262]: ERROR : immich/postgres/pg_logical/replorigin_checkpoint: Failed to copy: failed to open source object: open /srv/.unifi-drive/SwarmDrive/.data/immich/postgres/pg_logical/replorigin_checkpoint: permission denied
2025-04-02T05:34:27-07:00 UNAS-Pro rclone[2262]: ERROR : pydio/data/services/pydio.grpc.jobs/tasklogs.bleve/store/000000005b20.zap: Failed to copy: failed to open source object: lstat /srv/.unifi-drive/SwarmDrive/.data/pydio/data/services/pydio.grpc.jobs/tasklogs.bleve/store/000000005b20.zap: no such file or directory

All failures seem related to permission related issues. I went ahead and ran chmod 777 on directories that were having permission issues, but the remaining sources of failures seem to be from temp files used by Postgres which are created with -rw------- permissions.

$ ls -l
-rw------- 1 unifi-drive-nfs unifi-drive 4445 Apr 2 11:06 db_0.stat

unifi-drive-nfs is the default user that the UNAS Pro squashes to for clients connected through NFS, thus it doesn't make too much sense to me that Unifi Drive's backup capability (which seems to use rclone under the hood based on the error logs) has permission errors with these files.

Looking at journalctl --output=verbose, rclone is running as

    _UID=988
    _GID=988

which based on /etc/passwd is user unifi-drive:x:988:988::/home/unifi-drive:/usr/sbin/nologin

Expectation is that rclone should be runnable as unifi-drive-nfs:x:977:988::/home/unifi-drive-nfs:/usr/sbin/nologin

I've gone ahead and filed this to Ubiquiti

https://community.ui.com/questions/Backup-failing-for-files-owned-by-unifi-drive-nfs-with-only-rw-permissions/8d4858fc-77ef-44bb-bc58-baaf820c13f9

Symlinks also seem to be an issue

Apr 02 11:07:06 UNAS-Pro rclone[2262]: NOTICE: nextcloud/var/www/html/apps/recommendations/js/recommendations-dashboard.js.map.license: Can't follow symlink without -L/--copy-links

These two last issues can be solved by modifying some settings on the UNAS Pro, first the user can be addressed by modifying the init script used by the NAS (change APP_USER to unifi-drive-nfs:

root@UNAS-Pro:~# cat /usr/share/unifi-drive-rclone/rclone-init.sh
#!/bin/sh

# The script will be executed before the rclone launch.

APP=unifi-drive
APP_USER=unifi-drive-nfs
APP_GROUP=unifi-drive
APP_DATA=/data/$APP

And the symlink issue can be resolved by changing the rclone configuration by adding --copy-links to the rclone systemd service:

root@UNAS-Pro:~# cat /lib/systemd/system/rclone-rcd.service
[Unit]
Description=rclone remote control daemon
After=network-online.target
Requires=network-online.target

[Service]
Type=notify
NotifyAccess=main
WorkingDirectory=/usr/share/unifi-drive
ExecStartPre=+/usr/share/unifi-drive-rclone/rclone-init.sh
ExecStart=/usr/sbin/rclone --config /data/unifi-drive/rclone/rclone.conf rcd --rc-addr unix://@unifi-drive-rclone --rc-no-auth --use-mmap --copy-links
Restart=always
RestartSec=5
StandardOutput=journal
User=unifi-drive-nfs
Group=unifi-drive

[Install]
WantedBy=multi-user.target

In the systemd service, we also need to make sure that User value is changed to unifi-drive-nfs as well. I needed to add some --exclude to ignore some symlinks that kept failing, but at the end of the day I was able to get the backup job to complete successfully:

These configuration seem a bit hacky and will get wiped on an OS update, so I really hope Ubiquiti provides capability in Unifi Drive to work around this.