Home NAS with Ubuntu and ZFS

#zfs #nas #ubuntu

My little adventure in building the ultimate home NAS using Ubuntu and ZFS. Let's get right into the hardware specs:

  • Rosewill RSV-L4312 Chassis with 12 3.5-inch hotswap bays
  • Intel Core i7-9800X (overclocked to 4.7Ghz)
  • 128GB Crucial Ballistix Sport LT 2666 MHz DDR4 DRAM
  • Corsair H100i v2 AIO CPU Cooler
  • Noctua NF-F12 industrialPPC-3000 PWM 120mm fans x5 (intake and radiator)
  • Noctua NF-A14 industrialPPC-3000 PWM (PCIe)
  • Noctua NF-A8 PWM x2 (exhaust)
  • Silverstone 8-Port PWM Fan Hub/Splitter
  • ASUS X299 TUF Mark 1 Motherboard
  • ASUS Hyper Card v2
  • LSI 9201-16i HBA
  • Cable Matters Mini SAS (SFF-8087) to SATA Forward Breakout cables
  • Seagate EXO x10 10TB (ST10000NM0086) HDD x12
  • Western Digital Blue 3D NAND SSD 250GB (OS Software RAID-1)
  • Samsung 970 Pro m.2 NVMe SSD x2 (ZFS l2arc cache)
  • Samsung 970 EVO Plus m.2 NVMe SSD x4 (ZFS l2arc cache)
  • Mellanox MCX311A-XCAT ConnectX-3 10Gbe SPF+ Network Adapter
  • ZOTAC GeForce GT 710
  • Startech 6U Vertical Wall mount

Chassis

Initially, I chose the iStarUSA M-4160-ATX, but all 4 backplanes were DOA. I really, really wanted this chassis to work, the 16 hotswap bays + 2x2.5 inch bays  was ideal for my setup. But at $600+ USD and all 4 backplanes DOA, I didn't trust a replacement. I found the Rosewill RSV-L4312 on newegg.com as an open box buy at $150 USD so I decided to try it.

The Rosewill had 4 less front bays and no internal bays for OS drives. Bummer. But the Rosewill fit larger motherboards, E-ATX vs ATX, and I was able to fit my Corsair H100i v2 AIO cooler inside, which was a bonus as I was able to keep my overclock settings. Something I had to dial back when I tried a Noctua NH-U9DX i4 4U air cooler.

Water Cooling

Water cooling was a nice bonus. The main support brace had a little groove in it that actually fit the H100i v2 radiator perfectly!

I just drilled new holes and mounted the brace upside down. I have emailed Rosewill parts department asking for another brace so I can clamp it down from the top; otherwise, I will have to manufacture some sort clamp or maybe just use some extra strength rubber bands to hold the radiator securely.

ZFS

Unlike Linus, Ubuntu + ZFS is awesome. I have 12 10TB disks, 11 in raidz3 with 1 hotspare, 6 1TB NVMe m.2 SSDs for l2arc cache. I have read articles saying l2arc cache can slow you down, but with using 10GB SFP+ connections and 4K video editing with Davinci Resolve 16 using network drives, the l2arc cache really makes a difference. I guess it would really depend on your workload. For things like Plex, I use a ZFS dataset in which l2arc is only used for metadata, as the sequential read once workload does not make a lot of sense to fill the l2arc cache.

$ zfs set secondarycache=metadata vol0/plex

Devices

Device       Model                          Firmware   Temperature
/dev/nvme0   Samsung SSD 970 PRO 1TB        1B2QEXP7   23°C
/dev/nvme1   Samsung SSD 970 PRO 1TB        1B2QEXP7   25°C
/dev/nvme2   Samsung SSD 970 EVO Plus 1TB   2B2QEXM7   26°C
/dev/nvme3   Samsung SSD 970 EVO Plus 1TB   2B2QEXM7   27°C
/dev/nvme4   Samsung SSD 970 EVO Plus 1TB   2B2QEXM7   27°C
/dev/nvme5   Samsung SSD 970 EVO Plus 1TB   2B2QEXM7   27°C
/dev/sda     ST10000NM0086-2AA101           SN05       25°C
/dev/sdb     ST10000NM0086-2AA101           SN05       26°C
/dev/sdc     ST10000NM0086-2AA101           SN05       26°C
/dev/sdd     ST10000NM0086-2AA101           SN05       26°C
/dev/sde     ST10000NM0086-2AA101           SN05       24°C
/dev/sdf     ST10000NM0086-2AA101           SN05       24°C
/dev/sdg     ST10000NM0086-2AA101           SN05       24°C
/dev/sdh     ST10000NM0086-2AA101           SN05       24°C
/dev/sdi     ST10000NM0086-2AA101           SN05       24°C
/dev/sdj     ST10000NM0086-2AA101           SN05       25°C
/dev/sdk     ST10000NM0086-2AA101           SN05       24°C
/dev/sdl     ST10000NM0086-2AA101           SN05       24°C
/dev/sdm     WDC WDS250G2B0A-00SM50         401000WD   28°C
/dev/sdn     WDC WDS250G2B0A-00SM50         401000WD   27°C

Using 2 m.2 slots on motherboard + 4 m.2 slots on ASUS Hyper Card v2 for 6 SSD l2arc cache. One of my Seagate drives had older firmware, but I was able to update via command line using Seagate tools from support site.

OS drives are just some generic Western Digital Blue SSDs in software raid-1, they don't see much action, so I am not worried about SSD wear.

IO udev Rules

/etc/udev/rules.d/99-schedulers.rules
ACTION=="add|change", KERNEL=="sd[a-z]", ATTRS{model}=="ST10000NM0086-2A", ATTR{queue/scheduler}="deadline", ATTR{queue/read_ahead_kb}="256", ATTR{queue/nr_requests}="256"
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="deadline", ATTR{queue/read_ahead_kb}="4096", ATTR{queue/nr_requests}="512"
ACTION=="add|change", KERNEL=="nvme[0-9]n1", ATTR{queue/read_ahead_kb}="4096"

Zpool Status

  pool: vol0
 state: ONLINE
  scan: scrub repaired 0B in 6h43m with 0 errors on Mon Jan 13 06:22:53 2020
config:

        NAME                         STATE     READ WRITE CKSUM
        vol0                         ONLINE       0     0     0
          raidz3-0                   ONLINE       0     0     0
            wwn-0x5000c500b37d1d7c   ONLINE       0     0     0
            wwn-0x5000c500b2769d77   ONLINE       0     0     0
            wwn-0x5000c500b26e6c80   ONLINE       0     0     0
            wwn-0x5000c500b152e4a2   ONLINE       0     0     0
            wwn-0x5000c500b26e22b6   ONLINE       0     0     0
            wwn-0x5000c500b26d8443   ONLINE       0     0     0
            wwn-0x5000c500b4f69bcb   ONLINE       0     0     0
            wwn-0x5000c500b697cd6a   ONLINE       0     0     0
            wwn-0x5000c500b4f9a95e   ONLINE       0     0     0
            wwn-0x5000c500b697219e   ONLINE       0     0     0
            wwn-0x5000c500c3b3928d   ONLINE       0     0     0
        cache
          nvme-eui.0025385881b01bc5  ONLINE       0     0     0
          nvme-eui.0025385881b01ba5  ONLINE       0     0     0
          nvme-eui.0025385b9150d35a  ONLINE       0     0     0
          nvme-eui.0025385391b2a3a6  ONLINE       0     0     0
          nvme-eui.0025385391b2a375  ONLINE       0     0     0
          nvme-eui.0025385b9150efa6  ONLINE       0     0     0
        spares
          wwn-0x5000c500b6970b24     AVAIL

errors: No known data errors

Zpool IO Status

                               capacity     operations     bandwidth
pool                         alloc   free   read  write   read  write
---------------------------  -----  -----  -----  -----  -----  -----
vol0                         27.3T  72.7T  1.24K  1.16K  30.3M  29.7M
  raidz3                     27.3T  72.7T  1.24K  1.16K  30.3M  29.7M
    wwn-0x5000c500b37d1d7c       -      -    116    108  2.75M  2.70M
    wwn-0x5000c500b2769d77       -      -    120    108  2.74M  2.70M
    wwn-0x5000c500b26e6c80       -      -    114    108  2.76M  2.70M
    wwn-0x5000c500b152e4a2       -      -    117    109  2.75M  2.70M
    wwn-0x5000c500b26e22b6       -      -    115    108  2.76M  2.70M
    wwn-0x5000c500b26d8443       -      -    114    107  2.76M  2.70M
    wwn-0x5000c500b4f69bcb       -      -    111    108  2.77M  2.70M
    wwn-0x5000c500b697cd6a       -      -    118    108  2.74M  2.70M
    wwn-0x5000c500b4f9a95e       -      -    113    107  2.77M  2.70M
    wwn-0x5000c500b697219e       -      -    115    108  2.76M  2.70M
    wwn-0x5000c500c3b3928d       -      -    114    107  2.76M  2.70M
cache                            -      -      -      -      -      -
  nvme-eui.0025385881b01bc5  3.27G   951G      0     22     99  2.44M
  nvme-eui.0025385881b01ba5  4.82G   949G      0     37     99  3.60M
  nvme-eui.0025385b9150d35a  3.58G   928G      0     30     99  2.68M
  nvme-eui.0025385391b2a3a6  3.86G   928G      0     29     99  2.87M
  nvme-eui.0025385391b2a375  3.81G   928G      0     43     99  2.84M
  nvme-eui.0025385b9150efa6  3.25G   928G      0     23     99  2.44M
---------------------------  -----  -----  -----  -----  -----  -----

Cooling and Noise

The Corsair AIO cooler is nice to have. My i7-9800X is overclocked to 4.7GHz all cores boost (using a -6 AVX offset as AVX instructions have an insane power and heat generation profile). Final config as 3 Noctua 120mm fans pulling off the hotswap bays and 2 more pulling on radiator. PCIe slots need a fan due to the LSI 9201-16i running insanely hot. So hot, even at idle, it caused the Mellanox ethernet adapter to throw kernel errors about core temps over 105C!

Idle Temps

 $ sensors
 coretemp-isa-0000
 Adapter: ISA adapter
i7-9800x:     +30.0°C  (high = +100.0°C, crit = +110.0°C)
Core 0:       +29.0°C  (high = +100.0°C, crit = +110.0°C)
Core 1:       +27.0°C  (high = +100.0°C, crit = +110.0°C)
Core 2:       +28.0°C  (high = +100.0°C, crit = +110.0°C)
Core 3:       +29.0°C  (high = +100.0°C, crit = +110.0°C)
Core 4:       +28.0°C  (high = +100.0°C, crit = +110.0°C)
Core 5:       +28.0°C  (high = +100.0°C, crit = +110.0°C)
Core 6:       +30.0°C  (high = +100.0°C, crit = +110.0°C)
Core 7:       +29.0°C  (high = +100.0°C, crit = +110.0°C)

Mprime95 Temps

$ sensors
coretemp-isa-0000
Adapter: ISA adapter
i7-9800x:     +75.0°C  (high = +100.0°C, crit = +110.0°C)
Core 0:       +69.0°C  (high = +100.0°C, crit = +110.0°C)
Core 1:       +74.0°C  (high = +100.0°C, crit = +110.0°C)
Core 2:       +72.0°C  (high = +100.0°C, crit = +110.0°C)
Core 3:       +73.0°C  (high = +100.0°C, crit = +110.0°C)
Core 4:       +60.0°C  (high = +100.0°C, crit = +110.0°C)
Core 5:       +75.0°C  (high = +100.0°C, crit = +110.0°C)
Core 6:       +68.0°C  (high = +100.0°C, crit = +110.0°C)
Core 7:       +66.0°C  (high = +100.0°C, crit = +110.0°C)

Never breaking 78C is not bad for a 4U chassis and air flow.

Vertical 6U (upside down)

I have a very narrow utility room in my 1952 ranch-style house, so I needed to vertical mount the server. I used a Startech 6U vertical mount. I also mounted things upside down, given that hot air rises! Using a 1U CyberPower OR1500LCDRM1U Smart App LCD UPS System, 1500VA/900W. I mounted the UPS closest to the wall, then left 1U of empty space and mounted the Rosewill chassis on the outer 4U. The 2-foot by 4-foot, 3/4-inch birch plywood is mounted directly to the wall studs, the Startech 6U rack is mounted to the plywood.