democratic-csi based simple guide to use Kubernetes cluster with TrueNAS Scale over API.
You can use the democratic-csi documentation and achieve the same results, but the reason I created this guide is that the democratic-csi docs cover multiple complex combinations of various technologies. If you just want to have NFS/iSCSI over API, then the whole setup guide can be much simpler.
You're here because you want to connect a Kubernetes cluster to a TrueNAS Scale-based NAS, right? So you need:
- K8S cluster - in my case deployed using k0sctl, but it shouldn't really matter what you use to create it.
- NAS based on TrueNAS Scale - in my case, it's ugly-nas
- Nodes section needs to be run on the nodes of your cluster, preferably Debian-based
- NAS section needs to be done on your NAS, and the assumption is that this is a TrueNAS Scale-based solution
- K8S section needs to be run on whatever machine you're using to manage your cluster
- Controlplane - 10.10.20.99
- TrueNAS Scale - 10.10.20.100
- node1 - 10.10.20.101
- node2 - 10.10.20.102
- node3 - 10.10.20.103
All your nodes should be capable of using NFS/iSCSI shares. This means that some extra packages need to be installed on them. In my case, it's being handled by dumb-provisioner which takes care of installing my terminals, but in general, you just have to run this on every node:
# apt install nfs-common open-iscsi multipath-tools scsitools lsscsi
# cat <<EOF > /etc/multipath.conf
defaults {
user_friendly_names yes
find_multipaths yes
}
EOFThe hash at the beginning of lines means you have to run them as root. So if you're on Ubuntu, just add sudo in front of these commands.
For Fedora/RedHat-based systems, something like this should also work:
# dnf install -y lsscsi iscsi-initiator-utils sg3_utils device-mapper-multipathThe key copied into the clipboard needs to be used in the YAML files below as apiKey.
Dataset creation for both NFS and iSCSI
Now iSCSI configuration.
And NFS configuration.
Be aware that the above setups aren't hardened at all. This is just a basic setup created for testing purposes. If you want to use it in a production environment, then you have to tweak it and make it more secure.
Side note: Below is an example of how to install democratic-csi using helm with a values file. I don't actually do it manually like this. In reality, I'm deploying democratic-csi using flux, which handles my homelab. To see the actual values files used to deploy democratic-csi on my cluster, take a look at my repo.
Let's start with adding repo.
# helm repo add democratic-csi https://democratic-csi.github.io/charts/
# helm repo updateNow a few words about YAML files used as values for helm charts. Since democratic-csi tries to be universal and they cover a bunch of different scenarios (which is great, BTW!), it means that to prepare a file to use as values for a helm chart, you have to combine at least two files from their repos. They explain which files and under which conditions. But if you want to use just NFS/iSCSI and just over API, then their documentation might be a bit confusing. The files prepared by me below are based on already combined files, but for reference, you will also find commands to create skeleton files that you can adapt yourself. But I want to make it clear: below you will find a simplified version of two files grabbed from the democratic-csi repo, merged together, with comments removed and adapted to my needs. Most people will have to do more or less the same, but NOT all of them. So be aware that my files might not be something that you can copy and paste into your environment.
csiDriver:
name: "nfs"
storageClasses:
- name: nfs
defaultClass: false
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
parameters:
fsType: nfs
detachedVolumesFromSnapshots: "false"
mountOptions:
- noatime
- nfsvers=3
secrets:
provisioner-secret:
controller-publish-secret:
node-stage-secret:
node-publish-secret:
controller-expand-secret:
volumeSnapshotClasses:
- name: nfs
parameters:
detachedSnapshots: "true"
driver:
config:
driver: freenas-api-nfs
instance_id:
httpConnection:
protocol: http
host: 10.10.20.100
port: 80
apiKey: 1-IvCjJtMLUhEUseYourOwnrK1HKRIFWd1UFK5ay52HogLUrwC2UxjHNQWODCRGhe
allowInsecure: true
zfs:
datasetParentName: storage/k8s/nfs/v
detachedSnapshotsDatasetParentName: storage/k8s/nfs/s
datasetEnableQuotas: true
datasetEnableReservation: false
datasetPermissionsMode: "0777"
datasetPermissionsUser: 0
datasetPermissionsGroup: 0
nfs:
shareHost: 10.10.20.100
shareAlldirs: false
shareAllowedHosts: []
shareAllowedNetworks: []
shareMaprootUser: root
shareMaprootGroup: root
shareMapallUser: ""
shareMapallGroup: ""As already mentioned, the above is a simplified/streamlined version of the values file. Obviously, you have to change at least the paths and IP of the server.
Below are the commands to get the full file with comments.
# wget https://raw.githubusercontent.com/democratic-csi/charts/master/stable/democratic-csi/examples/freenas-nfs.yaml -O - | sed '/INLINE/,$d' > nfs.yaml
# wget https://raw.githubusercontent.com/democratic-csi/democratic-csi/master/examples/freenas-api-nfs.yaml -O - | sed -e 's/^/ /g' >> nfs.yamlNotice that since TrueNAS 25, the TrueNAS API no longer returns "SCALE" in the version string, which will stop the democratic-csi controller from starting up properly. A fix has been applied in the "next" branch of the democratic-csi repository. As a result, you should add the following to your nfs.yaml:
controller:
driver:
image:
tag: nextThis will change the csi-driver image from the "latest" tag to the "next" tag.
Make sure to double check your actual portal ID in CLI or over API as TrueNAS WebUI isn't really reliable with regard to that.
Here's how to check it from NAS cli:
root@nas[~]# cli
[nas]> sharing iscsi portal query
+----+-----+---------+--------+----------------------+---------------------+
| id | tag | comment | listen | discovery_authmethod | discovery_authgroup |
+----+-----+---------+--------+----------------------+---------------------+
| 8 | 1 | iscsi | <list> | NONE | <null> |
+----+-----+---------+--------+----------------------+---------------------+Or straight from Linux shell but also using NAS cli:
root@nas[~]# cli -c
9DA7
"sharing iscsi portal query"
+----+-----+---------+--------+----------------------+---------------------+
| id | tag | comment | listen | discovery_authmethod | discovery_authgroup |
+----+-----+---------+--------+----------------------+---------------------+
| 8 | 1 | iscsi | <list> | NONE | <null> |
+----+-----+---------+--------+----------------------+---------------------+csiDriver:
name: "iscsi"
storageClasses:
- name: iscsi
defaultClass: false
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
parameters:
fsType: ext4
detachedVolumesFromSnapshots: "false"
mountOptions: []
secrets:
provisioner-secret:
controller-publish-secret:
node-stage-secret:
node-publish-secret:
controller-expand-secret:
volumeSnapshotClasses:
- name: iscsi
parameters:
detachedSnapshots: "true"
driver:
config:
driver: freenas-api-iscsi
instance_id:
httpConnection:
protocol: http
host: 10.10.20.100
port: 80
apiKey: 1-IvCjJtMLUhEUseYourOwnrK1HKRIFWd1UFK5ay52HogLUrwC2UxjHNQWODCRGhe
allowInsecure: true
zfs:
datasetParentName: storage/k8s/iscsi/v
detachedSnapshotsDatasetParentName: storage/k8s/iscsi/s
zvolCompression:
zvolDedup:
zvolEnableReservation: false
zvolBlocksize:
iscsi:
targetPortal: "10.10.20.100:3260"
targetPortals: []
interface:
namePrefix: csi-
nameSuffix: "-clustera"
targetGroups:
- targetGroupPortalGroup: 8
targetGroupInitiatorGroup: 14
targetGroupAuthType: None
targetGroupAuthGroup:
extentInsecureTpc: true
extentXenCompat: false
extentDisablePhysicalBlocksize: true
extentBlocksize: 512
extentRpm: "SSD"
extentAvailThreshold: 0Again, the above is a simplified/streamlined version of the values file. Below are the commands to get the file with comments. You have to change the portal details, so at least its IP, group, and initiator group.
# wget https://raw.githubusercontent.com/democratic-csi/charts/master/stable/democratic-csi/examples/freenas-iscsi.yaml -O - | sed '/INLINE/,$d' > iscsi.yaml
# wget https://raw.githubusercontent.com/democratic-csi/democratic-csi/master/examples/freenas-api-iscsi.yaml -O - | sed -e 's/^/ /g' >> iscsi.yamlOnce you've got your files you can install democratic-csi like this:
# helm upgrade --install --create-namespace --values nfs.yaml --namespace storage nfs democratic-csi/democratic-csi
# helm upgrade --install --create-namespace --values iscsi.yaml --namespace storage iscsi democratic-csi/democratic-csiIf you're experimenting with various settings, make sure to uninstall the helm release every time, as the above command will try to update the existing release if it already exists.
State after installation (depends on how many nodes you've got in your cluster):
# helm ls -n storage
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
iscsi storage 1 2023-06-07 18:54:00.318001 +0200 CEST deployed democratic-csi-0.13.7 1.0
nfs storage 1 2023-06-07 18:55:42.713606 +0200 CEST deployed democratic-csi-0.13.7 1.0
# kubectl get all -n storage
NAME READY STATUS RESTARTS AGE
pod/iscsi-democratic-csi-controller-5bf8f859f6-4xcs4 5/5 Running 0 18h
pod/iscsi-democratic-csi-node-cszgj 4/4 Running 0 18h
pod/iscsi-democratic-csi-node-f9cpm 4/4 Running 0 18h
pod/iscsi-democratic-csi-node-r9zgv 4/4 Running 0 18h
pod/nfs-democratic-csi-controller-658b478c97-v4qb9 5/5 Running 0 18h
pod/nfs-democratic-csi-node-h5h7z 4/4 Running 0 18h
pod/nfs-democratic-csi-node-ql9lt 4/4 Running 0 18h
pod/nfs-democratic-csi-node-sfppn 4/4 Running 0 18h
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/iscsi-democratic-csi-node 3 3 3 3 3 kubernetes.io/os=linux 18h
daemonset.apps/nfs-democratic-csi-node 3 3 3 3 3 kubernetes.io/os=linux 18h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/iscsi-democratic-csi-controller 1/1 1 1 18h
deployment.apps/nfs-democratic-csi-controller 1/1 1 1 18h
NAME DESIRED CURRENT READY AGE
replicaset.apps/iscsi-democratic-csi-controller-5bf8f859f6 1 1 1 18h
replicaset.apps/nfs-democratic-csi-controller-658b478c97 1 1 1 18hFiles that we can use to test the actual PVC:
pvc-iscsi.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: test-claim-iscsi
annotations:
volume.beta.kubernetes.io/storage-class: "iscsi"
spec:
storageClassName: iscsi
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gipvc-nfs.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: test-claim-nfs
annotations:
volume.beta.kubernetes.io/storage-class: "nfs"
spec:
storageClassName: nfs
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1GiAnd the actual test:
# kubectl apply -f pvc-iscsi.yaml -f pvc-nfs.yaml
persistentvolumeclaim/test-claim-iscsi created
persistentvolumeclaim/test-claim-nfs created
# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
test-claim-iscsi Bound pvc-461a9ab4-106b-4f62-b06b-746d9c8e8c73 1Gi RWO iscsi 11s
test-claim-nfs Bound pvc-86623c48-6f32-4d29-8baf-7881c8da1ec2 1Gi RWO nfs 11sAnd cleaning up:
# kubectl delete -f pvc-iscsi.yaml -f pvc-nfs.yaml
persistentvolumeclaim "test-claim-iscsi" deleted
persistentvolumeclaim "test-claim-nfs" deleted