One of the great new features of PSO 6 is ability to create a storage class with a pre-defined limit on IO or bandwidth (or both). Watch the following short demo to check it out.
Now if you used any of the old versions of PSO you know it can smart provision across Pure Storage arrays with a single storageClass for block and one for file. Today I am proud to share the mysterious and sometimes confusing third storageClass pure is no longer installed with PSO 6. The long story is that storage class was to support legacy systems that use the 1.0 version of our driver. There has been 2.5 years to get used to pure-block. So now with the upgrade you can make the right choice.
jowings@asgard ~/pso-values k get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
pure-block pure-csi Delete Immediate true 56s
pure-file pure-csi Delete Immediate true 56s
I want to attach to a share that is already used by a physical server or some other device. I also want to attach containers that are orchestrated by K8s. This scenario is one customers have been asking for since the first version of Pure Service Orchestrator. When you normally create a PVC the PSO provisioner creates a volume or filesystem that looks something like this:
The first version of PSO’s FlexVolume Driver supported an import feature but it would take an existing volume and rename it to something like in the screenshot above. With the new “soft import” feature now in the latest PSO CSI driver you can now create a PVC tied to any existing volume and it won’t rename it. So any external connections or applications are not interrupted. How can you do this?
apiVersion: v1
kind: PersistentVolume
metadata:
annotations:
pv.kubernetes.io/provisioned-by: pure-csi
name: pv-import
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 1Ti
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
# TODO: change to the PVC you want to bind this PV.
# If you don't pre-bind PVC here, the PV might be automatically bound to a PVC by scheduler.
name: pvc-import1
# Namespace of the PVC
namespace: app1
csi:
driver: pure-csi
# TODO: change to the volume name in backend.
# Volume with any name that exists in backend can be imported, and will not be renamed.
volumeHandle: externalfiles
volumeAttributes:
backend: file
# TODO: configure your desired reclaim policy,
# Use Retain if you don't want your volume to get deleted when the PV is deleted.
persistentVolumeReclaimPolicy: Retain
storageClassName: pure-file
volumeMode: Filesystem
Very Important Note: persistentVolumeReclaimPolicy is set to Retain. This ensures the filesystem is not deleted if the PV is deleted.
Notice the externalfiles volumeHandle matches the filesystem already in use on the FlashBlade.
3. Now we have to create a PVC to match the namespace and name specified in the PV.
apiVersion: "v1"
kind: "PersistentVolumeClaim"
metadata:
name: pvc-import1
spec:
accessModes:
- "ReadWriteOnce"
resources:
requests:
storage: "1Ti"
# Note: These two fields are not required for pre-bound PV.
storageClassName: pure-file
volumeMode: Filesystem
# TODO: Change to the name of the imported PV.
volumeName: pv-import
Notice the volumeName matches the PV we created earlier.
Now your Pod can mount the PVC. Even if it is already mounted. For that kind of multi-attach NFS is required.
Coming July 14 at 12 EST or 9am PST there will be a combined Kasten and Pure webinar about Kubernetes backup and mobility. As you are working on providing the expected levels of enterprise grade backup and recovery for you k8s based applications this will be a great webinar to help you learn more about what you can use to fill those requirements. Register here:
I do a lot of CLI demos when showing off PSO and K8s and Helm and all the cool things they can do. I made the switch to ZSH and oh my zsh and I really like using it for the past year.
After trying a few different themes and prompts I settled on the following using the “Agnoster” theme for oh-my-zsh. I added the kube-ps1 plugin so I can see what cluster I am using.
Pretty sweet prompt
One annoyance is getting when I am in some github repo for a demo deploying containers and ingresses and stuff the prompt gets so long the wordwrap on iTerm window makes me crazy.
I don’t even know what I am typing anymore
Break down of my full prompt
User name – good to know, I have cloud instances that are ‘ubuntu’ or ‘ec2-user’ so this helps me know what I am working on.
Hostname – I can shorten this, but there is a jamf policy on this machine that sets it back to this monstrosity. I mean, really who forces the username into the hostname?
Path to code – Do I need to see this full path all the time? Probably not.
Git Branch – Good to know when I am coding.
K8s Icon – Non-negotiable. It is awesome.
K8s Cluster/Context – I currently am testing 9 versions of K8s and different clusters. This keeps me sane.
K8s Namespace – This for me is nearly always default since I use kubectl -n <namespace> to do things in namespaces.
I saw someone with a different shell with the cool shorthand path and I felt like that should be an option for my prompt. I like everything else about the agnoster theme. So how can I fix it.
Of course it is a function that calls a bunch of others and “prompt_dir” is the one we want to edit.
Change it from this:
prompt_dir() {
prompt_segment blue $CURRENT_FG '%~'
}
To this:
prompt_dir() {
SHORT=$(shrink_path -f)
prompt_segment blue $CURRENT_FG $SHORT
}
Now we are using “shrink_path -f” to shorten the directories in the path and storing it in the SHORT variable. Then displaying it in the same spot as full path was before.
Look at how that path was shortened! So far this is working for me on:
Every so ofter I get in a situation where I need to see if my configuration is still valid for every cluster in my Kubernetes config. I blogged a while back about managing multiple configs. Then, eventually, your K8s config ends up looking like this:
Some k8s clustersThen TKG comes and helps it multiply
It gets tedious to go through every config like this:
kubectl config use-context <the next context>
kubectl get pod
<blah blah>
Through a little googling I now use this command to go through each cluster and run a command.
kubectl config get-contexts -o name | xargs -I {} kubectl --context={} get nodes -o wide
Now I can quickly see if a cluster fails to authenticate because I did something like rebuilt it and didn’t update the configuration on my local machine.
Another favorite of mine is:
kubectl config get-contexts -o name | xargs -I {} kubectl --context={} get pvc -A
Since I have to check all the storage on each cluster.
PVC’s for days
Yes, there is a theme to my cluster names and users.
TL;DR – Move Kubernetes volumes from legacy storage to Pure Storage.
So you have an amazing new Pure Storage array in the datacenter or in public cloud. The Container Storage Interface doesn’t provide a built in way to migrate data between backend devices. I previously blogged about a few ways to clone and migrate data between clusters but the data has to already be located on a Pure FlashArray.
Lately, Pure has been working with a new partner Kasten. While more is yet to come. Check out this demo (just 5:30) and see just how easy it is to move PVC’s while maintaining the config of the rest of the k8s application.
Demo EBS to CBS (this could be used to migrate off other devices too)
This demo used EKS in AWS for the Kubernetes cluster.
Application initially installed using a PVC for MySQL on EBS.
Kasten is used to backup the entire state of the app with the PVC to S3. This target could be a FlashBlade in your datacenter.
The application is restored to the same namespace but a Kasten Transform is used to convert the PVC to the “pure-block” StorageClass.
Application is live and using PSO for the storage on Cloud Block Store.
Why
Like the book says, “End with why”. Ok maybe it doesn’t actually say that. Let’s answer the “why should I do this?”
First: Why move EBS to CBS This PVC is 10GB on EBS. At this point in time it consumes about 30MB. How much does the AWS bill on the 10GB EBS volume? 10GB. On Cloud Block Store this data is reduced (compressed and deduped) and thin provisioned. How much is on the CBS? 3MB in this case. Does this make sense for 1 or 2 volumes? Nope. If your CIO has stated “move it all to the cloud!” This can be a significant savings on overall storage cost.
Second: Why move from (some other thing) to Pure? I am biased to PSO for Kubernetes so I will start there and then give a few bullets of why Pure, but this isn’t the sales pitch blog. Pure Service Orchestrator allows you a simple single line to install and begin getting storage on demand for your container clusters. One customer says, “It just works, we kind of forget it is there.” and another commented, “I want 100GB of storage for my app, and everything else is automated for me.”
Why Pure?
Efficiency – Get more out of the all-flash, higher dedupe with no performance penalty does matter.
Availability – 6×9’s uptime measured across our customer base, not an array in a validation lab. Actual customers love us.
TL;DR – EBS Volumes fail to mount when multipathd is installed on EKS worker nodes.
EKS and PSO Go Great together!
AWS Elastic Kubernetes Service is a great way to dive in with managed Kubernetes in the cloud. Pure Service Orchestrator integrates EKS worker nodes into the Cloud Block Store on AWS. I created this ansible playbook to make sure the right packages and services are started on my worker nodes.
In my previous testing with PSO and EKS I was basically focused on using PSO only. Recently the use case of migrating from EBS to CBS has shown to be pretty valuable to our customers in the cloud. To create the demo I used an app I often use for demoing PSO. It is 2 Web server containers attached to a mySQL container with a persistent volume. Very easy. I noticed though as I was using the built in gp2 Storage Class it started behaving super odd after I installed PSO. I installed the AWS EBS CSI driver. Same thing. It could not mount volumes or snapshot them in EBS. PSO volumes on CBS worked just fine. I figure most customers don’t want me to break EBS.
After digging around the internet and random old Github issues there was no one thing seemingly having the same issue. People were having problems that had like 1 of the 4 symptoms. I decided to test when in my process it broke after I enabled the package device-mapper-multipath. So it wasn’t PSO as much as a very important pre-requisite to PSO causing the issue. What it came down to is the EBS volumes were getting grabbed by multipathd and the Storage Class didn’t know how to handle the different device names. So I had to find how to use multipathd for just the Pure volumes. The right settings in multipath.conf solved this. This is what I used as an example:
I am telling multipathd to ignore everything BUT Pure. This solved my issue. So I saved this into the local directory and added the section in the ansible playbook to copy that file to each worker node in EKS. 1. Copy the ansible playbook above to a file prereqs.yaml 2. Copy the above multipath blacklist settings to multipath.conf and save to the same directory as prereqs.yaml 3. Run the ansible playbook as shown below. (make sure the inventory.ini has IP’s and you have the SSH key to login to each worker node.
# Make sure inventory.ini has the ssh IP's of each node.
# prereqs.yaml includes the content from above
ansible-playbook -i inventory.ini -b -v prereqs.yaml -u ec2-user
This will install the packages, copy multipath.conf to /etc and restart the services to make sure they pick up the new config.
