Namespace Issues when Removing CRD/Operators

With the latest release of Pure Service Orchestrator, we added support for a non-Helm installation for environments that do not allow Helm. This new method uses an Operator to setup and install PSO. The result is the same exact functionality but uses a security model more agreeable to some K8s distro vendors.

I do live demos of PSO a handful of times a day. Even though I use Terraform and Ansible to automate the creation of my lab K8s clusters I don’t want to do this many times a day. I usually just tear down PSO and leave my cluster ready for the next demo.

Removing the CRD and the Namespace created when installing the Operator has a couple of issues. One small issue is the Operator method creates a new namespace “pso-operator”. This is the default name, and you can choose your own namespace name during install time. I often choose “pso” for simplicity. As we have discovered, deleting a namespace that had a CRD installed into hangs in the status “Terminating”, for like, forever. FOR-EV-ER. This seems to be an issue dating back quite a ways in K8s land.

https://github.com/kubernetes/kubernetes/issues/60807#issuecomment-448120772

via GIPHY

From a couple of GitHub issues and the help of Simon “I don’t do the twitter” Dodsley This is the process for deleting the CRD first and the Namespace. This method keeps the namespace form hanging in the state “Terminating”.

# Removing the pso-operator
kubectl delete all --all -n pso-operator

# If you haven't don't it already don't delete the namespace yet.
kubectl get ns
NAME          STATUS   AGE
default       Active   2d21h
kube-public   Active   2d21h
kube-system   Active   2d21h
pso-operator  Active   14h

kubectl get crd
NAME                         CREATED AT
psoplugins.purestorage.com   2019-04-17T01:37:31Z

# ok so...
kubectl delete crd psoplugins.purestorage.com
customresourcedefinition.apiextensions.k8s.io "psoplugins.purestorage.com" deleted

# does it hang? yeah it does
^C
# stuck terminating? 
kubectl describe crd psoplugins.purestorage.com
# snipping non-relevant output
...
Conditions:
    Last Transition Time:  2019-04-17T01:37:31Z
    Message:               no conflicts found
    Reason:                NoConflicts
    Status:                True
    Type:                  NamesAccepted
    Last Transition Time:  <nil>
    Message:               the initial names have been accepted
    Reason:                InitialNamesAccepted
    Status:                True
    Type:                  Established
    Last Transition Time:  2019-04-18T13:54:36Z
    Message:               CustomResource deletion is in progress
    Reason:                InstanceDeletionInProgress
    Status:                True
    Type:                  Terminating
  Stored Versions:
    v1

# Run this command to allow it to delete
kubectl patch crd/psoplugins.purestorage.com -p '{"metadata":{"finalizers":[]}}' --type=merge
customresourcedefinition.apiextensions.k8s.io/psoplugins.purestorage.com patched

# Re-run the crd delete
kubectl delete crd psoplugins.purestorage.com

# Confirm it is gone
kubectl get crd
No resources found.

# Remove the Namespace
kubectl delete ns pso-operator
namespace "pso-operator" deleted

#Verify removal
kubectl get ns
NAME          STATUS   AGE
default       Active   2d21h
kube-public   Active   2d21h
kube-system   Active   2d21h

If you sort of ignored my warning above and tried to remove the namespace BEFORE successfully removing the CRD follow the following procedure.

Namespace Removal

# Find that pesky 'Terminating' namespace
kubectl get ns
NAME           STATUS        AGE
default        Active        2d20h
kube-public    Active        2d20h
kube-system    Active        2d20h
pso            Active        13h
pso-operator   Terminating   35h

kubectl cluster-info
# run the kube-proxy
kubectl proxy &

# output the namespace to json
kubectl get namespace pso-operator -o json >tmp.json

# Edit the tmp.json to remove the finalizer the spec: should look like this:
"spec": {
        "finalizers": [
        ]
    },

# Now send that tmp.json to the API server
curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json http://127.0.0.1:8001/api/v1/namespaces/pso-operator/finalize

# Check your namespaces
kubectl get ns
NAME          STATUS   AGE
default       Active   2d20h
kube-public   Active   2d20h
kube-system   Active   2d20h
pso           Active   13h

# disable the kube-proxy, bring it back to the foreground and ctrl-C

fg
^C