started: (0/1/1) "[sig-storage] PersistentVolumes-local [Volume type: dir] Two pods mounting a local volume at the same time should be able to write from pod1 and read from pod2 [Suite:openshift/conformance/parallel] [Suite:k8s]" I0615 18:03:16.305531 671230 test_context.go:423] Tolerating taints "node-role.kubernetes.io/master" when considering if nodes are ready Jun 15 18:03:16.334: INFO: Waiting up to 30m0s for all (but 100) nodes to be schedulable Jun 15 18:03:18.263: INFO: Waiting up to 10m0s for all pods (need at least 0) in namespace 'kube-system' to be running and ready Jun 15 18:03:18.821: INFO: 0 / 0 pods in namespace 'kube-system' are running and ready (0 seconds elapsed) Jun 15 18:03:18.821: INFO: expected 0 pod replicas in namespace 'kube-system', 0 are Running and Ready. Jun 15 18:03:18.821: INFO: Waiting up to 5m0s for all daemonsets in namespace 'kube-system' to start Jun 15 18:03:19.003: INFO: e2e test version: v1.18.3 Jun 15 18:03:19.174: INFO: kube-apiserver version: v1.18.3+a637491 Jun 15 18:03:19.389: INFO: Cluster IP family: ipv4 [BeforeEach] [Top Level] /home/fidencio/src/upstream/origin/_output/local/go/src/github.com/openshift/origin/test/extended/util/framework.go:1413 [BeforeEach] [Top Level] /home/fidencio/src/upstream/origin/_output/local/go/src/github.com/openshift/origin/test/extended/util/framework.go:1413 [BeforeEach] [Top Level] /home/fidencio/src/upstream/origin/_output/local/go/src/github.com/openshift/origin/test/extended/util/test.go:58 [BeforeEach] [sig-storage] PersistentVolumes-local /home/fidencio/src/upstream/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:178 STEP: Creating a kubernetes client STEP: Building a namespace api object, basename persistent-local-volumes-test Jun 15 18:03:20.045: INFO: About to run a Kube e2e test, ensuring namespace is privileged Jun 15 18:03:24.174: INFO: No PodSecurityPolicies found; assuming PodSecurityPolicy is disabled. STEP: Waiting for a default service account to be provisioned in namespace [BeforeEach] [sig-storage] PersistentVolumes-local /home/fidencio/src/upstream/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/storage/persistent_volumes-local.go:155 [BeforeEach] [Volume type: dir] /home/fidencio/src/upstream/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/storage/persistent_volumes-local.go:191 STEP: Initializing test volumes Jun 15 18:03:27.786: INFO: ExecWithOptions {Command:[nsenter --mount=/rootfs/proc/1/ns/mnt -- sh -c mkdir -p /tmp/local-volume-test-3c8d802b-b24c-4f69-a522-e1f40cb6ee47] Namespace:e2e-persistent-local-volumes-test-2956 PodName:hostexec-master-0-0-jvm8l ContainerName:agnhost Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true} STEP: Creating local PVCs and PVs Jun 15 18:03:29.059: INFO: Creating a PV followed by a PVC Jun 15 18:03:29.406: INFO: Waiting for PV local-pv9mtxs to bind to PVC pvc-clkq9 Jun 15 18:03:29.406: INFO: Waiting up to 3m0s for PersistentVolumeClaims [pvc-clkq9] to have phase Bound Jun 15 18:03:29.629: INFO: PersistentVolumeClaim pvc-clkq9 found and phase=Bound (222.033063ms) Jun 15 18:03:29.629: INFO: Waiting up to 3m0s for PersistentVolume local-pv9mtxs to have phase Bound Jun 15 18:03:29.831: INFO: PersistentVolume local-pv9mtxs found and phase=Bound (202.473035ms) [It] should be able to write from pod1 and read from pod2 [Suite:openshift/conformance/parallel] [Suite:k8s] /home/fidencio/src/upstream/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/storage/persistent_volumes-local.go:245 STEP: Creating pod1 to write to the PV STEP: Creating a pod Jun 15 18:03:43.112: INFO: pod "security-context-08c75167-fd6b-4ebf-853b-f9ff6bf5e232" created on Node "master-0-0" STEP: Writing in pod1 Jun 15 18:03:43.112: INFO: ExecWithOptions {Command:[/bin/sh -c mkdir -p /mnt/volume1; echo test-file-content > /mnt/volume1/test-file] Namespace:e2e-persistent-local-volumes-test-2956 PodName:security-context-08c75167-fd6b-4ebf-853b-f9ff6bf5e232 ContainerName:write-pod Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:false} Jun 15 18:03:44.577: INFO: podRWCmdExec out: "" err: Jun 15 18:03:44.577: INFO: ExecWithOptions {Command:[/bin/sh -c cat /mnt/volume1/test-file] Namespace:e2e-persistent-local-volumes-test-2956 PodName:security-context-08c75167-fd6b-4ebf-853b-f9ff6bf5e232 ContainerName:write-pod Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:false} Jun 15 18:03:45.703: INFO: podRWCmdExec out: "test-file-content" err: STEP: Creating pod2 to read from the PV STEP: Creating a pod Jun 15 18:04:00.690: INFO: pod "security-context-c5020d8f-90db-46b2-8c25-5bc8a23b6353" created on Node "master-0-0" Jun 15 18:04:00.691: INFO: ExecWithOptions {Command:[/bin/sh -c cat /mnt/volume1/test-file] Namespace:e2e-persistent-local-volumes-test-2956 PodName:security-context-c5020d8f-90db-46b2-8c25-5bc8a23b6353 ContainerName:write-pod Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:false} Jun 15 18:04:01.960: INFO: podRWCmdExec out: "test-file-content" err: STEP: Writing in pod2 Jun 15 18:04:01.960: INFO: ExecWithOptions {Command:[/bin/sh -c mkdir -p /mnt/volume1; echo /tmp/local-volume-test-3c8d802b-b24c-4f69-a522-e1f40cb6ee47 > /mnt/volume1/test-file] Namespace:e2e-persistent-local-volumes-test-2956 PodName:security-context-c5020d8f-90db-46b2-8c25-5bc8a23b6353 ContainerName:write-pod Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:false} Jun 15 18:04:03.190: INFO: podRWCmdExec out: "" err: STEP: Reading in pod1 Jun 15 18:04:03.190: INFO: ExecWithOptions {Command:[/bin/sh -c cat /mnt/volume1/test-file] Namespace:e2e-persistent-local-volumes-test-2956 PodName:security-context-08c75167-fd6b-4ebf-853b-f9ff6bf5e232 ContainerName:write-pod Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:false} Jun 15 18:04:04.488: INFO: podRWCmdExec out: "test-file-content" err: [AfterEach] [Volume type: dir] /home/fidencio/src/upstream/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/storage/persistent_volumes-local.go:200 STEP: Cleaning up PVC and PV Jun 15 18:04:04.490: INFO: Deleting PersistentVolumeClaim "pvc-clkq9" Jun 15 18:04:04.752: INFO: Deleting PersistentVolume "local-pv9mtxs" STEP: Removing the test directory Jun 15 18:04:04.942: INFO: ExecWithOptions {Command:[nsenter --mount=/rootfs/proc/1/ns/mnt -- sh -c rm -r /tmp/local-volume-test-3c8d802b-b24c-4f69-a522-e1f40cb6ee47] Namespace:e2e-persistent-local-volumes-test-2956 PodName:hostexec-master-0-0-jvm8l ContainerName:agnhost Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true} [AfterEach] [sig-storage] PersistentVolumes-local /home/fidencio/src/upstream/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:179 STEP: Collecting events from namespace "e2e-persistent-local-volumes-test-2956". STEP: Found 14 events. Jun 15 18:04:06.796: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for hostexec-master-0-0-jvm8l: {default-scheduler } Scheduled: Successfully assigned e2e-persistent-local-volumes-test-2956/hostexec-master-0-0-jvm8l to master-0-0 Jun 15 18:04:06.796: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for security-context-08c75167-fd6b-4ebf-853b-f9ff6bf5e232: {default-scheduler } Scheduled: Successfully assigned e2e-persistent-local-volumes-test-2956/security-context-08c75167-fd6b-4ebf-853b-f9ff6bf5e232 to master-0-0 Jun 15 18:04:06.796: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for security-context-c5020d8f-90db-46b2-8c25-5bc8a23b6353: {default-scheduler } Scheduled: Successfully assigned e2e-persistent-local-volumes-test-2956/security-context-c5020d8f-90db-46b2-8c25-5bc8a23b6353 to master-0-0 Jun 15 18:04:06.796: INFO: At 2020-06-15 18:03:25 +0200 CEST - event for hostexec-master-0-0-jvm8l: {kubelet master-0-0} Pulled: Container image "us.gcr.io/k8s-artifacts-prod/e2e-test-images/agnhost:2.12" already present on machine Jun 15 18:04:06.796: INFO: At 2020-06-15 18:03:26 +0200 CEST - event for hostexec-master-0-0-jvm8l: {kubelet master-0-0} Created: Created container agnhost Jun 15 18:04:06.796: INFO: At 2020-06-15 18:03:26 +0200 CEST - event for hostexec-master-0-0-jvm8l: {kubelet master-0-0} Started: Started container agnhost Jun 15 18:04:06.796: INFO: At 2020-06-15 18:03:32 +0200 CEST - event for security-context-08c75167-fd6b-4ebf-853b-f9ff6bf5e232: {multus } AddedInterface: Add eth0 [10.129.0.28/23] Jun 15 18:04:06.796: INFO: At 2020-06-15 18:03:40 +0200 CEST - event for security-context-08c75167-fd6b-4ebf-853b-f9ff6bf5e232: {kubelet master-0-0} Pulled: Container image "docker.io/library/busybox:1.29" already present on machine Jun 15 18:04:06.796: INFO: At 2020-06-15 18:03:41 +0200 CEST - event for security-context-08c75167-fd6b-4ebf-853b-f9ff6bf5e232: {kubelet master-0-0} Created: Created container write-pod Jun 15 18:04:06.796: INFO: At 2020-06-15 18:03:41 +0200 CEST - event for security-context-08c75167-fd6b-4ebf-853b-f9ff6bf5e232: {kubelet master-0-0} Started: Started container write-pod Jun 15 18:04:06.796: INFO: At 2020-06-15 18:03:48 +0200 CEST - event for security-context-c5020d8f-90db-46b2-8c25-5bc8a23b6353: {multus } AddedInterface: Add eth0 [10.129.0.29/23] Jun 15 18:04:06.796: INFO: At 2020-06-15 18:03:57 +0200 CEST - event for security-context-c5020d8f-90db-46b2-8c25-5bc8a23b6353: {kubelet master-0-0} Pulled: Container image "docker.io/library/busybox:1.29" already present on machine Jun 15 18:04:06.796: INFO: At 2020-06-15 18:03:57 +0200 CEST - event for security-context-c5020d8f-90db-46b2-8c25-5bc8a23b6353: {kubelet master-0-0} Created: Created container write-pod Jun 15 18:04:06.796: INFO: At 2020-06-15 18:03:57 +0200 CEST - event for security-context-c5020d8f-90db-46b2-8c25-5bc8a23b6353: {kubelet master-0-0} Started: Started container write-pod Jun 15 18:04:07.004: INFO: POD NODE PHASE GRACE CONDITIONS Jun 15 18:04:07.004: INFO: hostexec-master-0-0-jvm8l master-0-0 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2020-06-15 18:03:25 +0200 CEST } {Ready True 0001-01-01 00:00:00 +0000 UTC 2020-06-15 18:03:27 +0200 CEST } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2020-06-15 18:03:27 +0200 CEST } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2020-06-15 18:03:25 +0200 CEST }] Jun 15 18:04:07.005: INFO: security-context-08c75167-fd6b-4ebf-853b-f9ff6bf5e232 master-0-0 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2020-06-15 18:03:30 +0200 CEST } {Ready True 0001-01-01 00:00:00 +0000 UTC 2020-06-15 18:03:42 +0200 CEST } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2020-06-15 18:03:42 +0200 CEST } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2020-06-15 18:03:30 +0200 CEST }] Jun 15 18:04:07.005: INFO: security-context-c5020d8f-90db-46b2-8c25-5bc8a23b6353 master-0-0 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2020-06-15 18:03:45 +0200 CEST } {Ready True 0001-01-01 00:00:00 +0000 UTC 2020-06-15 18:03:58 +0200 CEST } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2020-06-15 18:03:58 +0200 CEST } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2020-06-15 18:03:45 +0200 CEST }] Jun 15 18:04:07.005: INFO: Jun 15 18:04:07.182: INFO: hostexec-master-0-0-jvm8l[e2e-persistent-local-volumes-test-2956].container[agnhost].log Paused Jun 15 18:04:07.412: INFO: security-context-08c75167-fd6b-4ebf-853b-f9ff6bf5e232[e2e-persistent-local-volumes-test-2956].container[write-pod].log Jun 15 18:04:07.616: INFO: security-context-c5020d8f-90db-46b2-8c25-5bc8a23b6353[e2e-persistent-local-volumes-test-2956].container[write-pod].log Jun 15 18:04:08.161: INFO: skipping dumping cluster info - cluster too large Jun 15 18:04:08.161: INFO: Waiting up to 7m0s for all (but 100) nodes to be ready STEP: Destroying namespace "e2e-persistent-local-volumes-test-2956" for this suite. Jun 15 18:04:08.818: INFO: Running AfterSuite actions on all nodes Jun 15 18:04:08.818: INFO: Running AfterSuite actions on node 1 fail [k8s.io/kubernetes/test/e2e/storage/persistent_volumes-local.go:1035]: Expected : test-file-content to contain substring : /tmp/local-volume-test-3c8d802b-b24c-4f69-a522-e1f40cb6ee47 failed: (52.7s) 2020-06-15T16:04:08 "[sig-storage] PersistentVolumes-local [Volume type: dir] Two pods mounting a local volume at the same time should be able to write from pod1 and read from pod2 [Suite:openshift/conformance/parallel] [Suite:k8s]" Timeline: Jun 15 18:03:16.170 - 52s I test="[sig-storage] PersistentVolumes-local [Volume type: dir] Two pods mounting a local volume at the same time should be able to write from pod1 and read from pod2 [Suite:openshift/conformance/parallel] [Suite:k8s]" running Jun 15 16:03:25.349 I ns/e2e-persistent-local-volumes-test-2956 pod/hostexec-master-0-0-jvm8l node/ reason/Created Jun 15 16:03:25.499 I ns/e2e-persistent-local-volumes-test-2956 pod/hostexec-master-0-0-jvm8l node/master-0-0 reason/Scheduled Jun 15 16:03:26.047 I ns/e2e-persistent-local-volumes-test-2956 pod/hostexec-master-0-0-jvm8l node/master-0-0 container/agnhost reason/Pulled image/us.gcr.io/k8s-artifacts-prod/e2e-test-images/agnhost:2.12 Jun 15 16:03:26.225 I ns/e2e-persistent-local-volumes-test-2956 pod/hostexec-master-0-0-jvm8l node/master-0-0 container/agnhost reason/Created Jun 15 16:03:26.261 I ns/e2e-persistent-local-volumes-test-2956 pod/hostexec-master-0-0-jvm8l node/master-0-0 container/agnhost reason/Started Jun 15 16:03:27.187 I ns/e2e-persistent-local-volumes-test-2956 pod/hostexec-master-0-0-jvm8l node/master-0-0 container/agnhost reason/Ready Jun 15 16:03:30.353 I ns/e2e-persistent-local-volumes-test-2956 pod/security-context-08c75167-fd6b-4ebf-853b-f9ff6bf5e232 node/ reason/Created Jun 15 16:03:30.550 I ns/e2e-persistent-local-volumes-test-2956 pod/security-context-08c75167-fd6b-4ebf-853b-f9ff6bf5e232 node/master-0-0 reason/Scheduled Jun 15 16:03:31.170 - 30s W ns/default pod/recycler-for-nfs-pv10 node/worker-0-0 pod has been pending longer than a minute Jun 15 16:03:32.999 I ns/e2e-persistent-local-volumes-test-2956 pod/security-context-08c75167-fd6b-4ebf-853b-f9ff6bf5e232 reason/AddedInterface Add eth0 [10.129.0.28/23] Jun 15 16:03:40.946 I ns/e2e-persistent-local-volumes-test-2956 pod/security-context-08c75167-fd6b-4ebf-853b-f9ff6bf5e232 node/master-0-0 container/write-pod reason/Pulled image/docker.io/library/busybox:1.29 Jun 15 16:03:41.328 I ns/e2e-persistent-local-volumes-test-2956 pod/security-context-08c75167-fd6b-4ebf-853b-f9ff6bf5e232 node/master-0-0 container/write-pod reason/Created Jun 15 16:03:41.334 I ns/e2e-persistent-local-volumes-test-2956 pod/security-context-08c75167-fd6b-4ebf-853b-f9ff6bf5e232 node/master-0-0 container/write-pod reason/Started Jun 15 16:03:42.270 I ns/e2e-persistent-local-volumes-test-2956 pod/security-context-08c75167-fd6b-4ebf-853b-f9ff6bf5e232 node/master-0-0 container/write-pod reason/Ready Jun 15 16:03:43.178 E ns/openshift-cluster-storage-operator pod/csi-snapshot-controller-654675b99d-kds5h node/worker-0-0 container/snapshot-controller container exited with code 255 (Error): Jun 15 16:03:43.223 W clusteroperator/csi-snapshot-controller changed Available to False: _AsExpected: Available: Waiting for Deployment to deploy csi-snapshot-controller pods Jun 15 16:03:43.223 W clusteroperator/csi-snapshot-controller changed Progressing to True: _AsExpected: Progressing: Waiting for Deployment to deploy csi-snapshot-controller pods Jun 15 16:03:45.910 I ns/e2e-persistent-local-volumes-test-2956 pod/security-context-c5020d8f-90db-46b2-8c25-5bc8a23b6353 node/ reason/Created Jun 15 16:03:45.911 I ns/e2e-persistent-local-volumes-test-2956 pod/security-context-c5020d8f-90db-46b2-8c25-5bc8a23b6353 node/master-0-0 reason/Scheduled Jun 15 16:03:48.572 I ns/e2e-persistent-local-volumes-test-2956 pod/security-context-c5020d8f-90db-46b2-8c25-5bc8a23b6353 reason/AddedInterface Add eth0 [10.129.0.29/23] Jun 15 16:03:57.575 I ns/e2e-persistent-local-volumes-test-2956 pod/security-context-c5020d8f-90db-46b2-8c25-5bc8a23b6353 node/master-0-0 container/write-pod reason/Pulled image/docker.io/library/busybox:1.29 Jun 15 16:03:57.752 W ns/default pod/recycler-for-nfs-pv10 node/worker-0-0 reason/FailedMount (combined from similar events): Unable to attach or mount volumes: unmounted volumes=[vol], unattached volumes=[default-token-w29vn vol]: timed out waiting for the condition (4 times) Jun 15 16:03:58.032 I ns/e2e-persistent-local-volumes-test-2956 pod/security-context-c5020d8f-90db-46b2-8c25-5bc8a23b6353 node/master-0-0 container/write-pod reason/Created Jun 15 16:03:58.036 I ns/e2e-persistent-local-volumes-test-2956 pod/security-context-c5020d8f-90db-46b2-8c25-5bc8a23b6353 node/master-0-0 container/write-pod reason/Started Jun 15 16:03:58.340 I ns/e2e-persistent-local-volumes-test-2956 pod/security-context-c5020d8f-90db-46b2-8c25-5bc8a23b6353 node/master-0-0 container/write-pod reason/Ready Jun 15 16:03:59.266 W ns/openshift-operator-lifecycle-manager pod/packageserver-84cd46d9b4-2slc2 node/master-0-1 reason/Unhealthy Liveness probe failed: Get https://10.130.0.11:5443/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers) (710 times) Jun 15 16:03:59.356 I ns/openshift-config-operator deployment/openshift-config-operator reason/KubeCloudConfigController openshift-config-managed/kube-cloud-config ConfigMap was deleted as no longer required (3936 times) Jun 15 16:04:00.431 W ns/default pod/recycler-for-nfs-pv10 node/worker-0-0 reason/FailedMount (combined from similar events): MountVolume.SetUp failed for volume "vol" : mount failed: exit status 32\nMounting command: systemd-run\nMounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/bb05dfa3-c60d-4864-b07b-543162357d04/volumes/kubernetes.io~nfs/vol --scope -- mount -t nfs registry.kata-fidencio-0.qe.lab.redhat.com:/mnt/pv10 /var/lib/kubelet/pods/bb05dfa3-c60d-4864-b07b-543162357d04/volumes/kubernetes.io~nfs/vol\nOutput: Running scope as unit: run-r833aa127ec0a4723a47a1d656ba80f80.scope\nmount.nfs: mounting registry.kata-fidencio-0.qe.lab.redhat.com:/mnt/pv10 failed, reason given by server: No such file or directory\n (5 times) Jun 15 18:04:08.832 I test="[sig-storage] PersistentVolumes-local [Volume type: dir] Two pods mounting a local volume at the same time should be able to write from pod1 and read from pod2 [Suite:openshift/conformance/parallel] [Suite:k8s]" failed Failing tests: [sig-storage] PersistentVolumes-local [Volume type: dir] Two pods mounting a local volume at the same time should be able to write from pod1 and read from pod2 [Suite:openshift/conformance/parallel] [Suite:k8s] error: 1 fail, 0 pass, 0 skip (52.7s)