Issues connecting to consul installed via Helm

I have a Kubernetes cluster that was created using KOPS which I have consul installed on via the helm chart with default values except server count and bootstrapexpect are both set to 1 because of this being a small test cluster - 2 workers and single master

I have a Node.js application that I am trying to connect to Consul. Below are the logs for what I think is relavent information, if I am missing something useful ask and I can provide it likely.

​

Error message given by my applcation seen by doing kubectl logs <pod>

{"message":"Error: Error: connect ECONNREFUSED 10.0.32.42:8500","level":"error"}
/opt/node_app/app/middleware/authorization.js:13
throw err;
^

Error: connect ECONNREFUSED 10.0.32.42:8500
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1128:14) {
errno: 'ECONNREFUSED',
code: 'ECONNREFUSED',
syscall: 'connect',
address: '10.0.32.42',
port: 8500
}

kubectl describe consul server 0

I'm aware this one is showing rediness probe failed however I can't figure out why

Name: consul-consul-server-0
Namespace: development
Priority: 0
Node: ip-10-0-32-42.ec2.internal/10.0.32.42
Start Time: Wed, 18 Dec 2019 19:32:24 -0500
Labels: app=consul
chart=consul-helm
component=server
controller-revision-hash=consul-consul-server-6c689958b9
hasDNS=true
release=consul
statefulset.kubernetes.io/pod-name=consul-consul-server-0
Annotations: consul.hashicorp.com/connect-inject: false
Status: Running
IP: 100.96.21.3
Controlled By: StatefulSet/consul-consul-server
Containers:
consul:
Container ID: docker://05acd44001b0702aa7219f6c5b2217ab3d9615b5b33b363720c98bc0b8c87ce9
Image: consul:1.6.2
Image ID: docker-pullable://consul@sha256:a167e7222c84687c3e7f392f13b23d9f391cac80b6b839052e58617dab714805
Ports: 8500/TCP, 8301/TCP, 8302/TCP, 8300/TCP, 8600/TCP, 8600/UDP
Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/UDP
Command:
/bin/sh
-ec
CONSUL_FULLNAME="consul-consul"

exec /bin/consul agent \
-advertise="${POD_IP}" \
-bind=0.0.0.0 \
-bootstrap-expect=1 \
-client=0.0.0.0 \
-config-dir=/consul/config \
-datacenter=dc1 \
-data-dir=/consul/data \
-domain=consul \
-hcl="connect { enabled = true }" \
-ui \
-retry-join=${CONSUL_FULLNAME}-server-0.${CONSUL_FULLNAME}-server.${NAMESPACE}.svc \
-server

State: Running
Started: Wed, 18 Dec 2019 19:32:33 -0500
Ready: True
Restart Count: 0
Readiness: exec [/bin/sh -ec curl http://127.0.0.1:8500/v1/status/leader 2>/dev/null | \
grep -E '".+"'
] delay=5s timeout=5s period=3s #success=1 #failure=2
Environment:
POD_IP: (v1:status.podIP)
NAMESPACE: development (v1:metadata.namespace)
Mounts:
/consul/config from config (rw)
/consul/data from data-development (rw)
/var/run/secrets/kubernetes.io/serviceaccount from consul-consul-server-token-42ss6 (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
data-development:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data-development-consul-consul-server-0
ReadOnly: false
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: consul-consul-server-config
Optional: false
consul-consul-server-token-

42ss6:
Type: Secret (a volume populated by a Secret)
SecretName: consul-consul-server-token-42ss6
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 5m53s (x7 over 6m9s) default-scheduler pod has unbound immediate PersistentVolumeClaims (repeated 2 times)
Normal Scheduled 5m53s default-scheduler Successfully assigned development/consul-consul-server-0 to ip-10-0-32-42.ec2.internal
Warning FailedAttachVolume 5m51s (x3 over 5m53s) attachdetach-controller AttachVolume.Attach failed for volume "pvc-fd6b0c40-21f6-11ea-9540-020941ae37b7" : "Error attaching EBS volume \"vol-0ff35f1bebdbe2330\"" to instance "i-03ee444d8bb8a0a88" since volume is in "creating" state
Normal SuccessfulAttachVolume 5m47s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-fd6b0c40-21f6-11ea-9540-020941ae37b7"
Normal Pulled 5m44s kubelet, ip-10-0-32-42.ec2.internal Container image "consul:1.6.2" already present on machine
Normal Created 5m44s kubelet, ip-10-0-32-42.ec2.internal Created container
Normal Started 5m44s kubelet, ip-10-0-32-42.ec2.internal Started container
Warning Unhealthy 5m37s kubelet, ip-10-0-32-42.ec2.internal Readiness probe failed:


&#x200B;

kubectl describe on a agent

Name: consul-consul-hzndl
Namespace: development
Priority: 0
Node: ip-10-0-32-42.ec2.internal/10.0.32.42
Start Time: Wed, 18 Dec 2019 19:32:08 -0500
Labels: app=consul
chart=consul-helm
component=client
controller-revision-hash=5db9fb58cd
hasDNS=true
pod-template-generation=1
release=consul
Annotations: consul.hashicorp.com/connect-inject: false
Status: Running
IP: 100.96.21.2
Controlled By: DaemonSet/consul-consul
Containers:
consul:
Container ID: docker://67cc1d0ae1698732cb944373c79e8685a51ba3899c4ca9d52a9ae934c7f2e479
Image: consul:1.6.2
Image ID: docker-pullable://consul@sha256:a167e7222c84687c3e7f392f13b23d9f391cac80b6b839052e58617dab714805
Ports: 8500/TCP, 8502/TCP, 8301/TCP, 8301/UDP, 8302/TCP, 8300/TCP, 8600/TCP, 8600/UDP
Host Ports: 8500/TCP, 8502/TCP, 0/TCP, 0/UDP, 0/TCP, 0/TCP, 0/TCP, 0/UDP
Command:
/bin/sh
-ec
CONSUL_FULLNAME="consul-consul"

exec /bin/consul agent \
-node="${NODE}" \
-advertise="${ADVERTISE_IP}" \
-bind=0.0.0.0 \
-client=0.0.0.0 \
-hcl="ports { grpc = 8502 }" \
-config-dir=/consul/config \
-datacenter=dc1 \
-data-dir=/consul/data \
-retry-join=${CONSUL_FULLNAME}-server-0.${CONSUL_FULLNAME}-server.${NAMESPACE}.svc \
-domain=consul

State: Running
Started: Wed, 18 Dec 2019 19:32:12 -0500
Ready: True
Restart Count: 0
Readiness: exec [/bin/sh -ec curl http://127.0.0.1:8500/v1/status/leader 2>/dev/null | \
grep -E '".+"'
] delay=0s timeout=1s period=10s #success=1 #failure=3
Environment:
ADVERTISE_IP: (v1:status.podIP)
NAMESPACE: development (v1:metadata.namespace)