Using Node, Git and SFTP for automated deployments

I plan to write a Node script that finds the changed files with Git and then deploys them to the server with SFTP. Is this an okay approach?

https://redd.it/ecokx7
@r_devops

I Want to learn Linux Administration

Hi everyone

I would like to learn Linux administration. i am very new to this.. I have already started with Linux Bible Ninth Edition. In the end I would like to get certified.

Can the experts in the community can help me as in any other books should i refer to.

​

Thank you

Sumesh

https://redd.it/ecohy7
@r_devops

Silly question: What exactly should I do if my build fails in CI/CD process?

So I am a newbie who implemented CI/CD using github actions for a node project. I pushed code which will fail tests and cause a build failure on push to github but the problem is my changes are committed to the branch even if the build failed. Is this expected behavior? How should I handle my build failures?

https://redd.it/efmtep
@r_devops

Issues connecting to consul installed via Helm

I have a Kubernetes cluster that was created using KOPS which I have consul installed on via the helm chart with default values except server count and bootstrapexpect are both set to 1 because of this being a small test cluster - 2 workers and single master

I have a Node.js application that I am trying to connect to Consul. Below are the logs for what I think is relavent information, if I am missing something useful ask and I can provide it likely.

​

Error message given by my applcation seen by doing kubectl logs <pod>

{"message":"Error: Error: connect ECONNREFUSED 10.0.32.42:8500","level":"error"}
/opt/node_app/app/middleware/authorization.js:13
throw err;
^

Error: connect ECONNREFUSED 10.0.32.42:8500
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1128:14) {
errno: 'ECONNREFUSED',
code: 'ECONNREFUSED',
syscall: 'connect',
address: '10.0.32.42',
port: 8500
}

kubectl describe consul server 0

I'm aware this one is showing rediness probe failed however I can't figure out why

Name: consul-consul-server-0
Namespace: development
Priority: 0
Node: ip-10-0-32-42.ec2.internal/10.0.32.42
Start Time: Wed, 18 Dec 2019 19:32:24 -0500
Labels: app=consul
chart=consul-helm
component=server
controller-revision-hash=consul-consul-server-6c689958b9
hasDNS=true
release=consul
statefulset.kubernetes.io/pod-name=consul-consul-server-0
Annotations: consul.hashicorp.com/connect-inject: false
Status: Running
IP: 100.96.21.3
Controlled By: StatefulSet/consul-consul-server
Containers:
consul:
Container ID: docker://05acd44001b0702aa7219f6c5b2217ab3d9615b5b33b363720c98bc0b8c87ce9
Image: consul:1.6.2
Image ID: docker-pullable://consul@sha256:a167e7222c84687c3e7f392f13b23d9f391cac80b6b839052e58617dab714805
Ports: 8500/TCP, 8301/TCP, 8302/TCP, 8300/TCP, 8600/TCP, 8600/UDP
Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/UDP
Command:
/bin/sh
-ec
CONSUL_FULLNAME="consul-consul"

exec /bin/consul agent \
-advertise="${POD_IP}" \
-bind=0.0.0.0 \
-bootstrap-expect=1 \
-client=0.0.0.0 \
-config-dir=/consul/config \
-datacenter=dc1 \
-data-dir=/consul/data \
-domain=consul \
-hcl="connect { enabled = true }" \
-ui \
-retry-join=${CONSUL_FULLNAME}-server-0.${CONSUL_FULLNAME}-server.${NAMESPACE}.svc \
-server

State: Running
Started: Wed, 18 Dec 2019 19:32:33 -0500
Ready: True
Restart Count: 0
Readiness: exec [/bin/sh -ec curl https://127.0.0.1:8500/v1/status/leader 2>/dev/null | \
grep -E '".+"'
] delay=5s timeout=5s period=3s #success=1 #failure=2
Environment:
POD_IP: (v1:status.podIP)
NAMESPACE: development (v1:metadata.namespace)
Mounts:
/consul/config from config (rw)
/consul/data from data-development (rw)
/var/run/secrets/kubernetes.io/serviceaccount from consul-consul-server-token-42ss6 (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
data-development:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data-development-consul-consul-server-0
ReadOnly: false
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: consul-consul-server-config
Optional: false
consul-consul-server-token-

42ss6:
Type: Secret (a volume populated by a Secret)
SecretName: consul-consul-server-token-42ss6
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 5m53s (x7 over 6m9s) default-scheduler pod has unbound immediate PersistentVolumeClaims (repeated 2 times)
Normal Scheduled 5m53s default-scheduler Successfully assigned development/consul-consul-server-0 to ip-10-0-32-42.ec2.internal
Warning FailedAttachVolume 5m51s (x3 over 5m53s) attachdetach-controller AttachVolume.Attach failed for volume "pvc-fd6b0c40-21f6-11ea-9540-020941ae37b7" : "Error attaching EBS volume \"vol-0ff35f1bebdbe2330\"" to instance "i-03ee444d8bb8a0a88" since volume is in "creating" state
Normal SuccessfulAttachVolume 5m47s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-fd6b0c40-21f6-11ea-9540-020941ae37b7"
Normal Pulled 5m44s kubelet, ip-10-0-32-42.ec2.internal Container image "consul:1.6.2" already present on machine
Normal Created 5m44s kubelet, ip-10-0-32-42.ec2.internal Created container
Normal Started 5m44s kubelet, ip-10-0-32-42.ec2.internal Started container
Warning Unhealthy 5m37s kubelet, ip-10-0-32-42.ec2.internal Readiness probe failed:


&#x200B;

kubectl describe on a agent

Name: consul-consul-hzndl
Namespace: development
Priority: 0
Node: ip-10-0-32-42.ec2.internal/10.0.32.42
Start Time: Wed, 18 Dec 2019 19:32:08 -0500
Labels: app=consul
chart=consul-helm
component=client
controller-revision-hash=5db9fb58cd
hasDNS=true
pod-template-generation=1
release=consul
Annotations: consul.hashicorp.com/connect-inject: false
Status: Running
IP: 100.96.21.2
Controlled By: DaemonSet/consul-consul
Containers:
consul:
Container ID: docker://67cc1d0ae1698732cb944373c79e8685a51ba3899c4ca9d52a9ae934c7f2e479
Image: consul:1.6.2
Image ID: docker-pullable://consul@sha256:a167e7222c84687c3e7f392f13b23d9f391cac80b6b839052e58617dab714805
Ports: 8500/TCP, 8502/TCP, 8301/TCP, 8301/UDP, 8302/TCP, 8300/TCP, 8600/TCP, 8600/UDP
Host Ports: 8500/TCP, 8502/TCP, 0/TCP, 0/UDP, 0/TCP, 0/TCP, 0/TCP, 0/UDP
Command:
/bin/sh
-ec
CONSUL_FULLNAME="consul-consul"

exec /bin/consul agent \
-node="${NODE}" \
-advertise="${ADVERTISE_IP}" \
-bind=0.0.0.0 \
-client=0.0.0.0 \
-hcl="ports { grpc = 8502 }" \
-config-dir=/consul/config \
-datacenter=dc1 \
-data-dir=/consul/data \
-retry-join=${CONSUL_FULLNAME}-server-0.${CONSUL_FULLNAME}-server.${NAMESPACE}.svc \
-domain=consul

State: Running
Started: Wed, 18 Dec 2019 19:32:12 -0500
Ready: True
Restart Count: 0
Readiness: exec [/bin/sh -ec curl https://127.0.0.1:8500/v1/status/leader 2>/dev/null | \
grep -E '".+"'
] delay=0s timeout=1s period=10s #success=1 #failure=3
Environment:
ADVERTISE_IP: (v1:status.podIP)
NAMESPACE: development (v1:metadata.namespace)

NODE: (v1:spec.nodeName)
Mounts:
/consul/config from config (rw)
/consul/data from data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from consul-consul-client-token-xp5q9 (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
data:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: consul-consul-client-config
Optional: false
consul-consul-client-token-xp5q9:
Type: Secret (a volume populated by a Secret)
SecretName: consul-consul-client-token-xp5q9
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/disk-pressure:NoSchedule
node.kubernetes.io/memory-pressure:NoSchedule
node.kubernetes.io/not-ready:NoExecute
node.kubernetes.io/unreachable:NoExecute
node.kubernetes.io/unschedulable:NoSchedule
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 5m44s default-scheduler Successfully assigned development/consul-consul-hzndl to ip-10-0-32-42.ec2.internal
Normal Pulling 5m43s kubelet, ip-10-0-32-42.ec2.internal pulling image "consul:1.6.2"
Normal Pulled 5m40s kubelet, ip-10-0-32-42.ec2.internal Successfully pulled image "consul:1.6.2"
Normal Created 5m40s kubelet, ip-10-0-32-42.ec2.internal Created container
Normal Started 5m40s kubelet, ip-10-0-32-42.ec2.internal Started container
Warning Unhealthy 5m12s (x3 over 5m32s) kubelet, ip-10-0-32-42.ec2.internal Readiness probe failed:


SSH'ing into the worker nodes and doing a curl on localhost (someone said to try it online so I gave it a shot) results in a connection refused.

&#x200B;

In my helm chart (deployment file) for my Node.js application I have this code which is providing the IP for consul to connect to

env:
- name: CONSUL
valueFrom:
fieldRef:
fieldPath: status.hostIP

I got the above from [here](https://www.consul.io/docs/platform/k8s/run.html#accessing-the-consul-http-api) so I expected it to work.

&#x200B;

So from my digging around I am fairly sure it's because the services are not passing the readiness checks even though kubectl get pods is showing 1/1 for all pods

&#x200B;

Suggestions on what I can try to get my application to connect?

https://redd.it/eclflw
@r_devops

How to Package virtualenv as zip/deb/tar and run it on another machine?

Hi,

I have a Django application and I've created a virtualenv for all dependencies for running my Django application and which does work perfectly. So, now when I zip the whole application Django files along with virtualenv. And trying to unpack it to another machine and unzip it and when I tried running the same application, it fails.

On machine X:

cd my_app
virtualenv env_django
source env_django/bin/activate
(env_django)# pip install -r requirements.txt
(env_django)# python manage.py -> this works

Now, Ziping it.

zip my_app.zip my_app

On machine Y:

unzip my_app.zip
source env_django/bin/activate
(env_django)# python manage.py -> this fails

throws out the below site-packages error.


from cryptography.hazmat.backends.openssl.backend import backend
File "/env_django/lib/python2.7/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 46, in <module>
from cryptography.hazmat.bindings._openssl import ffi as _ffi
ImportError: libssl.so.10: cannot open shared object file: No such file or directory

The application really works on Machine X, but not on machine Y. I've tried installing system libraries too. and linked the lib files using [library link](https://askubuntu.com/questions/339364/libssl-so-10-cannot-open-shared-object-file-no-such-file-or-directory), but no luck.

libdev-ssl
libcrypto

I really want to package it and ship to the customers, I want to use only zip/deb/tar.gz package but not as a Docker.

So, is there any way we can ship the virtualenv to different machines via packaging as deb/tar?

https://redd.it/ecigik
@r_devops

Which CI/CD mechanism do you all use?

I was just curious which mechanism did you guys find efficient among the all you used till now.
I know everyone has a different way of doing it, which one did you like.

Thanks.

https://redd.it/efqoz6
@r_devops

Automating Developer Workspaces

Hello,

I'm trying to provision a fair amount of developer workspaces, in all 3 of the major OSes, with a few different types of preset application packs based on the role of the user.

Currently, my strategy is to try out Packer with Terraform or Ansible and try to automatically install these fresh images on empty machines, but I'm not sure exactly if it is possible yet to achieve this with these tools.

Also, would it be possible to somehow connect some laptops to our network and get the OS images to autoinstall themselves?

Anyone from the fields that has any advice? :D Much appreciated

https://redd.it/eclyr7
@r_devops

Alternatives to DockerHub

I'm paying $22/month for DockerHub, which is pretty pricey.

I have my own k8s cluster, and I'm wondering what the best way to either create my own registry or use a cheaper alternative.

- Create a service inside of my cluster.
- Create an EC2 instance outside the cluster.
- Use a 3rd party.

Any insight?

https://redd.it/ech512
@r_devops

Artifactory system level importing and exporting from the command line?

Im trying to replicate an existing Artifactory server into an offline environment. Does anyone have experience with jfrog products? Im looking to automate the export of the systems but I cant find any docs on how to do it other than from the GUI.

https://redd.it/ecirlv
@r_devops

Implementing a CI pipeline using Perforce

Hello there,
I'm working in devops for a game Dev company and our depot size is ~ 200gb - Building the game locally takes ~ 20 mins. We are currently evaluating ways to improve/redevelop our CI pipeline.
Current setup polls the swarm server for new reviews and builds the shelves associated with them but since artists & game designers don't use swarm...it's just semi-helpful.
We have several architectures in our mind (submit trigger, staging depot) but each has some serious drawbacks.
What's the established, usual way to implement a CI pipeline with p4?

https://redd.it/ecipbr
@r_devops

Traffic generation tool that can be used both for baselines and stress tests

Hey everyone, here's one that I think is a relatively straightforward question but I haven't found a solid answer to online. We're trying to run some load testing on our sites, and ideally use one tool for both baseline measuring and stress testing. The trouble I've run into is that some tools out there are great for generating a bunch of traffic and hammering endpoints, thus giving consistent request/response time baselines (like apache bench), and some tools are great at generating a huge load of realistic user behvior (like locust). We've tried running both baseline testing and performance testing with locust, but we've found some marked unreliability in terms of request/response times with locust, indicating to me that there's either some noise in our test infrastructure or the test itself is not keeping all variables the same and is thus unsuitable for baselines.

Have any of you out there used one tool for both use-cases, or are they measured differently inherently and worth using two separate tools for?

https://redd.it/ecfzmh
@r_devops

Where can I start to modernize my DevOps environment?

Please read through my quick overview of our environments at my company, and let me know where you think is a good place to start in order to get this environment into a more modern DevOps culture.

* Production and staging’s technology stack is as follows:
- Legacy .NET Framework APIs running on 8 windows 2008 R2 IIS Servers (EC2 instances).
-Data is managed by a very large monolithic SQL server cluster (2 x i3.16xlarge and 1 i3.8xlarge).
-The new microservices are running on AWS Lambda/DynamoDB/S3 – each with their own Cloudformation templates.
* The development environments:
-20 different dev environments running a single EC2 instance each, and each holding all of the legacy APIs on that 1 EC2.
-Each dev environment also has its own EC2 running SQL server 2008
-The microservices/lambdas for the dev environments are created by me on a per-needed basis.
-Basically if a software developer is working on one of the serverless services, they will come to me to add that service to their development environment.

* Source Control and the CI/CD Pipeline:
-Microsoft’s Team Foundation Server (TFS)
-Developers check in the code to TFS and then manually queue a build from the TFS console
-Deployments are manually initiated on TFS, but then executed by AWS CodeDeploy. CodeDeploy uses the CodeDeploy agent to talk to the dev environment EC2 instances to deploy the code (which was zipped up to S3 via TFS tasks).
-The microservices are deployed via powershell scripts (AWS tools for powershell) from the TFS deploy server

Though our developers try to follow Agile principles, we very much do Waterfall deployments. We are unable to do blue/green or rolling deployments because of the massive legacy SQL server. Every deployment that I’ve run, the SQL server has had very large schema changes which require us to run migration scripts to update the schema.

The goal of the development team is to breakup the endpoints that live in the legacy .NET framework APIs and convert them over to microservices while also removing the dependency of the large SQL server. But in the meantime, every month or so we have a 6 hour outage on a Friday evening to do a large Waterfall deployment.

The things that I want to do in order to change culture here would require such a massive overhaul, I just don’t know if it’s possible without a huge investment from the company.
I guess in the meantime I would like some suggestions on anything you see fit to suggest.
Here are a couple of big-ticket problem items that I can’t figure out:

* How to better manage having a ridiculous 20 dev environments (EC2+Sql server combo) with microservices added to each as-needed.
* Assuming the SQL server dependency doesn’t change for a while- how to better manage that
* Best entry point to start engineering for live production releases rather than a Friday evening downtime once a month.
* Anything else you can think of

https://redd.it/ec3pwl
@r_devops

My DevOps Learnings in 2019. List Yours:)

1. DevOps is not an individual role, it is a group effort.
2. Your infra should be able to handle a failed deployment automatically and rollback. If it can’t, you’re doing it wrong.
3. DevOps means different things to different people.
4. If any task cannot be automated, remove it instantly from the task list.
5. Communication is more important than tooling and will make or break an organization.
6. Shift the focus of Security to the left in the development life-cycle.
7. Jenkins != CI/CD …
8. Always design with autoscaling in mind. Automate-all-the-things. One-click deploy to one-click revert.
9. Create ready to use images in AWS and use those images for auto-scaling instead of scripted from GitHub builds.
10. There is a real skillset shortage in the DevOps field.
11. The CI/CD pipeline is one of the best practices for DevOps teams to implement, for delivering code changes more frequently and reliably.
12. Do not automate a bad process, you end up with an automated bad process — \[Credits: DOES18 \]
13. Testers who know how to code and automate scripts to test various cases are in massive demand in DevOps.
14. Add monitoring to infrastructure as well as applications. The monitoring system should send alerts on every important events.
15. Utilize Infrastructure as a code templates to build the infra. Ex. cloud-formation, terraform etc
16. If you need to SSH into your VM then your automation is failed.
17. Serverless is not actually serverless, it is simply 'pay as you use' model
18. Cloud-native is not a synonym for microservices or Kubernetes, it is all about utilizing the advantages of the cloud computing model

https://redd.it/eg6jjf
@r_devops

Terraform single vs multiple workspace for team collaboration

Hi guys

We are migrating to new cloud vendor and starting to use terraform to build our infrastructure with code.

We have many people in our team, and probably have around 5 people who will be focused on developing terraforms. Within these 5, only 2 will have super admin privilege (full access in cloud stacks)

We are making use of terraform modules, where each module is dedicated for specific needs (i.e. network, instance, etc)

While i’m still learning beat practices with terraform, i see the following 2 options at this point (please note i’m a beginner)

1) One master terraform workspace where each module is a subfolder. Terraform state will be stored in cloud object storage (like s3), and multiple people will use this single workspace for developement. Pro is we have only one place where we define our infrastructure, and our modules are in centeal workspace. Con is that it makes it difficult to control the blast radius - theoretically, non admin who does not have full infra knowledge can make mistakes or push incorrect changes that can screw up something like network infrastructure

2) Multiple terraform workspaces.
For example, one workspace will be dedicated for network (vpn, subnets, routes, etc) where only admins have access to develop. In our shop, network is something that does not change very often.
The other workspace will be dedicated for something like compute services (i.e. provisioning new ec2 instance) - everyone will have access to this. Even if someone screws up something, the worst they can so is destroy instances or whatever the scope of that workspace is.
Downside here is now our infrastructure is defined in multiple places, modules will be scattered, will create some dependencies (i.e. network needs to be set up before you can provision instance in the right network)


For me option #2 feels more appropriate since it will make it easy to divide the work between full admin and non admin. It will also reduce blast radius in case something goes wrong.

What do you guys think? Maybe I’m not understanding terraform very well therefore not making an educated assessment. If you guys are using terraform in your shop, please share how you manage your workspaces and collaborate within your team.

https://redd.it/eg53rs
@r_devops

What do you use for infrastructure CD

Hey there. I'm just wondering what everyone is using for automatically updating infrastructure. I'm currently trying to get all my provisioning and configuration as code, and while it works great manually, pipelining it all together seems a bit of a task, especially with lots of inputs and outputs of different tasks. We currently have a multiple Jenkins pipelines, with the "main" one that provisions a k8s cluster looking like the followings:
- Terraform to provision VMS. This gives us a tfstate file, which also gets outputted to Jenkins as an artifact.
- Ansible to configure each VM, using Inventory from our tfstate, using the Terraform inventory plugin. This runs multiple playbooks but not all, because we need to...
- Bootstrap Rancher via terraform, which is running as a lone container. This will give us a Rancher URL and token, which can be used to...
- Create a K8S RKE cluster via Terraform. This will give us a cluster URL and a Token which can be used to...
- Install Rancher Agent on each VM via Ansible

As you can see, lots of input is getting piped back and forth, with no easy way of seeing what is and isn't working, without looking at the text output from each job. Does this seem right to you guys? Ideally we would have one big playbook which would do everything, but Ansible and Terraform both have different strengths at different things. All the current CICD tooling seems to be really dev focused

https://redd.it/eg155y
@r_devops

Associate Devops role or stay as a sys admin?

Hi guys,

&#x200B;

Recently been offered a role to an associate devops for a msp. It is what I have been looking for, except, its a junior role and the pay is slightly lower than mine (8k aud less, I am on 98k)

I am a sr sys admin, but my work dont have much to do with clouds and they lack the cloud vision to moving forward as a company. The offered company has been cloud Consultant in aus for some time now, so they have some big clients like major banks and Airlines.

What would you guys do? Stay as a sys admin / potentially becoming a SE or go to a junior devops role and move on from there?

https://redd.it/eg42m5
@r_devops

elassandra questions

Hey guys,

My company and I are looking into setting up elassandra to be one of our data stores. We want to start small, so what would the difficulty be in setting up a single elassandra cluster? Also, are there any good resources out there besides the docs?

https://redd.it/efz9ek
@r_devops

write ansible playbook that outputs each configuration items in a format that can be imported into a spreadsheet in tab-delimited format

I need to write a playbook an output printing in csv file using python. The output is the ip address config.

https://redd.it/efykws
@r_devops