Deploy to production: best practices?

Hi!

I have memories of the way a sysadmin was deploying to production in a company I was working for, many years ago: he would rsync/scp from staging to a new folder on the production server, and then change the `root` in the vhost file and then nginx reload (and then delete the old `root` folder if everything went well).

Is this the best practice? Or is there another, more commonly accepted way to do it?

For now I am using `git pull` and then `composer install` but it is not ideal at all... If I encounter an issue, the server will be in a broken state (please note that I am not a sysadmin or devop... I am a humble web dev).

Also, is it even possible to have zero downtime/bugs when the deployment implies running a few scripts that modify the database structure and update some stored data?

Thank you all :)

https://redd.it/slm9h1
@r_devops

I don’t know if I can make it past a phone screen

I’ve been working in devops for 5 years. In that time, I’ve migrated 3 companies from hosted solutions or monoliths to microservices in k8s. I’ve created and worked on IaC with terraform, written helm charts, done basic Linux server maintenance. I’ve written CI/CD solutions for maybe 100 different pipelines, using different pipeline solutions, implementing and debugging remote caching tools for many. I’ve written some bash scripts for local dev config. I’ve migrated cobbled together systems to AWS.

I can barely sort arrays. I need to Google how to mutate a graphql query. The most basic leetcode questions, like “dedupe 2 linked lists” stumps me. When it comes to actually coding an app, it looks juvenile and is in no way optimized, if I can even do it at all.

I feel like every position i’ve had I got by some miracle. I know imposter syndrome exists, but I really feel like it’s a justified thought in this case.

I want to move on to another position, but I could never make it past a phone screen where they want me to whiteboard a question about algorithms.

Do I need to go to school to get a CS degree? Would anyone ever take a chance on someone self taught, who has experience but none of the education background? Is there any situation someone would take a chance and hire aomeone like me?

https://redd.it/sln5qe
@r_devops

Examples of when you used multicloud infrastructures and whether it has been useful

I’ve got a fair amount of experience with infrastructure as code using HashiCorp’s Terraform but have only ever used one provider for my cloud infrastructure needs (i.e., AWS).

Have you incorporated multicloud solutions before and do you find that they have been beneficial for your projects/work? If so, did situations ever come up when you were glad that you had another cloud provider to depend on?

https://redd.it/slpds7
@r_devops

How do you deliver logs from ELK to users?

My experience in visibility and observability is quite poor, so I am a bit confused on this topic.

We have mostly monolithic PHP applications which have previously been running on ad-hoc managed servers, and have relatively recently been containerized and are now running on EKS. We have a single ELK stack collecting logs from production applications (and another for pre-prod) and other containerized tools, as well as storing infrastructure metrics from Prometheus, cluster events etc.

Most of the application logs are not in a consistent format (they were implemented as human readable log lines), although there are some exceptions, but no specific log format standard. Previously the users (development teams) have been used to having direct SSH access to the servers, where they could grep the logs and so on to diagnose issues. This would, of course, not make sense for the now containerized and auto-scaled workloads.

So the question is, how would you deliver the application logs to the developers? Currently we query for some application logs and present them as a table in Grafana, but that doesn't really meet their needs. Would you just give them access to Kibana and let them query everything themselves? That seems a bit excessive and I think it might be too complicated for daily troubleshooting. Is there a better way?

https://redd.it/slrao0
@r_devops

Is there an app to help manage multiple repos on a local dev env?

It's really common to have at least 2 repos that make up a working solution. My last project had 9.

Is there a tool that exists that can help manage these repos? Something with a UI that can show what's spun up and what isn't, which branch they're on etc...

I'm tempted to build one if not.

https://redd.it/slsmey
@r_devops

Career transition from Data analyst to DevOps/Cloud/Infra Engineer

I m 28 based in Europe, I have 3.2 years of experience as a data analyst (BI analyst and SQL) planning to shift my career to Devops /cloud Engineer. I was recently offered a job as an Application Support Engineer (Server monitoring - DATADOG, SQL Server and basic troubleshooting ) at an average salary and I was offered another position ie a Data analyst position at IBM at a good salary, which one should I take opt?

Currently, my plan is to learn these and shift to Cloud/DevOps Engineer positions in future ie 6-8 months?

1.LINUX RHCSA

2. CI-CD

3. JENKINS

4. ANSIBLE

5. DOCKER

6. AWS

7. TERRAFORM

My current Skillset

1. Python basic (No DSA)

2.SQL

3. Tableau and PBI

Do companies consider profiles like mine for Cloud/DevOps Engineer roles if I possess good technical skills for these roles? Am I On the right path?

https://redd.it/sllzv7
@r_devops

Deploy same stack several times with kubernetes

Hey guys, i have a question

My deployment atm is:
A container having a django app with a container having postgresql, deployed as a stack with docker-compose -p (tag)


Every user i create a stack for him with his volumes and i route the traffic to his stack using sub domains

For example a user John and a user jane:
Docker-compose -p john
And
Docker-compose -p jane
On the same docker-compose.yml file

That way i have the same stack deployed 2 times each isolated from the other, then with nginx i route jane.mywebsite.com
And john.mywebsite.com each to his stack

My question is how can i automate this deployment with kubernetes and with CI/CD

I have looked around and found that i can isolate deployments in kubernetes with namespaces but namespaces doesnt give me the ability to control/update the containers of all stacks at the same time

Thanks, appreciate the help

https://redd.it/sljzuh
@r_devops

Deploy same stack several times with kubernetes

Hey guys, i have a question

My deployment atm is:
A container having a django app with a container having postgresql, deployed as a stack with docker-compose -p (tag)


Every user i create a stack for him with his volumes and i route the traffic to his stack using sub domains

For example a user John and a user jane:
Docker-compose -p john
And
Docker-compose -p jane
On the same docker-compose.yml file

That way i have the same stack deployed 2 times each isolated from the other, then with nginx i route jane.mywebsite.com
And john.mywebsite.com each to his stack

My question is how can i automate this deployment with kubernetes and with CI/CD

I have looked around and found that i can isolate deployments in kubernetes with namespaces but namespaces doesnt give me the ability to control/update the containers of all stacks at the same time

Thanks, appreciate the help

https://redd.it/sljzuh
@r_devops

Installing Docker Desktop causes my laptop to blue screen on restart and I cannot find a solution. I would appreciate any help you can provide.

So, I'm in school still and we've begun using Docker Desktop this semester, however I cannot install it on my laptop without it causing blue screens and errors.

​

When I press Close and Restart in the install wizard, upon trying to restart, I am met with a blue screen that tells me my PC has run into an error and needs to restart. This then restarts again and takes me to the advanced startup menu? (I think that's what it's called) where I can choose to start in safe mode, system restore etc etc.

​

Does anyone have any advice on why this may be happening, or how to fix it? So far I have tried system restores, I have tried following the advice on the docker hub about enabled VM stuff (can't remember what the setting is now) as well as a fresh windows install. None of these have yielded an ability to install Docker Desktop.

https://redd.it/slxoni
@r_devops

Do any of the companies you're working where Jenkins is used plan to keep using it?

Hello folks! In recent months I've been talking to plenty of companies that are either planning to or already migrating from full on-premise enviros with Jenkins to GitLab, CircleCI or GH Actions + various other more modern CI/CD pipelines.

Like in the title: is any of the companies you're working at or any company you know well enough articulating that they're going to stick with Jenkins for the next 3-5 years regardless of the general trend to move to other solutions?

I kinda understand that lots of business have invested masses in making it all work with Jenkins but I'm curious if any value the money and time spent over new, potentially better alternatives?

https://redd.it/slzxc1
@r_devops

Would python’s tox build system count as infrastructure as code?

I recently started using tox at work. You’re specifying your environments, your commands to run within those environments. Does this not count as IAC? I’ve previously used vagrant, so I’m at least somewhat familiar with the concept. Or am I missing something?

https://redd.it/sm1sis
@r_devops

Road map for beginner

Hello everyone, could you help with a road map, I work as a Linux administrator for about two years and want to switch to DevOps.

For example at work I use:

bind, keepalived, iptables, gragana+prometheus, postgresql+pgbouncer, nginx+phpfpm, tomcat, graylog+splunk (a little bit EFK), veam+vsphere, docker.

In the process of learning Git, Gitlab, and GitLab-runner (I am trying to commit changes on the "bind" through Gitlab)

the above is made manually, without automation.

the following tools and technologies in what order should I learn?

Ansible,

terraform,

Cloud (Azure or AWS),

Kubernetes (managed k8s or local only).

https://redd.it/sm1183
@r_devops

How to use Jenkins the DevOps way in 2022

I've seen a lot of "Who would use Jenkins now?" posts on this board, and they tend to rub me the wrong way.

I think a lot of people here have a baseline understanding of Jenkins that is some variant of "Someone manually creates a pet instance and manually creates jobs using the web UI", which is obviously silly in 2022, and here on /r/devops to make a reasonable comparison it would make sense to compare against a properly DevOps-ified Jenkins deployment. This post hopefully will explain how to do that.

The basics of running Jenkins in a DevOps way:

1. Install it using automation. We mainly host it VM's, and provision with Ansible.

2. Do the base configuration (auth, plugins, etc.) using Jenkins Configuration as Code, as a part of #1: https://www.jenkins.io/projects/jcasc/

3. For jobs, create them in a GitOps-style meta language. Jenkins Job Builder is one good option: https://docs.openstack.org/infra/jenkins-job-builder/ . Using Declarative Pipeline is another option, and they can also be used in combination.

4. For task runners, primarily use Docker executors or ephemeral VMs. The images for these are created with Packer: https://www.packer.io , with installation tasks handled in Ansible so that they can be shared across all image types, and on physical hardware when needed.

5. Use Jenkins to automate Jenkins - have a job running out of CI that handles job creation (3) and image creation (4).

A good source for how to do most of the above is the Linux Foundation Release Engineering docs: https://docs.releng.linuxfoundation.org/en/latest/jenkins.html

Also, the general benefits of Jenkins are massive flexibility:

Works with any target test platform
Doesn't lock you into specific code hosting and review platform
Works in any environment - cloud hosted, on-prem, etc.
There's a plugin for everything, although you generally should pick maintained and popular plugins.

Now, not to say that Jenkins is perfect - it still has rough edges and legacy baggage, but in some cases it's the right choice. And if you run it the DevOps way, it's actually quite a slick and flexible system.

https://redd.it/sm4eux
@r_devops

What's your org strategy to manage your prod k8s clusters ?

I've joined lately an org with quite the messy infrastructure, they have 4 clusters on gcp for prod env, and the cluster nodes are quite far behind in terms of version in comparison the the managed control place, they didn't upgrade until now, because they were like... prod, working, no touch. So yea they were afraid to break it, lately the team has been reformed and we agreed to make it priority to keep our infra to date, and it fell on me.

so, what is your process for managing your clusters ? do you just upgrade manually and hope nothing breaks, also is there any utilities to help making the upgrade process smoother in the future(using terraform, ansible ?)

https://redd.it/sm5x4b
@r_devops

Deploy same stack several times with kubernetes

Hey guys, i have a question

My deployment atm is:
A container having a django app with a container having postgresql, deployed as a stack with docker-compose -p (tag)


Every user i create a stack for him with his volumes and i route the traffic to his stack using sub domains

For example a user John and a user jane:
Docker-compose -p john
And
Docker-compose -p jane
On the same docker-compose.yml file

That way i have the same stack deployed 2 times each isolated from the other, then with nginx i route jane.mywebsite.com
And john.mywebsite.com each to his stack

My question is how can i automate this deployment with kubernetes and with CI/CD

I have looked around and found that i can isolate deployments in kubernetes with namespaces but namespaces doesnt give me the ability to control/update the containers of all stacks at the same time

Thanks, appreciate the help

https://redd.it/sljzuh
@r_devops

Pros and Cons of GitLab for CI|CD?

what are some experiences you've had with Gitlab, we are looking into it mainly for CD since we deploy to production quite a bit.

https://redd.it/slid4w
@r_devops

Do any companies/projects publish their Terraform code publicly?

I’m looking for Terraform examples from companies/projects. For example, Arch Linux has all their Ansible/Terraform in a public GitLab repo.

https://gitlab.archlinux.org/archlinux/infrastructure

Are there any other examples like this? Looking more for companies/projects instead of a random person’s homelab.

https://redd.it/smfr2e
@r_devops

A friend tried to talk me into DevOps.

Well, title.

Right now I am a student of Earth and Environmental Science and I am failing hard. We have a large IT scene in my city and a friend who is a back end developer with a masters degree in computer science tells me it is easy getting into it if you know how to google and some stuff about computers.

I told him that I can’t code whatsoever. I tried learning C# once but it was boring me out of my mind so I stopped after hello world.

But I am good at figuring out how software operates however obscure the UI may be and I am quite skilled at solving software and hardware related problems (identifying and fixing) by using what I know and combining that with google.

He told me DevOps it is for me and DevOps guys are sought after as well. So I decided to try out a bootcamp and then find a job. My alternative would be a state supervised school where you work half the time in a company and the other half go to school to learn theory. It is shit pay for three years but after that it’s pretty on par with what I know of Junior Devs income.

That job would probably involve more practical work, assembling computer systems at large scale for companies or managing the IT infrastructure of buildings and such. (There I would learn basic python only)

So final question: Does DevOps sound like something I could do as a complete beginner or is that more something for people who know some coding or scripting?

https://redd.it/smcn5v
@r_devops

Is multi-master possible with k3d? If not or it's a hassle to set up, are there any good self-hosted options that support it?

I'm looking to kind of simulate a production HA environment without paying crazy high cost of EKS/AKS/GKE. This is specifically for a home environment though as I want to be able to test things like operators, CRDs, and other various concepts and tools.

https://redd.it/smc870
@r_devops

K3s, Jenkins and Docker Build

Going to ask here incase someone can help.

I have a K3S cluster running Jenkins with the kubernetes plugin. I can get my builds to build, spinning up build agents in pods. I cannot for the life of me get docker build to work at all.


Dond:

Agent docker-kdb22 is provisioned from template docker
---
apiVersion: "v1"
kind: "Pod"
metadata:
  labels:
    jenkins/jenkins-jenkins-agent: "true"
    jenkins/label-digest: "1f9ffaa0a312a3be56683597d58f3df0d8226bdb"
    jenkins/label: "jenkins-docker_docker"
  name: "docker-kdb22"
spec:
  containers:
  - command:
    - "cat"
    image: "docker:latest"
    imagePullPolicy: "IfNotPresent"
    name: "docker"
    resources:
      limits: {}
      requests: {}
    securityContext:
      privileged: true
    tty: true
    volumeMounts:
    - mountPath: "/var/run/docker.sock"
      name: "volume-0"
      readOnly: false
    - mountPath: "/home/jenkins/agent"
      name: "workspace-volume"
      readOnly: false
    workingDir: "/home/jenkins/agent"
  - env:
    - name: "JENKINS_SECRET"
      value: "********"
    - name: "JENKINS_TUNNEL"
      value: "jenkins-agent.ci.svc.cluster.local:50000"
    - name: "JENKINS_AGENT_NAME"
      value: "docker-kdb22"
    - name: "JENKINS_NAME"
      value: "docker-kdb22"
    - name: "JENKINS_AGENT_WORKDIR"
      value: "/home/jenkins/agent"
    - name: "JENKINS_URL"
      value: "https://jenkins.ci.svc.cluster.local:8080/"
    image: "jenkins/inbound-agent:4.11-1-jdk11"
    name: "jnlp"
    resources:
      limits: {}
      requests:
        memory: "256Mi"
        cpu: "100m"
    volumeMounts:
    - mountPath: "/var/run/docker.sock"
      name: "volume-0"
      readOnly: false
    - mountPath: "/home/jenkins/agent"
      name: "workspace-volume"
      readOnly: false
  hostNetwork: false
  nodeSelector:
    kubernetes.io/os: "linux"
  restartPolicy: "Never"
  serviceAccountName: "jenkins-agent"
  volumes:
  - hostPath:
      path: "/var/run/docker.sock"
    name: "volume-0"
  - emptyDir:
      medium: ""
    name: "workspace-volume"

script:

node('docker') {
    stage('Build') {
        container('docker') {
            echo 'Building....'
            sh 'docker pull alpine:latest'
        }
    }
    stage('Test') {
        echo 'Testing....'
        sh 'docker image tag alpine:latest container-registry-docker-registry.prod.svc.cluster.local:5000/baseimages/alpine:latest'
    }
    stage('Deploy') {
        echo 'Deploying....'
        sh 'docker image push container-registry-docker-registry.prod.svc.cluster.local:5000/baseimages/alpine:latest'
    }
}

I get

[Pipeline] {
[Pipeline] stage
[Pipeline] { (Build)
[Pipeline] container
[Pipeline] {
[Pipeline] echo
Building....
[Pipeline] sh
+ docker pull alpine:latest
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 1
Finished: FAILURE

I've followed just about every guide on google i can find and my config looks ok.

(using helm chart currently for jenkins)

https://redd.it/sml8of
@r_devops

Is multi-master possible with k3d? If not or it's a hassle to set up, are there any good self-hosted options that support it?

I'm looking to kind of simulate a production HA environment without paying crazy high cost of EKS/AKS/GKE. This is specifically for a home environment though as I want to be able to test things like operators, CRDs, and other various concepts and tools.

https://redd.it/smc870
@r_devops