CD Pipeline for Flask + DB

Hey everyone,

I'm working on how exactly to architect a continuous delivery pipeline for a small project that will probably grow in scope over the next year. At the end of the day, it's a basic web server (Flask) with a relatively independent database.

I've got both components set up in their own docker containers and configured with a docker-compose script for local development. While I have experience with docker and deploying a single application on a server manually, I would like to set up a pipeline through GitLab to keep the server updated. I'm planning to deploy on Linode but I'd like to figure out a pipeline for the continuous delivery piece before I get too far into manually configuring a server.

From my experience, deploying a new set of code should just be a matter of logging in, pulling the latest code from git, re-building/updating containers, then restarting the services. I would prefer zero downtime but it's not a necessity at this point. Is there any way to automate this within a GitLab runner? I've heard of them but have never used them, so this may be an uninformed question.

I've also heard of Ansible, Chef, Jenkins, and a bunch of other CI/CD services, but I have 0 experience with any of them. Are there any that work particularly better with docker? Do any of these solve this problem easier than others? In terms of scalability, are any better if I needed to scale?

If you've got any advice on the docker configuration, VPS configuration, or general DevOps, let me hear it! Thanks in advance for the help!

https://redd.it/em4fdk
@r_devops

I feel so stupid

I work as a DevOps engineer. I’ve been at a startup for 10 months. Before this I was a QA Engineer for 2 years. I face so many problems that literally short circuit my brain and I just can’t figure it out in less than a day. Easy stuff like setting up Gitlab Ci pipelines

I know it shouldn’t take that long but it does. Honestly I just feel so inadequate and stupid and like I maybe moved too fast.

Has anyone else had these feelings and how did you fix them? I want to be smarter, sharper, but I just can’t.

https://redd.it/em1m9r
@r_devops

Developer - DevOps all in one server.

I know this is overkill, I work with a giant team very much Tribe, mentality scrum + XP development cycle.

I have a personal project I am working and trying to put some of those procedures into my process (even though its just me) to manage my project. I am building a widget. I have all the scrum stuff in place but want to pretend this is a real product and set up a DevOps server. I have a relatively new machine and was thinking I would use HyperVisor to create the following VMs,

1. Team City
2. Database server with all the databases are.
3. Development web server - [dev.foo.com](https://dev.foo.com) accessibly only locally.
4. Stage web server - [stg.foo.com](https://stg.foo.com) accessibly only locally.
5. Test web server - tst.foo.com accessibly only locally, potentially going live if I get there.
6. Production web server - [www.foo.com](https://www.foo.com) accessible locally or on the web.

Is this crazy or stupid? TC would be configured to draw from different branches in some automated some requiring user intervention.

​

Should I do this on Windows 10, or use my MSDN license to get some server brand of Windows?

​

Any help would be appreciated.

https://redd.it/em3ibt
@r_devops

Devops engineer vs an organization that truly practices DevOps?

Mini rant I suppose.

There are books like The DevOps Handbook and Accelerate which teach us the art and practice of what DevOps is and what it’s supposed to be. However most organizations don’t do this. They typically just hire DevOps engineers and think “now we’re doing DevOps m”. It doesn’t seem like management or leadership are interested in what it means to truly practice DevOps at an organizational level.

Is anyone else experiencing this? What do you do to cope with it and “accept” it?

https://redd.it/em3cp3
@r_devops

noob that would like to put some order to my workplace

I'm extraordinarily new to devops, I barely know some stuff and I would like to help my workplace have a better order on implementing software/updates.

in here we don't use docker or git, so I was thinking of how to learn git so I can have branches and deploy those on the server, I mean, there are lot's of tutorials on GIT and how to commit stuff and all that, but how do you use git to push things to a server, how to automate test, the best practices for branches and all that.

also, if I would like to do continuos deployment (which I assume is updating the server), is it possible to do it in one of those servers that use cpanel and all that?

is there like a roadmap to do all this?

https://redd.it/emw0on
@r_devops

One or many repos for a single project with several lambdas?

Currently working on a serverless project including two lambdas, going to be adding a third here real quick. I thought initially it would make since to have a repo per lambda but only one cloudformation template so it's not really working out to be as separate as I had thought it would be. I see a couple of options:

1. All-in-one - all code in a single repo with single build process and cloudformation template for depolyment/updates
2. Semi-split - current state of things, one repo for "main" part of project with cloudformation template and second repo for second lambda. Currently the single template references a build artifact from the second lambda build process.
3. Full-split - One repo per lambda with code and smaller "atomic" template, and another for the project template with the "atomic" templates included as well as the documentation and misc.

​

I would love to hear about your thought, other options, best practice documentation?

Thanks

P.S. If this would be better over on r/AWS just let me know.

https://redd.it/emvumo
@r_devops

How to setup NuGet caching in Azure DevOps

Azure Pipeline NuGet caching can save your builds from painfully downloading the same packages over and over on each new run. I ran into some problems while setting this up, and there wasn't much of a tutorial out there going through the entire process. It saves a bunch of time on our build runs and I thought I'd share:

[https://blog.soenneker.com/how-to-setup-nuget-caching-in-azure-devops-8d94c57b5321](https://blog.soenneker.com/how-to-setup-nuget-caching-in-azure-devops-8d94c57b5321)

https://redd.it/emss5m
@r_devops

What are the best emails or emails you are more likely to respond to regarding new solutions.

So I’m very new to the industry and I get emails everyday and it’s hard to keep track, especially when they are ones that don’t require immediate attention. Also, many of the times they are things I’m not interested in or that just waste my time.

I wanted to see what everyone’s viewpoint was in emails that you get and which ones I should keep my eye out for.

Just looking for a good discourse to have.

https://redd.it/emud7z
@r_devops

Is datadog worth the cost?

How does datadog compare to prometheus operator when talking about Kubernetes clusters? I know Data dog costs a hell of a lot more them prometheus however it's dead simple to setup where prometheus is a royal pain to do. I've tried both and I like datadog just not sure about the cost of it.

​

Is there any major advantages to datadog that would not be present in prometheus? Trying to figure out if I should shell out for datadog or not

https://redd.it/emr6pi
@r_devops

Best installation script practices

Hey everyone, I’m writing an installation script for our infrastructure. Everything is Docker based and we have a Makefile that will deploy the infrastructure. However there is some setup required before the Makefile can run on a new system (docker must be installed along with some other system configs). I wrote a bash script to handle all that and it works great, but I started musing about what best practices are for system installation. What’s the best language to create an installer that can install to different OS (Windows, Ubuntu, RHEL, etc). I’m thinking about a Python script that detects the system configuration and then generates a bash or powershell script with the appropriate commands. Are there better languages for this? Are there installer frameworks out there that I don’t know about? I enjoy crowd sourcing opinions so I’d love to hear from you all.

https://redd.it/emu9v8
@r_devops

What could I build to make your life easier?

I'd like to start a for-fun, side project. Not sure what to build though. Anything I could build to help you (e.g. open source, CLI tools, IAC, scripts, etc.)?

Most my background is in Node, Terraform, AWS, and Digital Ocean.

https://redd.it/en2mfk
@r_devops

Where does Serverless Fit in nowadays with everyone running K8s

So it seems to me to be Kubernetes everywhere or at least a lot of companies and environments. I know there are plenty of traditional set ups with simple VMs. What about Serverless Technologies? What is the general feeling towards this technology right now? I feel like it’s taken a back seat.

Personally I want to focus in that direction a bit - honestly it just seems a little bit more interesting to me. I don’t know a ton about it. I know more of the AWS side about different ways to set it up but just curious about everyone’s thoughts.

Thought I do remember hearing some of you guys went through some serious pain with it.

https://redd.it/en1oa9
@r_devops

Looking for design questions and scenarios?

Some clown posted a question on a Facebook group I'm in, he was given a sort of pre-interview question and was asking for someone to do it for him.

The exercise actually looked really interesting and I would enjoy a book or video series that went through questions of a similar nature explained possible solutions, pros, cons, traps ect. Does such a thing exist?

​

Below is the question just for reference.

​

​

Design Question

We have an e-shop application that contains multiple microservices. Each service provides a
few APIs. Some are public APIs while the others are for internal use only. The application uses a
database for data storage.
Service list:
• user-service
o POST /api/v1/login (public)
o POST /api/v1/logout (public)
o POST /api/v1/signup (public)
• product-service
o GET /api/v1/products (public)
• payment-service
o POST /api/v1/pay (public)
o POST /api/v1/cancel (internal)
• db-service
o GET, POST, DELETE /api/v1/user (internal)
o GET, POST, DELETE /api/v1/creditcard (internal)
o GET, POST, DELETE /api/v1/product (internal)

We want to deploy this e-shop application into our Production environment now so we can sell
our products online. It has such requirements:
1. People can access this e-shop application via www.happyshopping.com
2. It should be fast
3. It should be secure
4. It should have high availability
5. It should be able to scale based on workloads
6. It must run within Kubernetes cluster
As a solution architect, you need to propose a deployment architecture for this Production environment:

1. Please describe the architecture with diagrams and some explanations. Feel free to use other services that you think will be helpful.
2. Please provide the Kubernetes manifest files at your best effort so we can see clearly what it

https://redd.it/emzw0h
@r_devops

Pathway for learning

I am interested in devops but have little knowledge about python akd network. Where do I start from?

https://redd.it/emzi9a
@r_devops

ISPConfig with remote dockerized webservers

I use IspConfig for managing some webservers and its great, I have some servers that are running PHP/Nginx in a docker compose environment but I would like to be able to remotely control and configure them using something like ISPConfig, anyone know of anything out there that is capable of this ?

https://redd.it/emr4xv
@r_devops

Starting new Job as a consultant - any advices?

Hello
Ill start a new job as a consultant for DevOps related projects.

I thought about running a private git server and wiki for documenting all the things I did and the lessons learned.

Apart from that, are there things I can prepare or keep in mind that help me performing better?

Kind regards

https://redd.it/emo99k
@r_devops

Gitlab AutoDevops: TLS handshake error

Hi guys,

Following this guide https://rancher.com/blog/2019/connecting-gitlab-autodevops-authorized-cluster-endpoints/ ; we integrated k8s cluster into GitLab, installed Helm & Gitlab Runner and they are running fine in a cluster. Created a project with Dockerfile in a root folder of the project. Enabled AutoDevops and ran pipeline. Build is failed https://prnt.sc/qlrwop with TLS handshake error, which seems like it cannot talk to the Internet and pull the dotnet image from DockerHub. Any suggestions/help on that?

Thank you

https://redd.it/empwgu
@r_devops

Teamcity Nodejs Test tab not showing

Hello,

I am using this Nodejs plugin [https://teamcity.jetbrains.com/viewLog.html?buildId=2539985&buildTypeId=bt434&tab=artifacts](https://teamcity.jetbrains.com/viewLog.html?buildId=2539985&buildTypeId=bt434&tab=artifacts)

I need to run test cases and generate code-coverage on Teamcity dashboard.  Does it require any additional step to be performed to get "Test" and "Code coverage" tabs to see the test results? #teamcity #continuous-integration

https://redd.it/emprdz
@r_devops

Any hosted SSH key managers that your trust?

I'm trying to get a grasp on all the ssh keys that are floating around but I don't want to add another internal service (like Vault) to maintain. Is there any hosted ssh key managers that you use? If not what would be holding you back from using them?

https://redd.it/emoi25
@r_devops

How common is it to get a signing bonus as a DevOps Engineer? If you get a job in the SF Bay area, do they help with relocation as well? If not, is it okay to ask if they do?

Im from Jersey and looking to move to SF Bay Area and want to explore my options before beginning the job application process.

https://redd.it/enau12
@r_devops

How to effectively monitor and respond to events in Openshift?

Hi

I got challenge to project dashboards or another solution where support the support team will see when there is a problem with application and where will be some place where in where in case of failure immediately they can see point of failure and incident response path. In small company with traditional java apps we can use nagios and check status of servers and load, ram, cpu and is java process. But now it will be something different that will be around 60 namespaces with around 30 microservices in one namespace. So I think that integration with nagios isn't good because this is changing enviroment and there will be many false postives.

We thought about Grafana. One main dashboard with all namespaces and there will be some value that will determine whether the namespace is working properly and then when we click on this tile with the name of namespace then we will see all tiles with microservices in this namespace and then when we again click on tile with name of microserivice some information what happens with each microservice (ram, cpu, requirements etc.). But that make problem that we will have a lot of dashboards in Grafana, a lot of clicking, a lot of queries to influxdb and prometheus and I'm not sure that isn't it too aggravating. For example 60 namespaces with 30 microservices it is 1800+dashboards in Grafana only for applications in private cloud. So maybe its always go into openshift and see whats going on. I don't know. I don't have too much expirience with this system. Maybe someone know how big companies monitor microservices and how they respond to failures? They have kibana with elasticsearch too.

https://redd.it/en8m29
@r_devops