EMPTY FILE IN GITLAB ARTIFACTS

Sorry if this is the wrong subreddit but I'm having an issue with Gitlab artifacts. We're trying to get test coverage written to a XML file as an artifact. We have marked tests split into two pipelines.

Build:
stage: Build
script:
- *setup-build
- *build
- touch banana.xml # where I write the code coverage collected by pytest-cov
- *push
artifacts:
paths:
- banana.xml
reports:
cobertura: banana.xml

Unit Test:
stage: Test
script:
- *setup-build
- *pull
- docker-compose run $DOCKER_IMG_NAME pytest -m unit --cov-report xml:banana.xml --cov=app --cov-append;
needs:
- Build

Validity Test:
stage: Test
script:
- *setup-build
- *pull
- docker-compose run $DOCKER_IMG_NAME pytest -m validity --cov-report xml:banana.xml --cov=app --cov-append;
needs:
- Build

After the Build and Test stages run, I go to download the banana.xml file from Gitlab but there's nothing in it, even though the jobs say `Coverage XML written to file banana.xml`

Am I missing something with how to get the total coverage written to an artifact file when splitting up marked tests in a gitlab pipeline stage?

https://redd.it/oozjqu
@r_devops

How's your experience with Pulumi?

I have been playing around Pulumi for a while and I have a couple of questions for DevOps developers and for Pulumi users specifically, I hope you don't mind answering them.

What are the challenges that you faced working with Pulumi?
Would you change anything in the tool if you have the power to do so?
Are there any pain points using the tool?
Thank you for your time.

https://redd.it/oow4sd
@r_devops

CircleCI strikes back again: hide per-user charges

Recently, I've decided to review our billing statements on CircleCI and discovered that the user payments disappeared. I was really surprised because when I complained about it to their team earlier this year, I was told that it is not going to change.

I went to their pricing page but it still stated that they charge per active-user. It took me a couple of minutes to find this post: https://discuss.circleci.com/t/performance-plan-billing-update/39502. In short, they hid the per-user charges by mixing it with the charges for running build jobs such that now it is harder to tell how much you are paying for the users. It is still trivial to compute it but it is not visible on the invoice anymore.

This per-active-user charges always annoyed me since we upgraded to Performance plan. It is basically allows them to grab more money from their customers claiming lower costs on their pricing page. They even charge you for a system user triggering "cron" jobs. So you are paying $15/moth for using the "cron" jobs feature. For us, the per-user charge reaches almost 50% of the total bill on an easy month so it is a significant part of the total charge. Hiding it in the invoice seems to be a dirty trick.

Anyhow, I would like to renew the discussion started on https://www.reddit.com/r/devops/comments/d3sqr0/whats\_your\_experience\_moving\_away\_from\_circleci/. What are the best alternatives to CircleCI in 2021?

https://redd.it/oou34v
@r_devops

Migrate from Heroku, Postgres, MongoDB to local server

I had made an application which was running a React frontend on Firebase, Django backend on Heroku, and using MongoDB & Heroku Postgres as database. As I was using Free Tier of all these services, I was approaching their usage quotas.

My university recently allotted me a bare linux machine (with generous specifications) which I could use for hosting my app. I've received SSH credentials for the same. Can someone advise me how I should go about the process?

https://redd.it/opazu5
@r_devops

ChaosNative Workshop

Hello everyone!

The ChaosNative community is glad to invite you to the Cloud-Native Chaos Engineering workshop where you can network with tech geeks around the world. Get yourself exposed to the world of resiliency and reliability. You will be having interactive talks and discussions with our experts about:

1. Hands-on experience with a workshop on Cloud Native Chaos Engineering which introduces you to the basics & fundamentals of Chaos Engineering.

2. A sneak-peek at the latest evolution in the Cloud Native Chaos Engineering world with LitmusChaos 2.0.

3. A demonstration of running Chaos experiments in just a few steps!

Mark your calendars for 12th August 2021, 11:30 AM EST | 5:30 PM CEST | 9 PM IST

Register here: https://chaosnative.com/cnce-workshop

https://redd.it/opblpz
@r_devops

What is IT like in your organization

We struggle with some basic competence and what I would consider core problem solving instincts and process of elimination when it comes to looping in IT. DevOps doesn't have the bandwidth for everything, though we try to automate more and more to not have to put in a ticket to IT.

Is IT maddeningly bad at your organization? I tried to give a terraform and ansible tutorial to a 5 member team and they were clearly struggling with concepts like a variable. We have to write the powershell scripts for them...Just feeling deeply frustrated and fed up.

https://redd.it/ootnoz
@r_devops

I've been working on some programs that interface with various saas services api's, and I need some methodology advice.

I'm just trying to see if my method is correct... I mean it works fine but it feels dirty... but I think it might just be my lack of experience.



So let's say this API wants XML data. What I've been doing is (in python) is something like this:

requestXML = """"""
requestXML += '<?xml version="1.0"?>'
requestXML += "<someRootElement>
requestXML += "<someElement0>{}</someElement0>".format(element1Data)
requestXML += "<someElement1>67890</someElement1>".format(element2Data)
so on and so forth...

I'm just adding in the xml data to this string, line by line, while also formatting them to include the variables I need. Then at the end I have this giant XML request string that I parse with urllib and then send, and its been working fine.


But is this really the best way to do this? I have this suspicion if someone more experienced ever saw this they would cringe lol



Would making a function to do this be any more "right"? House the XML "template" in the function and fill it in with args.


Thanks!

https://redd.it/opdmzu
@r_devops

Tutoring on the side

Just a question out of curiosity - has anyone been successful as marketing themselves as a "devops tutor" on the side? Teaching technologies and concepts on the devops roadmap, holding the hand of folks who are new to IT, or want to learn more about devops specifically?

Thinking of doing some of that in my off time and was wondering what others' experiences have been.

https://redd.it/opdu6n
@r_devops

How to Deploy a Secure API with FastAPI, Docker and Traefik

Putting your API to production comes with securing it with HTTPS and encrypting data transfer: something a lot of people neglect or take for granted.

HTTPS cannot just be turned on by changing a config file: a few steps are required.

In this post, I detail the process of securing a FastAPI app with HTTPS by using Docker and Traefik (the procedure is the same for other types of web applications)

Here's what's covered

A brief introduction to HTTPS: how does it work and why you should care about it?
Building a simple API with FastAPI
Introducing Traefik as and how it can handle HTTPS by integrating with Let's Encrypt and Docker
Deploying on AWS

https://towardsdatascience.com/how-to-deploy-a-secure-api-with-fastapi-docker-and-traefik-b1ca065b100f

https://redd.it/ope9ww
@r_devops

504 gateway timeout on containers

My company just started moving towards containers
Some of our containers throw random timeout errors to multiple users. When looking at my dashboard, those containers don’t seem to be reaching their CPU and Memory limit.
I’m fairly new to all of this and don’t understand why we would get this error if not all resources are used

https://redd.it/ope3iy
@r_devops

Cloud Veterans, really need your adept opinion!

Hi Folks,

I’m in the MVP stage of a solution, which I believe would greatly assist DevOps in performing their daily routine and save hours of work.

Also, shedding more light on what is MatosSphere. It’s an observability and remediation solution that was built for IT, DevOps, and Engineering with the goal of masking complexity, simplifying workflows, and accelerating operational productivity by bringing the most commonly used tools and services to a single integrated solution. We provide complete visibility to your core infrastructure services, by leveraging deep APIs integrations and policy enforcements and allow you to fix common problems while ensuring remediation is a single or no action task.

Appreciate it if you will share your feedback on MatosSphere or on things I can improve. Ping me and I will send you a 2-minute demo.

About me. I’m the Founder of Bayinfotech and CloudMatos (MatosSphere). So having two companies, one SaaS another Cloud Services Firm. Would love to help you with any answers, suggestions or assist you with advice.

Chat with you soon!

https://redd.it/opdv4m
@r_devops

ChallENGES with MOBILE APPS CI/CD?

I am a mobile developer, Android and iOS. And I've used CircleCI, GitHub workflows, and now Tekton to setup CI for my personal/consulting work. When I was working for bigger teams/companies- I experienced that mobile CI setup had lot of challenges and took us sometime working with the DevOps team to get it dialed in. Mostly because DevOps team wasn't aware of nuances of mobile toolchains and worse we ourselves were clueless about setting up our build system to be used in CI environment eg injecting variables and secrets.

 

As DevOps professionals what challenges are you seeing (if any) around mobile CI/CD setup?

 

note: idk why the title is mixed case :/

https://redd.it/opjaow
@r_devops

Need Dedicated Server with Unlimited, unthrottled bandwidth

Hi there, I'm working on a project which requires the streaming of music, and this music streaming could easily exceed the typical bandwidth limits on most regular hosting services. I'm a full-stack JS dev and I had a moderate amount of DevOps experience, but I'm certainly no expert, so I wanted to reach out to the community and ask: Is there a dedicated hosting service where we can have unlimited bandwidth? I know it'll more expensive than a regular hosting service, but that's ok!

Any feedback is much appreciated! Thanks :D

https://redd.it/opkl7l
@r_devops

Rebuilding pipelines for multi-stage Dockerfiles and changes to dev workflow

I’m in the process revising my dev environment and CI/CD pipelines based on a bunch of helpful feedback I recently received.

For instance, I wasn’t using multi-stage Dockerfiles and had some anti-patterns in my CI pipeline that I’m in the process of researching and fixing.

Since I’m implementing multi-stage Dockerfiles, it also affects my dev environment which currently uses Minikube and Skaffold (and a separate Dockerfile.dev for each microservice). This is also causing me to consider better ways of doing some common tasks like running unit tests and managing database migrations.

I’m also considering moving away from Skaffold to something like Devspace, Tilt or Telepresence.

That being said I can’t articulate why I’d need to switch because I have some lingering workflow questions related to unit tests and database migrations. Having answers to those will help me determine if Skaffold will work for me in my new workflow, or if I should look at tools like Devspace, Tilt or Telepresence.

My questions:

First, I know I’d need to target the dev stage in the Dockerfile which, looking at the documentation for Skaffold, would probably involve just doing something like:

build:
docker:
dockerfile: Dockerfile
target: development-stage
buildArgs:
key1: value1
key2: value2

Is this correct?

Second, when it comes to unit tests I know there should be a test-runner stage in the Dockerfile. I’m thinking these should just be run for the microservice that was changed automatically upon save. Thus, there shouldn’t be a need to run something manually, or have a --watch flag, in a separate terminal. Is this correct?

Not sure if this would fall under the test section of Skaffold or Custom Tests. I know that Devspace has Hooks that I would think would allow this.

Third, and lastly, when it comes to running migrations I mostly have an answer for this. At least for DevSpace, it sounds like there would be a “post-sync hook” that would run database model migrations. If not, it looks like you can run something like devspace enter python manage.py migrate to manually do the migrations. Looking at the Skaffold documentation I’m not entirely sure how this would be done or of it would just be using kubectl exec -it on the pod to run migrations after the fact. Can someone elaborate?

Thanks for the help and clarifications on this.

https://redd.it/opm9w2
@r_devops

Starting new role...any advice welcomed.

So after spending the last 12 months taking the jump from traditional Ops guy to a more DevOps orientated role I've been asked to lead the DevOps implementation/provision at my work place.

Although my new title will make alot of people cringe "DevOps Manager" I'm mainly tasked with all things automation, IaC, supporting the dev team and general implementation of the DevOps culture across the business.

I was hoping some good people out there could offer any advice on transitioning into the role, dos and donts and any other experiences they've had with a similar role shift...any advice is much appreciated 🙂

https://redd.it/opm1ex
@r_devops

Filebeat on Ubuntu-18.04

God afternoon family,

I have a filebeat agent running on my Ubuntu-18.04, and I would like to keep it running on the background. Should also start automatically once the server is is up without manual intervention.

Any suggestion?

https://redd.it/opo21t
@r_devops

Microservices architectures

Hello,

I'm a delivery engineer at a relatively small startup. We are trying to position ourselves as a software editor (we do some professional services on the side).

The product is designed based on microservices architecture, and it causing me a huge problems when trying to integrate to clients infrastructure.

Usually the product is installed on the client infrastructure but it can also be delivered as a VM appliance.

The product is composed of 18 docker containers, and we're using docker-compose as an orchestration tool.

Integrating these docker containers is very frustrating because of multiple reasons:

Containers images are not documented, and no one want to take the responsibility to document them.
The containers produce a huge amount of logs which make debugging a very difficult task.
docker-compose sometimes fails with TIMEOUT issues when launching the containers (I have to increase COMPOSE\_HTTP\_TIMEOUT).
Every container image has its own version which gives headaches, because every client uses a different version of microservices.
Updating the product is a pain in the ass, especially when they change/add/delete containers environment variables/volumes.
The system requirement are just insane (16go RAM, 4 CPU ...).
Supervision is very hard.
...

I feel that we're doing so many thing the wrong way and I don't feel that docker-compose is for prodution. The product/dev teams won't change their processes and way of work and I have to face the client an explain/justify all this mess.

Are any of you guys using docker/docker-compose this way ?

Any suggestion on how to improve things are welcomed.

I'm not sure if this is the right sub to ask my questions.

https://redd.it/opkwu4
@r_devops

Hiring managers of /r/devops, how often do you actually look at the candidates github profile if they have one? I feel like if more people actually looked the job search would be breezy.

I can do the work, I've proven I can do the work through code, I have years of experience and a history of proving I can adapt to change, and I like to think I can put on a friendly face and prove I'm not a complete ass to work with. Why do I still have to go through the process of interviewing and getting rejected at hundreds of places still to land a decent gig? Feels like the process hasn't changed since I started as a junior almost 10 years ago.

https://redd.it/opp5wy
@r_devops

Script management/deployment

Hi all,

We have a lot of powershell scripts across our estate. We've not really implemented any source control or management/deployment tools. You can imagine the hours lost when we need to make any changes across 3 separate datacentres.

Does anyone have any suggestions for a source control/deployment tool? There's been discussions about using Chocolatey for 3rd party software/package deployment and we use Octopus for our in-house apps.

https://redd.it/oormd7
@r_devops

What I need for new small software company

Hey dudes and dud-ettes.


So I got hired for a small project where they have an app that they want to possibly grow and scale.

I did some googling and I couldn't find a general list (Super general question, I know) of aspects of a new small software company that I should consider. Here is what I came up with but I would like to see if I can get some input from The Reddit at the risk of being to general and exposing my naivety.


Documentation
wiki?
Secret Manager
aws secrets manager
keybase?
Image Container repo
docker hub image repo
Communication
slack
mattermost
google chat
Monitoring
GAP stack
Code Repo
gitlab (We decided on this already)
CI/CD tool
gitlab's tool set
Ticketing System
jira
monday?

I'm open for suggestions around each bullet point but I'm more worried about the root topics and what I may be missing.


I'm fully aware of the broad ask here so please feel free to be general with your responses.

Thank you for any help in advance

https://redd.it/ontb9j
@r_devops

I made a lot of noise about getting an actual raise... And I got one!

While the title is probably not surprising for dedicated DevOps Engineers or those further along in their careers, it's certainly a well-known trend that we work somewhere for two or three years, and then bounce for a real raise.

I started at a startup in a major city as employee #36, moving out from a role paying half as much in the rural Midwest as a QA Engineer. Because of this incredible opportunity, I was willing to start at 80k, mainly because that already seemed like so much money; asking for anything higher felt a bit ridiculous, and I was coming in as a "Cloud Operations Support Engineer".

As one of the founding members of the DevOps team in a true DevOps organization from the C-Suite and founding engineers down, I learned a ton. I was promoted to a DevOps Engineer, and working directly with seasoned Software Engineers on my team, and being a software delivery team as well, I had to ramp up my skill-set very quickly to keep up. Eventually I became a pretty core employee and the company size has nearly tripled since then and I got a few minor raises that brought me to 100k, but that was still substantially below my market value, which I confirmed through multiple interviews and offers I received while still employed in my current position.

The thing is, I really didn't want to leave. I love my company, the work, the team. Everything. And I've learned such an insane amount that I became very confident in my worth to the company and also realized in the grand scheme of things, 100k is a drop in the bucket for most organizations.

I had multiple conversations with my extremely attentive and caring manager who went to bat for me and got me a raise to 130k! I know that's probably not a lot for some of you guys, but for me, who was making $8.25 as a line cook at Buffalo Wild Wings just five years ago, no degree, no certifications (but a massive bookshelf!) it's really a dream come true to feel like I have "made it", and that I can rise to the occasion and carry my weight and be a leader on my team.

I'm sure for the company it was weird that someone would make such a fuss about their pay and not leave with allegedly multiple offers on the table, but it was a good lesson for me to learn to stand up for myself, and not just bitch about my pay to coworkers and friends. A lesson I will carry forward in my career and encourage other people to do as well.

All in all, I am making 50k more than I was when I started 2.5 years ago, and I have to give props to my org for being attentive to me.

https://redd.it/optq8o
@r_devops