Introducing new processes to my team.

In my team I got the task to introduce and supervise new processes. This was not really specified. We are a little R&D team with embedded projects. One or so year ago I introduced the team to the concept of unit test. It is hard to do in embedded projects except for modules and drivers. e.g.: specific temperature sensor is mockable and it is worth to unit test since the driver is reusable in other projects.
Not so long ago I started to introduce the idea of CI/CD to the team, and they like it. I got three months to implement this new processes. Since I myself is selftaught in this I have a vague understanding, I would like to ask for your help to not make obvious mistakes. We use Bitbucket and JIRA and rarely Confluence. In the last year or so I learned about CI/CD, docker, ceedling for C test.
My ideas so far:

1. Delegate unit tests for the team. ( we have a few driver which would worth to unit test).
2. Implement a coding guideline or even a linter. (Of course team decision which one)
3. Introduce new processes, like:
1. How a pull request is accepted(who review which kind of code, how many reviewers)
2. How you should name your commits. (JIRA ticket number, descriptive msg)
3. Documentation. ( I am not big on documentation, so I would accept any advice here.)
4. Better documentation for tools.
5. Introduce some kind of retrospective. ( I have a really vague idea about Agile, and I don't dare to throw it in, since when I my team used Agile in my previous company, according to a senior dev it was many thing but not the real Agile. )
4. Also we use git submodules and I heard it is obsolate solution. The idea of how we want to use it is:
1. We develop a driver (e.g.: to a specific Accelerometer).
2. We write unit tests. ( I know TDD would be better, but first I am happy If they learn the ceedling part).
3. We push it to its specific repo where other projects can reach it download it and modify it if necessary (e.g. you not implement the full driver for your project, but other projact can use the left out feature).
4. ... profit. I am not sure how to introduce it to my or others project the specific driver to not ruin the version control part or use the slighltly obscure submodules.

If you have other ideas, or correction please feel free to write about it. If you want to talk about your team horrific processes you are still welcome.

https://redd.it/o5kt09
@r_devops

How much do you spend on monitoring?

Curious if there's an industry benchmark for saas companies. We spend about 7% of hosting on logs metrics etc. And i know it's too much because i can see the waste in there. But it got me thinking, how much should it be?

https://redd.it/o5mimh
@r_devops

Is learning vue/frontend beneficial?

I am working as a backend and a bit curious if i want to be a devops, is learning vue/frontend framework would be beneficial for my end goal?

Or those time are better spent learning and mastering more on infrastructure/server/network/container side?

https://redd.it/o5mqyz
@r_devops

Upload bash script to Ubuntu machine deploymed by MAAS

Info + objective:
----------------

I'm using MAAS to deploy workstations with Ubuntu.

MAAS just deploys the machine with stock Ubuntu, and I then run a bash script I wrote to set up everything needed.

So far, I've ran that bash script manually on the newly deployed machines. Now, I'm trying to have MAAS run that script automatically.

 
 

What I did + error:
-----------

In the MAAS machine, I create the following file curtin file called /var/snap/maas/current/preseeds/curtin_userdata_ubuntu which contains the following:


writefiles:
bashscript:
path: /root/script.sh
content: |
#!/bin/bash
echo blabla
... very long bash script
permissions: '0755'

latecommands:
runscript: "/bin/bash /root/script.sh"

However, in the log, I see the following:

known-caiman cloud-init1372: Command: '/bin/bash /root/script.sh'
known-caiman cloud-init1372: Exit code: -
known-caiman cloud-init1372: Reason: Errno 2 No such file or directory: '/bin/bash /root/script.sh': '/bin/bash /root/script.sh'

 

 

Question
--------
I'm not sure putting such a large bash script in the curtin file is a good idea. Is there a way to store the bash script on the MAAS machine, and have curtin upload it to the server, and then execute it? If not, Is it possible to fix the error I'm having?

Thanks ahead!

https://redd.it/o5kiuj
@r_devops

Tools to create a development VM

Hello,

I'm pretty ignorant on devops tooling. I'm trying to figure out the baseline tools needed to create a development vm for my development team.

Effectively, I want to create a Ubuntu 20.04 iso for VM Ware with some basic software already installed and configured: compilers, IDEs, docker, some more software packages. I'd like a reproducible way to create this VM, but once it is created I can host it for developers to pull down.

Tools like Packer / Vagrant look promising, but I'm not sure. Does anyone have a guide for this sort of developer environment setup?

https://redd.it/o5q24n
@r_devops

Enterprise On-Prem Container Image Hosting

Currently, we are using Harbor as our container registry. Either due to lack of knowledge or limitations of Harbor, we are finding it very hard to manage it at scale. For audit and security compliance reasons we have 5 instances that are separated at the network layer with only some users having the ability to replicate images to make sure only approved images are able to be run in production.

The biggest issues we have are

Nested group membership in AD
Keeping project settings in sync between the multiple instances
Ensuring only approved images go into a project that is used for production
Cleaning up users who have been offboarded. (why do we have to do this at the database level if they are ldap users)

Does anybody have any suggestions for a more robust on-prem container image hosting service that works for larger enterprises with strict security compliance requirements that allow less than technical members to approve images for production use?

Thank you,

https://redd.it/o5ozf6
@r_devops

How do you handle 'silent errors'?

Our applicative logs contain error entries: Commonly, the code keeps reporting about custom errors, 99% are not severe rather non-happy scenarios that are easy recoverable (e.g. failed to approach some 3rd party API). Naturally with time, given this noisy log, we started to ignore these error entries. Everything is OK usually, the app is up, until one day some error will be really meaningful and lead to customer-facing consequences. How do you handle the visibility and alerts of your errors?


Yes, we have common alerts for black-box metrics (e.g. latency, error rate), it still doesn't protect us from ignoring a really severe error one day

https://redd.it/o5oeth
@r_devops

What are your experiences using a PaaS for hosting your containerized apps?

If so, which one? We're primarily an AWS shop, so wondering if EKS is a good route. Or if there are good abstractions that sit on top of EKS that may be worth a look? Ideally looking for something that streamlines the deploy/release stage, too.

 

HashiCorp Waypoint looks interesting, although fairly young. What else should I be looking at?

https://redd.it/o5tyya
@r_devops

How to monitor for exposed domains?

We have qa/stage domains that sometimes get exposed when admins renew a cert, or whatever reason. Are there any tools, preferably free, that can monitor if our stage domains are exposed to the world?

https://redd.it/o5oka3
@r_devops

Here is something worth reading- How much testing is enough.

A familiar question every software developer and team grapples with is, “How much testing is enough to qualify a software release?” A lot depends on the type of software, its purpose, and its target audience. One would expect a far more rigorous approach to testing commercial search engine than a simple smartphone flashlight application. Yet no matter what the application, the question of how much testing is sufficient .....

https://testing.googleblog.com/2021/06/how-much-testing-is-enough.html

https://redd.it/o5kmq9
@r_devops

Too many security products, too little time - help?

Right now we have a lot of security and code analysis products in place at my company. We're transitioning from "Nobody works in devops" to DevSecOps + platform engineering - it's already too much to do, so we're looking at what we can consolidate for all of that, and I'm not a security guy.

Current security products and what they do that we have in place:

* Anomaly detection - AWS guarduty -> Alienvault
* Host vuln -> Alienvault
* Static code analysis -> codacy
* Dependency analysis -> snyk.io
* Rapid7 Insight VM -> dashboard and security project management
* Rapid7 InsightAppSec -> Dynamic Application Testing
* Container scanning -> AWS ECR

I'm looking for help finding good products that we can consolidate on to meet our needs, or if we really do need a million different security products. How are you guys doing security and what products do you use?


Things we care about:

* Infrastructure as code, even for our security products if possible - Rapid7 still has a lot of manual configuration steps that make it hard to maintain
* As few products as possible to meet our needs - we have way too many logins and dashboards for my team to maintain right now - trying to reduce that surface area.

https://redd.it/o5v627
@r_devops

For those that use Sonarqube for SAST, what security vulnerabilities do you break build on?

Just wondering what you guys break build on? We are implementing Sonarqube and wanted some guidance on what to consider.

https://redd.it/o5zvsa
@r_devops

Live Terraform Coding Interview Session

I’ve posted this in r/sysadmin but would like to share it here as well.

I was interviewing with a company that I really wanted to work for. Position is Senior DevOps Engineer. 3rd interview was live terraform coding session. Nothing else but Terraform.

I’ve interviewed with few companies and they all had live coding session using high level programming language but none actually did live terraform coding session.

I was asked to create environment to have nginx container up and running.

I can explain from top to bottom what’s needed to achieve this and how I would implement the environment but live Terraform coding session??

Has anyone ran into this before? I might be wrong but feel like this is just poor interview setup.

https://redd.it/o5wybe
@r_devops

Looking for an HTTP load-testing tool that can POST variable payloads

I'm doing load-testing of a backend API service, and I need to be able to increment a JSON field in the payload on every request. Most of the HTTP load testers I've looked at seem to be oriented at reproducing complex workflows for front-end stuff. I just need to hit one endpoint at variable rates while incrementing a counter in the payload. Does anyone happen to know of a tool that'd be able to accomplish this? Simpler is better in this case.

https://redd.it/o60i7g
@r_devops

Phorklift is an HTTP daemon with clear, powerful and dynamic configuration

The main feature of Phorklift is using Lua as configuration. It's very clear in most cases. While it can be powerful if need. It can even create/update/delete the configuration dynamiclly, may be used for e.g. micro-service.

The project's page describes this in details and gives some configuration fragments. There are also some more complete configuration examples.

I think this is suitable for devops.

https://redd.it/o63cnh
@r_devops

Creating stored procedures through UrbanCodeDeploy pipelines.

I was wondering if anyone here has some idea on creating stored procedures in Microsoft SQL Server through UrbanCodeDeploy pipelines. I tried googling but wasn't able to find any relevant solution

https://redd.it/o62qog
@r_devops

For those doing CI/CD for Xamarin, do any of you use GitHub Actions but offload to your own server?

Do you recommend this as one way to go or do you prefer using AppCenter and integrating that with GitHub?

https://redd.it/o64sgv
@r_devops

Software Dev to Devops

I am thinking of getting a few certs to make a career change:

- RHCSA
- AWS certified Developer
- AWS Devops Engineer

I don’t have much experience in any of these areas and have a training center near by so I am thinking it couldn’t hurt. Would you include anything like VMware? Do you think certs hold value? Appreciate all opinions!

https://redd.it/o64kpj
@r_devops

Is it just me, or do companies think DevOps engineers can solve all their problems

I've recently been looking at applying at companies to grow my career as a DevOps engineer.

I'm noticing that allot of companies don't seem to understand the role of a DevOps engineer or the culture of DevOps.

For example, they would list requirements that are not at all DevOps related and more security or network related, which in that case should they not just hire a network engineer ?

And regarding security, should they not then list the position as a DevSecOps engineer ?

Im getting really frustrated.

It almost feels that companies think if they have a issue they can't solve, they need to hire a DevOps engineer , since we are some type of magician that can solve all of their problems.

Is it just me thinking like this ?

https://redd.it/o66jto
@r_devops

Build Cross-Platform apps with .NET Core with these best practices

https://arohi-adhyaru.medium.com/build-cross-platform-apps-with-net-core-with-these-best-practices-9dcc7d31d15a

https://redd.it/o661lz
@r_devops

Getting into DevOps

Hello,

I’m after some information that would help me get into the DevOps role as I unfortunately flunked out of my CS degree (final year) as I was not in a good situation.

Currently, I am in the “do everything from web dev, customer service, setup and configure networks, AD management, to backups, patch management for software and servers, AV endpoint management, incident response, Azure/365 admin, diagnose common problems with software, hardware and all three major OSes, hardware repair, random security related tasks and the list goes on” role at a small company for 1 year. (Don’t even know the title for my job would be)

I am working towards getting entry-level AWS certificates and RHCSA as I see them to be the best for getting my foot in the door (would prefer to be vendor agnostic with Linux), as well as playing with Git, Docker, Ansible, Terraform and K8s, while concurrently playing with Python, Golang and ML. Going to start with Jenkins soon. I have always been interested in the Pen testing role (also working towards OSCP), but am open to the wider InfoSec/CyberSec role.

Am based in the UK. Any suggestions on what I should do break into the space? Not many junior roles around me and worried that the lack of degree would close a lot of doors on me.

Thank you.

https://redd.it/o68mxc
@r_devops