Encrypting server-side emails using serverless workflows

G'day DevOps,

We wanted to share something we worked on as a PoC for our serverless workflow engine. The idea was not ours, but something that the group who ran the PoC dreamt up!

The problem they tried to solve was the fact that emails sent from internal systems typically only have an SMTP (or email) configuration with the generic username, password and transport security settings. But their requirement was that all of the attachments from the system sent to external emails (vendor support, managed service support or outsourced support) be compressed and encrypted.

Direktiv (open source edition) was configured with an SMTP listener, converts the email to a CloudEvent and deconstructs it into JSON objects. From that point forward the workflow does whatever they want to do (zip, encrypt, SMS password to a number).

We thought it was pretty cool and applicable to a lot of users - let us know what you think!

We've written a blog article about it below:

https://blog.direktiv.io/direktiv-encrypting-server-side-email-attachments-in-the-real-world-d18a7bccb36c

We also released version 0.3.4, a lot of features added:

https://github.com/vorteil/direktiv/releases/tag/v0.3.4

As always - we welcome feedback and questions!

https://redd.it/p2vpuz
@r_devops

The Platform Engineer - What it takes

https://engineering.razorpay.com/the-platform-engineer-db2b21434911

https://redd.it/p2usty
@r_devops

How does Autopilot augment Data dog to reduce risk in a CI/CD pipeline?



This blog is a continuation of the Autopilot story where we discuss how one can reduce the risk of releases by augmenting an exiting monitoring platform like Datadog. autopilot provides Realtime risk assessment of releases before a code is deployed into production and also deny releases that fail a minimum threshold.

Once Autopilot is configured, it will automatically fetch the logs from applications, pipelines and metrics. During the execution of a pipeline, it can compare risk scores of a new release against a baseline run to assert the quality of a release. Autopilot determines if it can promote a new update fully to production or push it back to the developer for debugging. The log analysis and risk- assessment get processed in a matter of seconds and provide automated decisions during the execution of a pipeline run.

The AI/ML-enabled intelligence layer in Autopilot uses supervised learning to improve its judgment abilities over time. SREs, as they evaluate the confidence score of any release, can change Autopilot’s assessment of the impact of errors and warnings. These inputs are like feedback to Autopilot, which helps it to develop a contextual understanding of specific applications and pipelines.

Read More How does Autopilot augment Data dog to reduce risk in a CI/CD pipeline?

https://redd.it/p2verc
@r_devops

Advice on CircleCI config

Here is my CircleCI config. I don't think I am using it "correctly" even though the tests run. Any thoughts on how I can improve it?

The app is run using Heroku, but I don't want to necessarily automatically upgrade Heroku because of database schema changes.


---

version: 2.1

workflows:
main:
jobs:
- build

jobs:
build:
machine:
image: ubuntu-2004:202107-02
steps:
- checkout

# Create network
- run: docker network create test_network

# Run postgres
- run: docker run -d -p 5432:5432 -e POSTGRES_PASSWORD=runner --name db --network test_network postgres

# Build flask image
- run: docker build -f flask/Dockerfile -t flask flask/

# Run flask image
- run: docker run -d -e TEST_DATABASE_URL=postgresql://postgres:runner@db:5432/db_test
-e DATABASE_URL=postgresql://postgres:postgres@db:5432/db_dev
--name flask --network test_network flask python manage.py run

# Run Tests
- run: docker exec flask pytest "app/tests" --cov="app" -p no:warnings

https://redd.it/p2y21u
@r_devops

New Book: CI/CD for Monorepos

Ie have a gift for you: a free, 50-page ebook on effective CI/CD for monorepos. The book is open source, and you can download it today.

https://redd.it/p2zq5c
@r_devops

Job title for someone who mainly works on CI/CD?

Interested to know what job titles people prefer for someone who primarily works on CI/CD in support of an Agile scrum, that isn't "DevOps Engineer" (e.g. DevOps is a culture, not a job title, etc).

The model we have right now is "DevOps Engineers" aligned to one or more Agile scrums. The DevOps Engineers are responsible for helping the scrum build, test and release software themselves using existing tools and APIs.

The DevOps Engineers don't touch the software code or support the apps in production (SREs do that), and they don't manage the cloud infrastructure (there is a separate "Platform Engineering" team for that).

Rather they help the app developers implement the right APIs in their apps to make sure things like logging, monitoring, unit testing, containerisation are all implemented and that configuration, secret storage and so on are all done properly.

"DevOps Engineer" seems to be okay, alongside SRE and Platform Engineer (for infrastructure), but in the spirit of the "DevOps as a culture, not a job title" I'm wondering if there is a better option for this type of CI/CD/Pipeline role?

https://redd.it/p2wy88
@r_devops

Domain knowledge for DevOps?

I am interviewing for a higher position (slightly inclined towards the business side) and the recruiter wants to know my domain knowledge. I was stumped because I have worked with banking clients, audit firms, healthcare and data analytics startups.

As a DevOps engineer, does the domain really matter since it's basically the same flow (SCM, IaC, Config, CICD, Monitoring)? Although I know the product we are building but I don't really know the nuances of these different sectors.

What domain knowledge should I look into if I have worked primarily for Banking and Audit clients?

PS. One pointer could be the difference in security audits across these sectors. For eg, healthcare has HIPAA.

https://redd.it/p31fol
@r_devops

Lost at new job, is it normal and how to overcome.

So this is my first devops jobs ever, it’s for a startup and they’ve given me projects that I need to complete. I’ve told them before in the interview that all my expertise with the tools are foundational and it’s simple and basic. terraform, docker, etc…

To which they seem to be fine with, otherwise I wouldn’t have gotten the job. But I’m actually lost as to what is going on and what I’m doing and it’s just the first week. The only things I’ve got is what they want me to do and that’s it.

I have been learning documentation and white paper for tools I need to learn. But I’m not to sure if I need to tell them I need some mentoring or if that will be an annoyance. I’m fine to do the work on my own, it’s just I need to know how to do it.

Last thing I want is for them to feel like they’re having to babysit me.

https://redd.it/p3377h
@r_devops

Dbt founder Tristan Handy on the changing face of the data stack

>“I don’t think it’s that [self-serve analytics\] are going to get more ‘complex’—it’s that they’re going to get more ‘sophisticated' ... The advancement that we saw in computer interfaces in the latter half of the 20th century was an increase in technological sophistication, but a decrease in end-user complexity.”

https://mixpanel.com/blog/tristan-handy-changing-data-stack/

https://redd.it/p32z4g
@r_devops

AMA Alert! We’re from Devtron Labs, one of India’s first open source platform for Kubernetes

We’ll be going live at 10pm EST and we look forward to your questions on DevOps, Kubernetes, running a start-up and working in the tech industry!

Check us out here - https://devtron.ai

https://redd.it/p36c74
@r_devops

How is bitBucket for cicd pipeline??

Anyone is using bitbucket for cicd? We have source code lying in bitbucket, that is reason I am trying to see if that worth exploring? How is it while compared to GitLab? I think GitLab provides end to end devops tool chain right from the planning to monitoring. Want to get reviews from the real users...

https://redd.it/p37777
@r_devops

What tool do you use to manage ECS Deployments?

We're thinking about using terraform to provision base infrastructure, (maybe with "stub" ECS services).

 

It would be nice to have a simple file that engineers could manage themselves (and that can live with application code), which when applied to ECS would create/modify services. e.g. set container images, env vars, scaling settings.

 

A key requirement here is really being able to do this via a declarative file format, and not by running ad-hoc commands in a CLI.

Does anyone have any good suggestions?


 

Thanks!

https://redd.it/p38993
@r_devops

Sharing some woes with Ubuntu and cloud-init creating a secondary IP on a single NIC

I had written out a question to ask r/devops about Packer building a template on VMware with Ubuntu 20.04.2 but was finally able to find the right combo of holding my tongue while wiggling my ears or whatever, and wanted to share. Mainly so in a month when I hit a similar wall I have somewhere for Google to find it.

The situation I was facing:

I had a weird thing where during the packer building of an Ubuntu template in VMware, the vm gets two IPs on the same nic. That fact isn't really a problem, but the resulting template has the same issue. If I build a vm off that template, either using VMware's OS customizations, not using them, or using any kind of terraform/ansible build process, the VM gets 2 IPs.

Why this bothered me was I've done every combination I could think of with the netplan yaml files to fix it on the final product. Also I had tried several different fixes on the packer build side, such as adding some user-data code to identify the nic (ens33/ens160), set mac as the dhcp-identifyier, etc etc.

VMware has a [KB article](https://kb.vmware.com/s/article/70601) that I thought was related somehow, but their workarounds are just editing the netplan yaml files, which didn't make any lasting change.

On a newly built template, building a VM, under /etc/netplan, I was seeing

00-installer-config.yaml
00-installer-config.yaml.BeforeVMwareCustomization
50-cloud-init.yaml
50-cloud-init.yaml.BeforeVMwareCustomization
99-netcfg-vmware.yaml

The contents of 50-cloud-init.yaml:

# This file is generated from information provided by the datasource. Changes
# to it will not persist across an instance reboot. To disable cloud-init's
# network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
version: 2
ethernets:
ens160:
dhcp4: true
match:
macaddress: 00:50:56:82:c5:59
set-name: ens160

I have attempted the fix, but putting a 99-disable-network-config.cfg in /etc/cloud/cloud.d doesn't help, and there is already a file under the directory containing:

/etc/cloud/cloud.cfg.d$ cat subiquity-disable-cloudinit-networking.cfg
network: {config: disabled}

that is put there by the autoinstaller. Both get ignored which I can't explain

In the end, the right combination was adding this code in the user-data:

network:
network:
version: 2
ethernets:
ens160:
dhcp4: true

and adding

'sed -i "s/dhcp4: true/&\n dhcp-identifier: mac/" /target/etc/netplan/00-installer-config.yaml'

as a late-command.

Also, in the build json file, having this for the boot command:

"boot_command": [
"<enter><enter><f6><esc><wait>",
"autoinstall ds=nocloud;<enter><wait>",
"<wait><enter>"
],

The difference was at some point I had added in between autoinstall and ds=... `ip=dchp` because I wasn't getting a dhcp lease for some reason.

I do have to set a different network block if I'm building in my local VMware install, because locally it assigns the interface ens33 vs ens160, but otherwise it's the same.

There's code to try and let cloud-init identify the nic, but it seems to fail so I just have to set it for either local or vsphere.

Hope this is helpful to someone out there... (also sorry if I mixed up a present/past tense somewhere, this started as a question and just edited to a solution)

https://redd.it/p36zeb
@r_devops

Who still uses vagrant and why?

Basically the title. Are people still using vagrant as opposed to something like containerizing an app? I just noticed that proton is using vagrant still and spins up a VM instead of a container to do build activities. Is this still normal instead of doing the same or similar work with docker?

https://redd.it/p3b75w
@r_devops

Everyone gets a container?

In this article, I attempt to answer some questions you have and explain some concepts you didn't know about software isolation i.e containers and virtual machines, docker and how some of those pieces fit together

https://blog.formpl.us/everyone-gets-a-container-b8f755b404e7

https://redd.it/p3apbr
@r_devops

How many of you actually like doing DevOps work, besides for the ridiculously good paycheck? What do you enjoy most about it?

I was reading this thread and found it hilarious that the top few comments are devops or directly related to devops, because I actually love what I do. Devs hating it means more job security for us and working doesn't actually feel soul crushing, at least for me. What do you think?

https://redd.it/p3d1nz
@r_devops

Hey all we are team Devtron! One of India’s leading open source Kubernetes startup - Ask Us Anything!

We established Devtron in 2019 to redefine how DevOps should be done.

Democratising devops is at the heart of everything that we do and we believe that something as fundamental as security shouldn’t be expensive which is why we have kept Devtron open-source. Our goal is to encourage developers to create new-age applications to enable large scale containerisation.

We bring together over 30 years of experience in the tech industry and have gained incredible experience and insight during our career and we’d love to answer your questions!

From running a start-up, working in the industry to what we ate for lunch - ask us anything!

Check us out here - https://devtron.ai/

https://redd.it/p3dylx
@r_devops

Use Jenkins or investigate a new CICD for front-end development work?

I recently joined a small company where I am the only front-end developer. I have very little experience with CICD tools in general, the only tool I used was CircleCI for my own personal project.

Currently, our back-end developers are using Jenkins which are hosted locally. I was thinking, I can reuse their Jenkins, or investigate a newer CICD tool like Gitlab, or Github actions etc.

I'm thinking of leaning more towards Jenkins because the infrastructure is already there (there is A LOT of Jenkins hate in the DevOps community, but I have no expertise in it to understand why).

What I want to do is something simple. When a developer creates a new PR, run tests, prevent the developer from merging until the tests pass; when the developer pushes the PR to the master branch, run the test again and also deploy it to production.

So while I was creating the pipelines, I came across two checkboxes.

\- Build when a change is pushed to BitBucket

\- GitHub hook trigger for GITScm polling (If Jenkins will receive PUSH GitHub hook from repo defined in Git SCM section it will trigger Git SCM polling logic. So polling logic in fact belongs to Git SCM.)

Can someone explain to me what is the difference between those two?

Thanks, everyone!

https://redd.it/p3e5da
@r_devops

Received equipment for my first job

This is new for me. What do you do when receiving you equipment for a project? I have to get used to a windows machine again, but aside from that, does anyone have any experienced suggestions or tips?

https://redd.it/p3fhld
@r_devops

Terraform - glorified documentation?

Hey,
I’ve been scratching my head over this - is Terraform really just a glorified documentation in most cases - in Kubernetes world? I use Terraform to just define networks, few VMs and a cluster. It doesn’t really fit in a CI/CD pipeline - there is a kubernetes provider but as far as I can tell it would be a pain to use it in continuous deployment.

What makes Terraform a „must have”? I could see the benefit when it comes to disaster recovery as you could deploy your whole infra with just a few commands but other than that how does it make your work quicker and more efficient?

Does anybody run terraform in their ci/cd pipelines? It would make sense if you had an application which is not dockerized and you would need your vms to look exactly as specified (so, packer + terraform). But in the container world?

I define my resources and apply everything manually and I have a weird feeling that I am missing something obvious.

https://redd.it/p39dlf
@r_devops

Weekly newsletter recommendations

Do you know about a cool Devops weekly newsletter that brings all the latest news in the world to your inbox?

https://redd.it/p3h6ge
@r_devops