Looking for an AWS centered Airflow/job/batch management (python)

Hi,

I'm having troubles finding projects like airflow/luigi but that would be centered on AWS.

My goal is to :

* run anychronous but still dependent processes in ECS
* write workflows/tasks as code
* uses cloudwatch for log storage and retrieval
* uses cloudwatch for metrics storage, retrival being done with grafana
* be able to use aa REST api to launch the workflows, monitor them, etc...

The absolute requirements:

* free
* open source, hackable
* made in python

I checked the airflow documentation, it seems it'd a great product but it's doing too much things by itself, that could instead use actual AWS services. It has some integrations with lambdas and S3 but it seems like afterthoughts, even if they may work properly.

How do you guys handle this ? I don't want to spend my time rewriting dashboards that would analyze cloudwatch, ECS logs, etc..

Thanks !

https://redd.it/dx6vo9
@r_devops

When do you decide that you need to use containerization in production? what is the benefits?do you have a checklist or something?



https://redd.it/dx3864
@r_devops

Kibana is not responding.

Kibana is not responding.

Please see this https://pastebin.com/LW0QY1Ma

what is the issue ?

https://redd.it/dvpzgj
@r_devops

Need Help Starting Out

Hey guys,

I think I might be in over my head here. I wanted to get a custom IDE for my startup built. So we found Eclipse Che and figured it would be a good starting point to build on. I am not a technical person at all and just looking at the landscape of it out is overwhelming. I would love to have all the cool features of Gitlab and I think it is possible with all the extensions and plugins Che has floating about. I guess my question is how do I get everything I need done without being a technical person? Where do I find an Eclipse expert?

https://redd.it/dvlfl5
@r_devops

Why do you still use Jenkins?


I mean seriously it is an absolute dinosaur. Hard to manage, difficult to scale. Why do so many still use it over the more modern alternatives like Circle CI or Gitlab? Am I missing something here?

https://redd.it/dv81r1
@r_devops

New to this, probably not doing things correctly. Jenkins/Artifactory

I work for a large company with an existing infrastructure that I'm trying to piggy-back off of. I maintain mostly SQL scripts and other non-compile-ables in a Bitbucket repo. I'm trying to establish a consistent push from Bitbucket to a Windows server (overwriting existing files).

Initially, I used Jenkins (on Linux) mounted the Windows directory and cp'd the files from the workspace to the Win server. Basic but it worked, though it required sudo to mount the remote path.

We're supposed to push any Jenkins build artifacts into Artifactory though, and I shouldn't be using elevated permissions. So I've changed changed things up to deposit build artifacts to Artifactory and SSH a zip of the files to my win server, I can use a ps1 script to extract where they need to go - but I have to do that manually or scheduled (is there a way to trigger a remote ps1 script or a SQL Agent job from Jenkins?).

I really don't know Artifactory at all, and not sure if I should be trying to use Jenkins to push from Artifactory to my server, or pulling Artifactory artifacts from the Windows server on a schedule, or deploying from Jenkins to both Artifactory and Windows separately.

I also am utilizing SonarQube via Jenkins, but that's working fine currently.

**TL;DR:** What is the best way to move scripts from Bitbucket to a Windows server via Jenkins (on Ubuntu)?

https://redd.it/dvkw90
@r_devops

Composition over inheritance applied to Docker

The main benefit of Docker containers is that they are self-contained. For developers, that means one just needs to inherit from the desired Docker image that contains the necessary required dependencies, and presto, one can build one’s application deliver it to production. Most of the times, the process is pretty straightforward. Containerization allows scaling on unprecedented scale.

However, the downside of this self-containedness is that updating the parent image(s) becomes a nightmare.

https://blog.frankel.ch/composition-over-inheritance-applied-docker/

https://redd.it/dxo4sa
@r_devops

Issue with login to ECR inside Jenkins build agent

I am using Jenkins to build something and I want it to run a custom image that is hosted on ECR. The keys have admin access (testing only - will restrict later) when I go to put it in I get a 401 error seen below. I have also included how I have my build agent set in my pipeline. I was hoping someone solved this already, google has just turned up the 'I figured it out thanks every' posts.

​

pipeline agent

agent {
docker {
image '<id>.dkr.ecr.us-east-1.amazonaws.com/node_build:latest'
registryUrl 'https://<id>.dkr.ecr.us-east-1.amazonaws.com/node_build'
registryCredentialsId 'aws-creds'
}
}

&#x200B;

Error message seen in Jenkins

Running on Jenkins in /var/lib/jenkins/workspace/Deploy UI
[Pipeline] {
[Pipeline] withEnv
[Pipeline] {
[Pipeline] withDockerRegistry
$ docker login -u <Access key> -p ******** https://<id>.dkr.ecr.us-east-1.amazonaws.com/node_build
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: login attempt to https://<id>.dkr.ecr.us-east-1.amazonaws.com/v2/ failed with status: 401 Unauthorized

https://redd.it/dvh4rk
@r_devops

Testing Packer Images

Is there a better way to test Packer builds beyond the fact that there were no errors in the build process? It doesn't look like Packer has anything for hooking some kind of validation of the system before finishing the build, so I'm not sure how or where it would happen. Has anyone ever done something like this?

https://redd.it/dxoelb
@r_devops

Are there new questions/tests you're hearing in interviews you weren't getting a year ago?

Hey, I'm pretty early in my devops career and have just been getting thrashed in my last couple interviews. Most of the 'here are questions you need to prep for' articles are a year old at best and I've read a ton of them.

Any newer ones you might be asking your candidates or are hearing yourselves that you didn't have a year ago?

https://redd.it/dxo6pt
@r_devops

Lambda function not being triggered by SNS notification

I have an application that does some processing which puts out a SNS notification depending on the results. The application works fine - I can see the console log of the notification being sent and see in cloudwatch that none are failing - however I am unable to get a lambda to run once the notification is sent out. There is an existing subscription to the SNS from the Lambda which was created using the serverless framework and I have confirmed in the console that a subscription exists.

&#x200B;

What are some steps I can take to troublesoot this? I'm a bit lost today. Also any ideas on cause would be great

https://redd.it/dxq0ce
@r_devops

protobox - gRPC dependency management

We needed a way to manage the protobuf dependencies in our projects were i work, so we made a tool for it. Check it out and give it a star if it tickles your fancy.

[https://github.com/UNIwise/protobox](https://github.com/UNIwise/protobox)

https://redd.it/dxmggt
@r_devops

How do you guys manage secrets and API codes in github ?

Are you using any specific tool such as vault or console or perhaps base64 to do it?
What do you think best way to do this ?

https://redd.it/dxksw8
@r_devops

Video-game programmer (c#) getting tired of my dayjob looking for the right path

I work as a bartender part-time as I pursue making my dream projects. Bartending has been fun/lucrative for a few years but I'm getting really tired of people. My friend recommended I get AWS certified and see where that takes me. My ultimate desire is to find a half-decent paying job that is a bit more cerebral than bartending and still allows me enough time to program. Any help is very appreciated.

https://redd.it/dxvqps
@r_devops

Unrecognized services in Puppet Enterprise

I have tried to run the following:

puppet resource service puppetdb ensure=running

and receive the error

Error: Could not start Service\[puppetdb\]: Execution of '/sbin/service puppet start' returned 1: puppetdb: unrecognized service

service { 'puppetdb': ensure => 'stopped', }

I am also running into the same issue with puppetserver. I suspect this may be a certification issue, but wanted to run it by others.

https://redd.it/dxxpsd
@r_devops

AWS ECS Fargate and security ?paranoia? - interview question

I have been a fan of Fargate since it launched, mostly because I like the idea of not having to manage any EC2 instances AND being lazy/reluctant to learn Kubernetes. I had an interview with one of the media companies and one of their senior infra/systems engineers asked me what is my opinion about security of Fargate from the perspective of not being able to "see" where containers are running and if someone in AWS had access to exec into them and lets say see all the secrets that are exposed as environment variables inside of those containers.

I honestly never thought about this and didn't know what to say so I just winged it and said that I think in my opinion the same question can be asked for any managed service, including EC2. There are contracts between us and cloud providers and when you signed up for a fully managed compute services such as Fargate (or any service at all) you already took into consideration and accepted those risks. Also, according to my understanding of AWS shared responsibility model, such incidents would mostly fall under AWS belt.

How would you have answered a similar question? Can we trust cloud providers? - IMO yes! that's the biggest cost and risk of cloud computing which clearly many are willing to pay.

Btw, I passed the interview - but didn't continue with the further process.

https://redd.it/dvk25k
@r_devops

Build and maybe deployment pipeline for hobby projects?

I'm a developer and mainly do development in Nim on MacOS and deploy to linux in the cloud. Nim generates a native executable using GCC. At the moment I have only a couple of artifacts - a crawler and a web server, but I expect the number of artifacts and instances of them to grow over time. At some point I'll need to add a queue and have multiple crawlers pulling jobs assignments off it. They use a hosted database. The code is all in github.

Since Nim generates native executables - which can't be shared between MacOS and Linux, I'm currently manually (well via scripts) pushing source code to a server and compiling it there before deploying it.

Any suggestions on how you'd set up a build pipeline for something like this? I need something simple and easy to maintain since it's only me. A nice bonus would be automatic deployments as well. I have experience with ansible and have only played around with Docker and k8s. I'm willing to learn about whatever tool but there are so many out there that I don't have the time to learn all of them. I need a narrow focus.

https://redd.it/dvigp4
@r_devops

Docker, Git, Jenkins and local image

Hi, we use Jenkins to build our docker container starting from a dockerfile pulled from Git (and other files). I have to build my container by dockerfile, i cannot modify Jenkins job. Docker build is executed by Jenkins, in my dockerfile i use a base image from docker hub but i want to export and use it locally (Reading from Jenkins workspace, stored in the same GIT repo of dockerfile ). Is there any way or workaround to start my dockerfile like "FROM local/directory" ? My server doesn't have access to internet and to docker hub.

https://redd.it/dvgl4g
@r_devops

Tutorial: Run NixOS using Vagrant

I've recently been experimenting with [NixOS](https://nixos.org/) and I've found it really interesting, so I wrote a tutorial on how to install and run NixOS using Vagrant. Includes instructions for running on both macOS and Ubuntu, but should work on most operating systems.

Maybe someone else will find NixOS as interesting as I have. :) I appreciate any and all feedback on the tutorial and the blog. Thanks! :)

https://lunar.computer/posts/vagrant-nixos/

https://redd.it/dveole
@r_devops

Has anyone here used NS1 for DNS service?

Reviewing providers and I had never heard of this one before.
https://ns1.com/

Thanks

https://redd.it/dv8z8t
@r_devops