iptables in container as a service, go or not go?

Following the containers best practices, that a single service should be a single running container, how would you approach the following requirement?

In the current environment, non containerized, I am running a single Logstash process. In same machine, I have set up the iptables to forward the UDP traffic from one port to another on same machine (interface).

​

Should be moving the iptables to its own container better approach?

​

If so, I need to took care of updating the iptables rule in case of scaling the Logstash.

​

Or, is it there a better way to forward the traffic? All other approaches seem like an overkill to be honest.


Yes, I am on AWS.


NLB is not the best choice as I have a lot of traffic and it could introduce higher operating costs.

https://redd.it/dy8w2s
@r_devops

Continous Deployment - Where to start?

Hello guys,

recently I already asked about this but my questions were too vague I guess so I want to try it again as I am pretty sure that there are more than enough experienced people who could give me some pointers.

I need help with the following:
I got a GitHub repository, a DO Fedora droplet, and a docker-compose file as well as a Dockerfile for my backend.

My questions are:
1. Where do I build my Dockerfile for my backend?
2. Where do I place my environment variables?
3. Is creating my own Jenkins for this already overkill?

I wanted to make use of GitHub actions but I am kinda confused by the workflow. Would I need to write a script which opens up a ssh connection to my server, clones the repo, builds the image and launches docker-compose?

Thanks so much!

https://redd.it/dy8iwt
@r_devops

Resources and learning path?

Hi all,

I'm looking for good resources (books, courses) to learn devops. Something that starts off with the logic and design and goes on to implement projects with the proper standards and tools in the industry, bonus point if it uses Python because I like the language.

Thanks for your help!

https://redd.it/dy2fnm
@r_devops

Terraform with multiple environments and workspaces

Hiya,

Thought this article might be of interest: [https://www.mooreds.com/wordpress/archives/3244](https://www.mooreds.com/wordpress/archives/3244) It covers how to use s3 based storage for open source Terraform for two different environments that might differ in terms of parameters. It also uses two different buckets to minimize the blast radius. As noted in the post, I didn't get a chance to implement this in production, but would love to get feedback on if this approach makes sense.

https://redd.it/dxz89d
@r_devops

What blogs, newsletters, or twitter accounts do you follow for your DevOps and technology watch?

Technology watch can sometimes be time-consuming because we don't always know where to look. So, I am giving you here the [9 sources (blogs, newsletters, twitter accounts) that I follow](https://www.padok.fr/en/blog/devops-blogs-newsletters-follow) daily. I'm interested to know which ones you follow too, tell me in the comment!

https://redd.it/dylj29
@r_devops

How easy is it to move out of devops (back to dev)?

Hi all, I'm a software developer with a couple years of experience, considering the move into devops. Given that I'm early in my career, I'm concerned about getting pigeon-holed into the field if I decide I don't like it as much as development. Is it reasonable to think that after a year or so of working in a Devops Engineer role I would easily be able to get a job in development again?

https://redd.it/dymj9r
@r_devops

Aws separated,segregated environment, how?

Google cloud has projects within an account, and all of its resources are separated, unless you setup vpc peering , there are no connections between the projects (environment)

I'd like to achieve this in AWS.
* Every env has its own vpc
* Completely disconnected from others.
* Has its own user access key that cannot view or list other environments

Can I achieve this in AWS with the above requirements? How?

https://redd.it/dymzbq
@r_devops

Question Re: Bitbucket Pipelines and setting up django tests

Notice: I'm not a devops engineer - just a django dev who is trying to automate more and learn more devops as I go. When it comes to CI/CD I'm very new and know very little. If you mention Docker, while I am familiar, we don't utilize Docker where I work so I am not a Docker Pro (entry level at best).

In Bitbucket, I have a 'project' repo that contains my django project level files, and each app for said project is in its own repo.

* `project_repo`
* `app_repo`
* `other_app_repo`

I want to ensure that my tests are passing (automatically) and I think Pipelines will allow me to do that.

I'm trying to configure Pipelines so I can run tests automatically when pushing up to an app repo to ensure tests pass, coverage is acceptable, etc.

Currently, to test an app here are my steps:

* clone in the project
* cd into project directory
* `virtualenv venv`
* `source venv/bin/activate`
* clone in the app `git clone https://bitbucket.com/myapp`
* cd up one level (back to the project directory)
* `pip install -r requirements.txt`
* `pytest myapp`

Is Pipelines capable of doing this or am I wasting my time?

https://redd.it/dyq15a
@r_devops

Looking for part-time DevOps related work

Hi guys. I am looking for a part time job/role (10-20hrs remote) during the weekends for some extra income. I am currently employed, but i have spare bandwidth on the weekends. High level overview of my skillsets are:

1. Terraform
2. Docker
3. Ansible/Puppet/Chef
4. AWS (VPC, EC2, ECS, ECR, Lambda, IAM, RDS, Cloudfront, Route53)
5. Elk (Log transformation/Ingestion/Graphs)
6. Jenkins
7. Python/Bash/Ruby/Java/C

Please send me a message if theres any interest. Thanks for reading

https://redd.it/dyr2sn
@r_devops

Application for route json messages to the multiple services

So i have a question if there is an application that can do this or if we have to build one on our own.

>We have a service which runs in multiple containers in ECS it can connect to A, B, C, D etc system depending on what is received in a json message. It will start a client module connected to the B system and do the action. All well and good the problem is that after that has happened we would like to route all new messages to the same container since it already has an active client connection.
>
>
>
>The messages are today consumed directly so i can't really use a loadbalancer unless i'm missing something it will just place the json messages in the container it Algorithm chooses and then we will end up with wasted resources since there will be client module B connection on all of our containers.

We have looked a bit at RabbitMQ or some other Message Broker which i think is the solution but i'm not sure if they can go down to such a detail or something else maybe anyone have a similar use case?

https://redd.it/dymifm
@r_devops

New release: CloudFormation CLI (open source) and support for third-party resource providers

This has been rumored to be coming for a while. But AWS has finally pulled the trigger. A new native open source CLI for CloudFormation and support for native third-party resource providers (not custom resources). Also, there's a CloudFormation registry out of the box now for future third-party providers to be hosted.


https://aws.amazon.com/blogs/aws/cloudformation-update-cli-third-party-resource-support-registry/

https://redd.it/dywrqe
@r_devops

Organizing Hashicorp Vault secrets?

Are there any best practices to organizing Vault secrets? Right now it seems like it’s the Wild West with secrets in different places.

I have AWS secrets here, client secrets there and I’m just a bit overwhelmed.

Any helps would be much appreciated!

https://redd.it/dyv3xd
@r_devops

Untested changes merged to our develop branch end up in production. What should we do?

We work by having a develop branch and a master branch. The develop branch can be deployed to a dev or test environment. The master branch can be deployed to QA and production. When we start work on a feature, we create a feature branch based on the develop branch, and then merge back into develop, when we want to start sending the feature through our TEST environment(where it gets verified by the business supposedly - rarely ever happens). In QA Operations are meant to verify that the deploy went okay.

Today we experienced that some old untested changes merged to the develop branch were automatically deployed as another team had merged their changes into the develop branch and merged it into master and deployed to production. We want to avoid this from happening.

My thoughts on this are the following:

\- We could drop the idea of having a develop branch and maybe ensure that we can spin up the test and QA environments. I'm not sure why it matters to have both a DEV, TEST and QA environment. When I ask I get some very vague answers. Instead when we're done with our work we merge directly to master.

\- To assume master is okay, we set up good automated test coverage.

\- We split up services into microservices, so we don't run into the issue of multiple teams working on the same service all the time. Many of our services could easily be split up.

\- We add feature toggling for new features, so they by default are disabled when deployed.

Myself and the company I work for are very far from working in accordance with all the DevOps guidelines, but I'm trying to push the ones that seem to solve our problems. I'm not too sure about our branching strategy as above though. Thoughts? I'm thinking in must be a very common problem that's been solved a million times.

https://redd.it/dys8rj
@r_devops

Chef inSpec use case

I do all of my deployment using Ansible. That being said, what would be the purpose and use case to add inSpec to my workflow?

https://redd.it/dyqqli
@r_devops

Terraform increase tag number on an auto-scaling group

I'm trying to figure out a way of increasing a number at the end of the name tag value for EC2 instances that get brought up by the ASG.

resource "aws_autoscaling_group" "asg-web-1" {
name = "asg-web-1"
launch_configuration = "${aws_launch_configuration.web-launch-config.name}"
min_size = 1
max_size = 3
vpc_zone_identifier = ["${aws_subnet.private-subnet-1.id}"]
target_group_arns = ["${aws_alb_target_group.web-alb-target-group.id}"]
health_check_grace_period = 60
health_check_type = "EC2"

tags = [
{
key = "name"
value = "b-t-web-"
propagate_at_launch = true
},
{
key = "role"
value = "b-t-web"
propagate_at_launch = true
}
]

lifecycle {
create_before_destroy = true
}
}

So for the value for the name tag of b-t-web-, I would like to add 1 to the end and count up from there for each instance that gets added. Does anyone have any recommendations on how to do this? I believe you'd normally use count.index + 1 but I'm not using the count object because its not included in the resource.

https://redd.it/dz1gfp
@r_devops

Monitoring row count on Oracle DB

We currently use a Nagios plugin to execute some SQL against an Oracle database that returns a row count. It alerts If the row count is outside of some threshold. It works fine, just getting tired of creating checks for users that are able to write SQL, but we don’t trust creating Nagios checks.

I’m wondering if some frontend exists where users can input their SQL and have a check created?

Or Is there another solution not as cumbersome as Nagios that we could transition these users towards?

https://redd.it/dynvq9
@r_devops

Can it be true????

I've been searching for a solution for managing multiple K8 clusters, I even tried creating my own but it proved to be futile & I didn't have enough time so I threw it in the bin. This sounds too good to be true. Has anybody used this? Does it actually work? Anybody at Kubecon that can verify any of this for me? I'd be soooooo grateful and will give you karma

[https://thenewstack.io/portshift-attempts-more-granular-security-for-what-istio-service-meshes-lack/](https://thenewstack.io/portshift-attempts-more-granular-security-for-what-istio-service-meshes-lack/)

https://redd.it/dz3on3
@r_devops

Container based SSH/VPN bastion backed by AWS IAM

A bastion comes in handy when one needs to have a presence inside the VPC for checking on internal-only services. While there are a number of solutions for this (IAM backed ssh) delivered as ec2 instances, we have a container-based infrastructure, so it didn't make sense to setup instances just to provide a bastion. I also don't like polling, and prefer the immediacy of denying access immediately after removing someone from the IAM group that controls access. Check it out and let me know what you think.

[https://github.com/cloudtruth/aws-bastion](https://github.com/cloudtruth/aws-bastion)

https://redd.it/dz2q2l
@r_devops

Gotta tell someone about my first real "devops" success

Today I had an epiphany, and haven't been able to work since.

I'll start by saying I have the freedom one does when working for a small company (<100). I'm new to DevOps - even now, its still just a bunch of tools that noone but I will use lol.

Today is also a good day because I pu the finishing touches on a CI pipeline, and connecting said pipe to my local Ansible-configured virtualbox environment. I'm going through working out the kinks in the aftermath of an Ansible deployment: Why won't Nginx let me connect from my laptop's web browser? Is this broken, or just slow? Do I REALLY have to go back an change the deployment scripts AGAIN?

The last one is when it hit, man. Our test/QA gals struggle with this one *constantly,* depending on the environment they're in. Every time the want to deploy, they have to mod the scripts half the time (thanks a lot, Bamboo). The epiphany was this:

`If devs used this CI pipeline, they would experience the same problems as test/QA`

Considering how the devs currently do things, I can't shake the feeling that this is a win. And there's more: My first problem is the exact same problem that's been on my plate for a week now! I can finally do my work now!!! This must be how the South Park gang felt after pwning the World of Warcraft guy.

Maybe for Christmas I'll get to tackle the second problem, the fun one! But hopefully, a dev will us our new CI and beat me to it.

:)

Thanks for letting me share!

https://redd.it/dz5gj9
@r_devops

Devops for Windows?

I accepted a DevOps position at a company which develops and hosts web applications for large companies. [ASP.NET](https://ASP.NET) , staunchly Windows shop, use of Linux for any purpose is mind-numbingly forbidden.

Powershell is becoming my best friend, but finding myself kneecapped by the inability to spin up a Linux server. Open source products, password management solutions, telemetry software, monitoring software, reporting software, automation platforms, all must be natively hosted on Windows. Naturally this eliminates about 80% of the tools available to do my job, **especially** the ones that won't cost 5-6 figures and a several month acquisition process. Anyone in a similar situation, have devops tools they use for Windows web shops, tips to handle?

https://redd.it/dz56ys
@r_devops