Guide to secure a server/vps

What are the resources or guides you would suggest for a developer who needs to set up and secure a web server.

I have basically collected this much:

* SSH
* use cert
* disable root login
* change port (contested)
* fail2ban
* Accounts
* principle of least privilege (use specific accounts for only what their needed for)
* Don't run as root
* Firewall
* only have the minimal ports open (http,https,ssh) using ufw or iptables
* SELinux or alternatives (advanced)
* Orchestration concerns(maybe not related to tile)
* do it over a private sub net
* use ssh even then
* Secrets management
* don't store api keys, or certs on disk if possible and load into memory
* user virtualization to isolate host in case webservers are compromised

​

* Misc
* take an inventory of running services and installed software
* keep only what you need
* Logging/perf monitoring
* email,slack for realtime notifications
* backing up your logs in close to real time (in case of compromise for example)
* Always update
* Secure your individual applications (nginx,db,node etc)
* Advanced
* specific distros like alpine or void or build your own
* way smaller attack surface
* musl lib c.
* busybox

​

Cool references i found are:

* Linode/Digital Ocean documentation (basic)
* Arch Linux docs in general but specifically on security/hardening or other distros
* Alot of stuff in github repos in terms of guides but none are authoritative/guaranteed up to date

https://redd.it/qjc1jw
@r_devops

What's the best way to deal with config drift from GUI usage?

Azure's GUI is good. At least good enough that some devs (including me) simply _forget_ IaC exists and use the GUI to make the small modifications necessary for ops. Maybe a scale up of a database here. Maybe changing some permissions there.
The friction of a new PR to the IaC seems to be so high, that people are just not keeping it updated. Fast forward one year and everything's out of whack and we can't replicate any environments.

The simplest solution to implement is a human-process level one, where we simply exhort everyone to update the IaC when they change something. Clearly that hasn't really worked.

The solution that might work better is a drift detector, and maybe auto-applying IaC so devs are forced to PR any changes to the code. But clearly, the devs don't enjoy applying changes to things using code (since they're human too, and everyone likes GUIs) and I'm looking for something better.

I'm thinking that the drift detector should detect changes and make a pull request to the IaCodebase automatically, for modification and acceptance by the owners - since they already made the changes in the GUI. Perhaps they copy-paste configs to some other envs, and merge the PR.
If they reject the PR, the drift is corrected automatically. If not, no further work is necessary by the maintainers - they don't feel like their effort and time updating stuff on the GUI is wasted.


I've looked at older posts like
- [https://www.reddit.com/r/devops/comments/cgcstz/show\_reddit\_configuration\_to\_automatically\_detect/](https://www.reddit.com/r/devops/comments/cgcstz/show_reddit_configuration_to_automatically_detect/): Not Azure, core reco is just not using the GUI. Not great UX IMO, see above.
- https://www.reddit.com/r/devops/comments/60n5qa/how_do_you_manage_configuration_drift/ - this one is too low level for me, but configuration management DB and drift detectors are a good idea.

Overall, UIs like the ones Pulumi or env0.com provide don't seem to be exactly this either. Env0 is close, but seems like they provide their own GUI for specific things instead of re-use current workflows.

Disclaimer - this might be a problem specific to Azure, were the GUI is good enough to use but Azure's IaC support is bad enough to prevent full usage of tools like Az-templates/TF/Pulumi.

https://redd.it/qjgft1
@r_devops

Atlantis with Azure Dev Server

We are using the on-premise version of Azure DevOps Server 2020. I am having trouble getting Atlantis to authenticate with a git repo hosted on our Azure DevOps Server. I would appreciate any help you can offer.

The first challenge I had was that the on-premise version does not set the Request-ID header in the webhook that is sent to Atlantis. This was fairly easily resolved by running an instance of HAProxy in front and adding the header.

The second challenge was that there are a few hard-coded references to dev.azure.com which works for the cloud version of Azure DevOps, but not the self-hosted one. Thankfully this has been resolved in the Atlantis repo in the last couple weeks. It hasn't been released yet, but I was able to use a dev build of the container.

Now I'm stuck with the Git authentication. In the pull request I get an error that says 'fatal: authentication failed'. The comment from Atlantis says that it tried to run this command (personal info redacted):

git clone --branch dev --depth=1 --single-branch https://[username]:[token]@[our_on_prem_url]/[site_collection]/[project]/_git/[repo]

From the command-line on my dev machine, that command also fails. I tried all sorts of combinations of username:password, username:token, username:base64-token, etc. All failed. I am able to get Git to authenticate when setting the authorization header this:

git -c http.extraheader="AUTHORIZATION: Basic abcdefghi" clone --branch dev --depth=1 --single-branch https://[our_on_prem_url]/[site_collection]/[project]/_git/[repo]

From what I have read, this is because it is trying to use NTLM authentication when the basic authorization header is not set.

Has anyone got Atlantis to work with the on-premise version of Azure DevOps Server? I have the webhooks and pull request commenting working, so I think this is my last hurdle before I can have Atlantis run Terraform.

I also had the same problem with ArgoCD. I got around that by using their SSH option for connecting to the repo and that has worked great so far.

https://redd.it/qj8omw
@r_devops

Humblebundle the ultimate DevOps bundle (books)

Hello everyone,

What do you think of the DevOps ultimate bundle?

25 books for approx. 15 dollars. Link to the bundle

Does anyone have experience with those books from Packt or can give an opinion/recommendation for a beginner?

Thank you.

https://redd.it/qikfk0
@r_devops

Consul HA structure

I'm trying to learn about service mesh with Consul and I'm trying to understand a minimal setup that needs to be done for a High Availabality Fail-over to work.

If I have two servers that run code, and two servers that host Vault (one is active and the other standby), do I just create a Consul agent on each of the 4 servers with the logic of if active Vault fails -> go to standby? Would this be enough for HA?

Or, do I need additional servers on top of that, such as a Consul server-side that would do all that logic? Like this

Huge thanks ahead!

https://redd.it/qjl2ju
@r_devops

Build with Github actions

Hello comrades, I was playing around for few days with Github workflows and now have a real world use case for that but I'm not sure if it will be possible to achieve. Shortly, I have a multi-stage dockerfile that I want to "translate" to github actions.
For example: I have a scala app in /scala-app-dir that I want to build with sbt, then I want the built folder to be copied into container, then I have an elixir app that I also need to compile and copy the binary to the container. I want to use 1 reusable workflow for building and compiling and one workflow for deployment which will then call the build one. So my question is, will I be able to use the output from build workflow in deploy one, specifically in the dockerfile, where I want to do smth like:


COPY /app-binary-from-build-workflow /app-dir

As the app binary will be created by another workflow (reusable).


I will be more than happy with some starting point at least, or maybe you have such experience.


Many thanks!

https://redd.it/qjkeus
@r_devops

Jenkins over TFS

Wanna convince my team to use Jenkins Enterprise Edition instead of TFS, share some good value points.

https://redd.it/qivlt7
@r_devops

curious about management tools, since i have learned about ansible only and starting terraform next week:

what makes ansible different than other configuration management tools?

https://redd.it/qir4g5
@r_devops

How many times do you find all you need from an enterprise license of an open source software is SSO (SAML, OIDC/OAuth)?

# How many times do you find all you need from an enterprise license of an open source software is SSO (SAML, OIDC/OAuth)?

View Poll

https://redd.it/qims4e
@r_devops

Career Advice for an on the fence Devops Intern

Hello all,

So I just wanted some career advice for a naive intern who has accidentally found himself in a devops role at a medium sized company. I first came in over the summer as C++ Software Engineering Intern and has slowly been moving towards a devops position at the company due to some skill sets I have and a need they have. They have extended my internship till graduation and have a job offer as a Devops Engineer after I graduate.

First some background about the company and what I do. First, I am a junior at a good University studying Computer Science and have good grades (3.8 GPA) and a couple of side projects. This company has had an extremely difficult time finding a good devops guy since they laid off there last one a couple years ago. I came in doing an internship developing out Software, and happened to know how Docker works and can write Dockerfiles so I was put on my first "devops" task creating custom Dockerfiles. I did a good enough job that they extended me and me learn what my senior devs know about devops. I've learned and have been doing the following:

-Jenkins administrative work (configuring Jenkins Slaves, maintaining Pipelines, etc)

-Maintaining our Software automation Testlab (we test on physical hardware). This has me working on some basic IT fixes on some of our machines, or sometimes I'm working directly with or creating custom dev boards with an automated task in mind.

-Maintaining our docker build environments

-Handling build tools on our latest software architecture.
This can mean updating the C++ version across our components (and fixing errors that arise), designing what our final release to the customer is and writing build scripts to make that happen, handling Cmake and compilation of our build systems, etc.

-Creating and maintaining build VMs on our VSphere cluster. We compile our code in our build VMs, and then test them on a variety of different test machines in our automation lab. These build VMs have all sorts if different flavors of Windows and Linux on them that we must guarantee our software runs on and I'm in charge of creating and maintaining them. I've been slowly trying to automate this process.

-Troubleshooting everything automation lab related (VMs, physical machines, build tools, etc)

-Other additional task like setup a VDI, Handling code signing logistics, etc.

-Occasional writing code on our latest software. I take on smaller development tasks that I do if the devops work dies down, which has happened occasionally.


And keep in mind, I am the only one doing these things with occasional help from my senior devs. I am currently making $22s an hour and my work has been very flexible with my college schedule, even letting me work less hours around midterms and finals so I can study.

So I have a couple questions I hope you all might answer for me:

- Is this a good position?

- Am I learning valuable devops skills, or will these skills not be transferable to other companies?

- Am I getting paid enough?

- What sort of pay would be involved if I continue this work?

- My end goal is eventually to be a Software Engineer. Does continuing with this internship peg me as a "devops" guy. Will I have trouble finding Software Engineer Jobs. Should I look for a more Software oriented job (even if it's less pay and doesn't work as well with school)?

- Just any advice for someone in my shoes?


I ask these because I got another job offer doing software development, but it looks to be less pay and less flexible with school, but more relevant to my interests and my end goal.

Any help is appreciated!

https://redd.it/qink6y
@r_devops

Recommendations for Good 2022 Events

Hey Everyone!

Wanted to get some advice from the community - hope this channel is okay to post in. I'm currently a platform engineer and want to move into a more software development focused role and then into app sec in a few years. As part of my career dev plan, my manager approved me to attend 2 conferences this year, so long as 1 is developer/app dev focused and 1 is app sec focused.

Are there any conferences that you'd recommend from your experience?

Thanks!

https://redd.it/qilp0u
@r_devops

Help - VM isn't starting in the GCP once after deployment

Hi there,

I have been learning DevOps for the past few days with GCP's trial plan and tried deploying a NodeJS web app but, once after deploying it's not showing up in the subdomain, which I deployed.

I did the same as shown in this video - DevOps-Crash-Course

Once after deployment, I couldn't run the VM either in my local or ssh in the new tab (an option via GCP)

tho, a few things as said in the video won't work/outdated I tried figuring out a few things and fixed stuff but still nothing shows up :(

the error I'm getting right now,

The initial connection between Cloudflare's network and the origin web server timed out. As a result. the web page can not be displayed.

Contact our hosting provider letting them know your web server not completing requests. An Error 522 means that the request was able to connect to your web server, but that the request didn't finish. The most likely cause is that something on your server is hogging resources. Additional troubleshooting

Any help would be appreciated, thanks in advance!

Ps: if you know any other resources for learning DevOps, drop them in the comments I would be happy to check those out!

https://redd.it/qjx3j5
@r_devops

Is Web Development Knowledge Necessary for Smooth Devops Transition Experience?

Hey guys

How's it going?

So umm been trying to get into devops for months now

So I started learning devops sometime last year but realised that a lot of things didn't click well with me coz I discover that I'd be working with lots of code especially backend code that will be deployed.

So I went back to the drawing board and started learning Node.js with Express because I noticed that most of the devops people work with, other than the yaml and python for scripting are backend code so I decided to learn backend dev which I'm finding really really enjoyable tbh before moving to devops.

So learning the backend dev started making a lot of devops practices click and they started making sense to me.

My question now is, do one need to know backend development or any kind of Web development for that matter to get really smooth with devops since you'll be working with the code before and during deployment ?

Sorry for the long post

I'd appreciate your input on this

https://redd.it/qilhxs
@r_devops

What is the cleanest way to deploy a docker-compose stack to a remote server?

Hello,

We are currently deploying our docker-compose stack using a bunch of ssh commands, to basically copy the docker-compose file to the remote machine, then running docker-compose up -d.


Is there a tool to help or a cleanest way to deploy a docker compose stack to a remote server? To me, our solution feels very much homemade and brittle, and I suspect it is a solved problem.

https://redd.it/qjzx5g
@r_devops

Web UI Framework for Internal Tools?

I need to build some tools that can be run by the NOC via a web interface (think drain a node, pave a node, run traceroute, etc). I've done frontend and backend webdev years and years ago but I'm hoping there's some kind of framework that makes it quick and easy to put up a page with a bunch of nice-looking buttons and dropdowns that trigger ansible playbooks or something of that nature in the backend.

https://redd.it/qk348r
@r_devops

Upskilling new starters undertaking career changes

We tend to hire self taught programmers from science/engineering backgrounds. For upskilling/on-boarding, we pair them with a member of staff and throw them in the deep end. It can be quite difficult for some.

What approach would you take to up-skill intelligent & motivated new starters? What general topics would you cover, what order, any particular format?

https://redd.it/qk1r3q
@r_devops

Android App for Practice Azure Certification Exam !

I sat for 6 months straight and developed this app from Design to Data entries during pandemic finally its live in PlayStore today !

Features

1. Daily Quiz on Cloud
2. Azure Certifications mock exam - more 300 questions
3. Life time access
4. Pricing affordable to everyone - price of buying a burger( $2 ) - ONE TIME

FREE APP

Note : Admin - I am not trying to spam here ! Please let me know if this violates the Channel policy.

https://play.google.com/store/apps/details?id=com.azure\_quiz (edited)

https://redd.it/qk9rso
@r_devops

Keeping Kubernetes Clusters Clean and Tidy

Hi /r/DevOps,

Today I published an article titled "Keeping Kubernetes Clusters Clean and Tidy", where I explain some of the approaches I use to keep Kubernetes clusters free of any resources or object that might be creating clutter or wasting compute resources. For example setting limits and quotas, finding unused resources or monitoring resource consumption.

Here's the link: https://towardsdatascience.com/keeping-kubernetes-clusters-clean-and-tidy-fad52a37f910

Feedback is very much appreciated!

https://redd.it/qkecgs
@r_devops

Monthly 'Shameless Self Promotion' thread - 2021/11

Feel free to post your personal projects here. Just keep it to one project per comment thread.

https://redd.it/qkguhp
@r_devops

Monthly 'Getting into DevOps' thread - 2021/11

What is DevOps?

[AWS has a great article](https://aws.amazon.com/devops/what-is-devops/) that outlines DevOps as a work environment where development and operations teams are no longer "siloed", but instead work together across the entire application lifecycle -- from development and test to deployment to operations -- and automate processes that historically have been manual and slow.

Books to Read

The Phoenix Project - one of the original books to delve into DevOps culture, explained through the story of a fictional company on the brink of failure.
[The DevOps Handbook](https://www.amazon.com/dp/1942788002) - a practical "sequel" to The Phoenix Project.
Google's Site Reliability Engineering - Google engineers explain how they build, deploy, monitor, and maintain their systems.
[The Site Reliability Workbook](https://landing.google.com/sre/workbook/toc/) - The practical companion to the Google's Site Reliability Engineering Book
The Unicorn Project - the "sequel" to The Phoenix Project.
[DevOps for Dummies](https://www.amazon.com/DevOps-Dummies-Computer-Tech-ebook/dp/B07VXMLK3J/) - don't let the name fool you.

What Should I Learn?

Emily Wood's essay - why infrastructure as code is so important into today's world.
[2019 DevOps Roadmap](https://github.com/kamranahmedse/developer-roadmap#devops-roadmap) - one developer's ideas for which skills are needed in the DevOps world. This roadmap is controversial, as it may be too use-case specific, but serves as a good starting point for what tools are currently in use by companies.
This comment by /u/mdaffin - just remember, DevOps is a mindset to solving problems. It's less about the specific tools you know or the certificates you have, as it is the way you approach problem solving.
[This comment by /u/jpswade](https://gist.github.com/jpswade/4135841363e72ece8086146bd7bb5d91) - what is DevOps and associated terminology.
Roadmap.sh - Step by step guide for DevOps or any other Operations Role

Remember: DevOps as a term and as a practice is still in flux, and is more about culture change than it is specific tooling. As such, specific skills and tool-sets are not universal, and recommendations for them should be taken only as suggestions.

Previous Threads
https://www.reddit.com/r/devops/comments/pza4yc/monthlygettingintodevopsthread2021010/

https://www.reddit.com/r/devops/comments/pfwn3g/monthlygettingintodevopsthread202109/

https://www.reddit.com/r/devops/comments/ow45jd/monthlygettingintodevopsthread202108/

https://www.reddit.com/r/devops/comments/obssx3/monthlygettingintodevopsthread202107/

https://www.reddit.com/r/devops/comments/npua0y/monthlygettingintodevopsthread202106/

https://www.reddit.com/r/devops/comments/n2n1jk/monthlygettingintodevopsthread202105/

https://www.reddit.com/r/devops/comments/mhx15t/monthlygettingintodevopsthread202104/

https://www.reddit.com/r/devops/comments/lvet1r/monthlygettingintodevopsthread202103/

https://www.reddit.com/r/devops/comments/la7j8w/monthlygettingintodevopsthread202102/

https://www.reddit.com/r/devops/comments/koijyu/monthlygettingintodevopsthread202101/

https://www.reddit.com/r/devops/comments/k4v7s0/monthlygettingintodevopsthread202012/

https://www.reddit.com/r/devops/comments/jmdce9/monthlygettingintodevopsthread202011/

Please keep this on topic (as a reference for those new to devops).

https://redd.it/qkgv5r
@r_devops

Certifications to start with

My team doesn’t have much work this week and we are allowed to learn new skills that we think might be useful in the future.

I was thinking about starting preparing for a certification for future job search. I’ve 6 months of experience using CICD tools (jenkins, Git etc) and tools such as k8s, docker, helm etc
We don’t use the cloud and I was thinking that I should learn about AWS or azure.
What type of certification can I start with and you would recommend ?

https://redd.it/qkmr4i
@r_devops