CI for open source project

Hi everyone,

​

I've used to setup TeamCity pipeline once, now decided to expand some knowledge about CI/CD and picked thesis topic which involves comparing two CI/CD systems and now my supervisor suggests to pick existing open source project to build pipelines for testing/gathering metrics purposes.

The question is - is that possible at all? If it is then give some tips for small-middle sized projects for those purposes.

​

Thanks in advance :)

https://redd.it/f7srao
@r_devops

Jenkins-github-GCP

Could someone link me to some appropriate resources for how to get Jenkins (running on a GCP debian instance) to test a github repo on pull to master?

I've tried oAuth, webhooks, GCP keys, nopasswd, setting my environmental variables in jenkins, and so on and so forth, but my shell executable script won't run, it fails every time I try to do anything with the repo files. It works with things which don't involve the repo, so things like 'whoami' and so on work just fine, but things like 'cd /application' just bottom out.

https://redd.it/f7s571
@r_devops

Where to build Qt application for macOS without owning Apple HW?

I need to build and try macOS distribution of my **Qt/C++ based** OSS application, but I don't own macOS machine. I need to build it every 2-3 months and it takes me \~1 to 5 days. Therefore I'm looking for a suggestion for a macOS (VM) hosting with a reasonable plan. Ideally one you have experience with and/or you use. CI is not enough as I need to also test whether app works as expected.

I want to make sure that if I rent a machine, I will be able to set up Qt and C++ development environment (or it will be there), I will have sufficient rights & connectivity and it won't be problem to build the application there w/o spending a lot of $.

If you have any other solution, how to solve this problem, please share it. Thank you in advance for your suggestions!

https://redd.it/f7pa7u
@r_devops

GKE, IAP and your own user permissions

I'm building an app which is running on Google Kubernetes Engine. We're using Identity-Aware Proxy as a means of authenticating a user

However, IAP only lets you know if a user has a particular IAM role - IAP-Secured Web User. You then receive a JWT containing their email, but that's all.

So we have a need to resolve this to a user in our database and retrieve a bunch of app level permissions.

Ideally I'd like a service at the front door on an incoming request to, decode the incoming jwt, get the user deets+perms, put it all in a jwt and proxy the request to the responsible service (ie for handling API requests).

But, I can't think of a nice way of doing this - I feel like I want something that augments the ingress. Any ideas? 😐

https://redd.it/f8d6gh
@r_devops

Looking for a Python Library to Process Large Files

Hi,

I'm looking for a Python library to process large files without performance problems. For example I want to transfer large files with hash calculating. Python faces memory problems with such processes. Is there any library that helps? (such pandas in data science)

Any help or documentation would be appreciated.

Thanks

https://redd.it/f88vgw
@r_devops

Versioning, change tracking in microservices environment

We are in the middle of migrating out of our data center and into AWS. We are also completely re-architecting the system to use mostly lambdas and microservices. We used to deploy the software after every sprint and all code was tagged with the release version (like "TheSystem_2.19.1").
The new system is spread across more than 100 individual projects/git repos. Each project is deployed to production individually and potentially multiple times per sprint. We are having trouble tracking what has changed between specific periods of time and knowing what exactly was in production at a specific time.
I'm guessing that we need to write some scripts to pull change logs across all the projects where we past in the start and end timestamps as parameters?
And should we be using a similar script to apply a tag across all the projects too?
And advice is appreciated.

https://redd.it/f800t4
@r_devops

Securing API tokens in scripts

Hey all,

​

I think I'm missing something on using APIs securely, particular in scripts like Lambdas or similar. I understand the basics of making a request and using a token to authenticate to the API endpoint, but how do I secure those when they are on the system?

I understand they can be loaded and referenced as an environmental variable so they are not hardcoded into the script, but what's setting that variable and from where? Is this something Hashicorp Vault or AWS Secrets Manager handles? Do I call the secret store from the script and set that variable in the script?

Here's my use case: I have multiple APIs I want to get data from and use my script to correlate and combine the data in a simple python script. These endpoints might be internal API endpoints, Google G Suite Admin, Okta, or other apps.

On a similar note, as I'm testing my API tokens and calls in something like [Postman](https://www.postman.com/), how do I ensure all my tokens are not synced to their servers? I'm a little paranoid of being "that guy" who syncs all my secrets to Github or similar. I see plenty of really basic "how to call an API" info out there, but less so on how to do it securely.

https://redd.it/f8fdhy
@r_devops

Advice on how to explain your previous DevOps experience to your hiring manager.

When ever this question pops up, I blank really hard.
Can anyone tell me how they’d answer this? ( I’ve Basic DevOps experience of 1 year which includes GIT,Jenkins, Puppet, AWS, Docker, Nagios, Python etc). TIA.

https://redd.it/f81ous
@r_devops

Azure CosmosDB vs MongoDB Atlas. Any experience?

Does anyone have experience with these two technologies?


My company is currently in the midst of a full migration Azure and we are at a crossroads on what would be the best solution for our use case.

​

We currently run on prem mongodb clusters. We were debating between Azure Cosmosdb with mongo API or MongoDB Atlas. They both fit our use-case and it really is a matter of deciding which offers even a slightly marginal benefit over the other.

Some considerations that we are debating

\- We are currently using Mongodb thus it would be nice to continue without re-architecting

\- CosmosDB has a smaller max document size which would require some re-architecting

\- CosmosDB API is wrapper and it would be preferable to use the true Mongo API

\- CosmosDB is the Azure solution for a document DB so it would be a natural choice for going all in on Azure

\- Seems as though if you don't properly account for your throughput with Cosmos that you run the risk of going over RUs and have to take this into account.

\- We are doing ongoing research into differences in sharding

\- Pricing differences (anyone have any experience with either of these techs?)

​

This is relatively early in our research. I am mostly just looking to see if anyone here has any experience with these technologies, or transitioning to them.

https://redd.it/f83wby
@r_devops

Why is Splunk still very popular? Its user interface is super ugly to say the least.



https://redd.it/f8jgo9
@r_devops

($?) Storing ~480GB of data with MongoDB

Hey all!

I'm in the need to store \~480GB of data in a MongoDB database.

If I go the Linode/DO route, it should cost \~$160/mo?
Does that seem like a good price for this amount, and if so, are there any recommendations on scaling back costs by a bit?


Thanks all!

\-P.

https://redd.it/f8kt7x
@r_devops

Can you describe your first two weeks at your new DevOps role? Do you start utilizing all required skills right away? Building CI/CD, Ansible automation, AWS, etc? Or are you given some time to understand your companies new architecture before you get your hand's dirty? How were your first weeks?

I'm suffering from imposter syndrome, even though I know deep inside that I know all the required stuff, and it's giving me terrible anxiety during the interview process.

https://redd.it/f8kprq
@r_devops

Prometheus – Looking for a good guide to learn Prometheus

HI, I have inherited a Prometheus server (with alert manager and Grafana) and for the most part I have a basic/ok understanding on its configuration, however I find Prometheus documentation to be very lacking / hard to understand.

So in terms of my knowledge I know how to add additional scraping jobs, drop metrics from scraps etc.

There are a number of things I just don’t understand why you would do it or only have a basic understanding what it is, such as:
* Why would you perform a relabel on a label?
* What the best practise is for splitting up scrap jobs – ie if all nodes have a node exporter should I have a job just for node exporter or a job for each kind of service and scrap all explorer these servers run (ie node exporter and haproxy exporter).
*alerting
* Remote read and writes
* Scaling / HA – I just assume deploy another instance to scrap everything
* etc. I find the documentation to be very lacking for example the “remote_read” under the configuration document only states “Settings related to the remote read feature”. It took be a bit of digging though to find out the basics of what remote_read are used for.

I also have a book form the humble bundle a while ago and reading thought it I also find it just assumes you know what things are without explain them as well.

So does anyone have a good guide on explaining Prometheus in a more easy to understand manor?

Ps: I know the issue of my understanding of Prometheus docs might just be down to me as my English is not very good

Thanks

https://redd.it/f8hxs2
@r_devops

Looking for feedback on LogDNA

We are currently in the process of adding a log management solution to our production servers (bare metal), and are comparing many options that are available in the space. We have narrowed it down to running our own ELK stack, or using a managed solution - among which LogDNA caught our eyes. I have not been able to find much feedback on that particular tool in here, despite its seemingly wide adoption.

Have you had experience with LogDNA? How does it compare to ELK in terms of ease of setup/effort/benefits? How does it compare to other centralized logging tools?

Thank you.

https://redd.it/f8eos1
@r_devops

What are the best practices to secure the production MongoDB on cloud (Azure/GCP/AWS)?

It'll be of great help because there has been a breach recently in our production Database. Can you guys help me with this? I'm totally new to DevOps and have been handling things in my company smoothly until this happened.

​

Thanks!...

https://redd.it/f8ng1w
@r_devops

Heart beat vs health check.

In checking on how a service is doing, I see to different strategies.

Heart beat: much like sending a trivial message over a long-lasting connection, such as with server sent events, this simply checks if there's an error in delivery. It's a good way to know if a service is down completely, and its frequency can be once a second.

Health check: health check is like a heart beat, except it happens less frequently, and perhaps does some more work to report on how a service is doing.

Do I have the distinction right? Is there a distinction?

https://redd.it/f8et0h
@r_devops

Overcoming "the dip" as a junior devops engineer?

I am a junior devops engineer with one year experience and I am the only one doing devops at my company.
For the most of my time in this role I have been learning, trying new things at a fast rate and implementing interesting technologies to improve processes where I work (ci cd, kubernetes, terraform, azure, automated backups etc ..)
Last couple of months, I have had a drop in confidence in my skills and went from feeling as if I knew everything to feeling like I know nothing and that I am not improving as devops professional (for no particular reason).
My question is: have you had that happen to you early in your carreer and if yes, how did you overcome it?
(This is my first Reddit post, so apologies if this doesn't belong here)

https://redd.it/f8elx2
@r_devops

How to Reformat Multiple Machines Remotely?

I plan to improve the way we set up computers in computer labs in the university. Current process is reformat computers, then install apps needed for the lab, one by one. This takes a lot of time and effort.

I've read about PXE booting and Puppet, but haven't yet found a complete guide on doing the remote install and setup. I hope you can give me guidance on this matter.

Thank you.

Edit:

+ we are using ubuntu

+ computer lab is used for programming classes e.g. Android dev, web dev, C/C++/Java/Python, Data Science etc

https://redd.it/f8t1mp
@r_devops

Would you consider adopting the "GitLab Platform"?

GitLab has a very broad portfolio - all the way from CI / CD to monitoring. To me its very strange to see something like this. We never adopt a platform at our firm - mostly best-of-breed solutions.

I wanted to make sure we are not missing something here and wanted to see if any one here thinks their strategy really makes sense for an enterprise (2k+ employees)? and if there is any tangible benefit of going all in with their platform?

P.S. The closest thing we have to a platform is a bunch of stuff from Hashi - secrets management, Dev environment, Terraform.

https://redd.it/f8uhet
@r_devops

Linux+ or RHCSA

Hello! I'm new to the devops community and devops in general.


I just got my CCNA and I want to begin the devops track. I've been using Linux and the command line for about 16 years but not really in depth.

I bough a RHCSA course on Udemy a while ago but I'm having trouble deciding which certification to pursue.
Which one could serve me most in a devops career? Any tips or recommendations?


I appreciate it.

https://redd.it/f8swcl
@r_devops

The Current State of IT.

Hey everyone. I’ve had a lot of questions bugging me lately that I wish some of you could provide the answers to. As someone with barely any IT experience trying to get an entry level help-desk job to build experience, and trying to set myself up for the next shift in IT (cloud), it seems like there’s a not a set path anymore.

Let me explain, and please correct me if I’m wrong. Right now, from what I’ve researched, the next lucrative thing in IT is the “cloud”. AWS, Azure, GCP is the name of the game. There have been posts I’ve seen on here every now and then, where someone manages to get a “DevOps” or “Cloud Engineering” job skipping help-desk (with no experience).

Q1) How does one get a cloud job, by skipping help-desk, if cloud jobs require a few YOE?

Another thing I want to comment on is why I think it’s kind of confusing for beginners in IT, who want to set themselves up properly for the foreseeable future (cloud). You see all these posts with beginners saying things like: “Is My RoadMap correct?”.

Usually these posts contain getting certs in order like: Security+ → Network+ → CCNA → RHCSA......

They also contain learning the proper skills: learn Python...... learn Docker...... learn A - Z...... you get the gist.

I think where the disconnect comes in, is that a lot of people with experience in this sub, have [The Curse of Knowledge](https://en.m.wikipedia.org/wiki/Curse_of_knowledge), and forget what it’s like to be a beginner, starting out in IT.

Usually the more experienced people will comment like: “yeah, sounds good, make sure to also learn: Python, Go, Linux, Windows, Azure, master Git!!!!!!, oh don’t forget to learn some networking, might want to learn how to create your own protocol as well”.

I wonder, do the people in this sub know everything they tell others to learn, or is it an unconscious form of gatekeeping?

Q2) Having to learn these skills, are entry level positions disappearing (besides help-desk)?

It seems like nowadays companies want employees who are entire IT departments, instead of engineers.

Q3) Is SWE a subset of Cloud Engineering, vice versa, or is Cloud Engineering it’s own entity?

Q4) There are two ways to become a Cloud Engineer / “DevOps” (I understand this is a methodology, but at this point I’m pretty sure you can consider it a job title), either starting off as a SysAdmin or a Developer. Is this correct?

Appreciate everyone taking the time to read this, let me know your insights.

https://redd.it/f8w48p
@r_devops