Airflow with k8s executor - issues with PVC

I am using a PVC to mount dags. The configs don't have examples and I am running to a very strange issue.

## name of dags pvc
dags_volume_claim = dags_pvc

## I don't know what below is supposed to be
## if PVC is mounted at /opt/app and airflow is in /opt/app/airflow
## and dags should be /opt/app/airflow/dags
## then does this sound right?
dags_volume_subpath = airflow/dags

Please correct me if the above is wrong.

Now the 2nd issue is that on this PVC I have all my dags, parsers and other scripts.
There is a python script on this PVC under airflow directory that is supposed to access a file inside hidden directory on root of PVC.

So for us root of PVC is /opt/app
The file is in /opt/app/.hidden/file1

When my webserver pod is coming up, it complains it cannot access

/opt/app/.hidden/file1


I have it chmod to 777 for testing, but still not having any luck.

This is a python script if that helps.

Any ideas?

Thanks!

https://redd.it/f7w4be
@r_devops

Advice on how to prepare for DevOps position

Hello guys,

For the last 4 years I've been working in something different from what I prepared for. I have a masters in web engineering but I've mostly been working with Puppet and Kubernetes. Now, I'm thinking on moving to another country and my intention is to find something as DevOps.

The thing is that I don't feel confident enough on doing the interviews or even joining any company (impostor syndrome?). What would you recommend to prepare/study/play with for preparing to this new phase in my life?

I appreciate any comment and book or course recommendation.

https://redd.it/f7xrll
@r_devops

Does AWS throttle bandwidth?

​

Question? Has anyone had a similar experience?

I spun up an ec2 instance to do some data processing on tens of thousands of S3 objects.

It was taking 10 days to do the download from S3.

I switched to a metal instance and ran multiple simultaneous downloads and the elapsed download time dropped to 3 minutes. Wow.

After a few sprints over a few days, the download time suddenly increased dramatically.

Using some standard tools, the new bandwidth clocks at less than 1 Gbps on what should be a 25 Gbps pipe and accessing only AWS resources (S3).

I have verified my units ( ie. not comparing MBps to Mbps )

Has anyone else experienced this?

https://redd.it/f7wecl
@r_devops

How do you keep track of how many deploys you have made over a period of time?

I've been reading some books and State of DevOps reports and saw some cool statsitics about how many times deploys are made every year. My company has a build per every commit, and if it isn't tagged, then it fails to fit a criteria to release. We generally release 20 pieces once a month, an improvement from 4 pieces quarterly a year ago. I would like to implement some metrics to help show how our efforts are improving the deployment process, but I don't know how to efficiently to keep track of releases as we deploy faster.

https://redd.it/f7xwpl
@r_devops

SEIM tools that are NOT SaaS

So, I work in a very security-oriented corner of the market, and as such cannot use aaS tools for our customers' data.

We've been researching better monitoring and log aggregation tools, but are finding the majority of offerings that provide modern automation and "AIOps" are only offered in an AsAService model. Those companies that offer in-house installation options seem to only provide watered-down editions of the tools, missing most of the efficiency-bringing features.

Does anyone have strong opinions or recommendations of tools that can be installed or managed in a private network?

https://redd.it/f81d16
@r_devops

Golang testing frameworks

Hi everyone!

I'm looking for recommendations for a golang test framework, more specifically, something that I can use to test a Postgres server.

I have written some tests using InSpec and the postgres_session resource which work great, but I find Ruby a little slow to run in CI.

So I thought why not teach myself some Go, in the form of a test framework. In my brief searching, I found https://github.com/smartystreets/goconvey which looks nice. I imagine integrating with a Go Postgres module shouldn't be too difficult.

What's your favourite Golang test framework?

https://redd.it/f7r0q7
@r_devops

CI for open source project

Hi everyone,

​

I've used to setup TeamCity pipeline once, now decided to expand some knowledge about CI/CD and picked thesis topic which involves comparing two CI/CD systems and now my supervisor suggests to pick existing open source project to build pipelines for testing/gathering metrics purposes.

The question is - is that possible at all? If it is then give some tips for small-middle sized projects for those purposes.

​

Thanks in advance :)

https://redd.it/f7srao
@r_devops

Jenkins-github-GCP

Could someone link me to some appropriate resources for how to get Jenkins (running on a GCP debian instance) to test a github repo on pull to master?

I've tried oAuth, webhooks, GCP keys, nopasswd, setting my environmental variables in jenkins, and so on and so forth, but my shell executable script won't run, it fails every time I try to do anything with the repo files. It works with things which don't involve the repo, so things like 'whoami' and so on work just fine, but things like 'cd /application' just bottom out.

https://redd.it/f7s571
@r_devops

Where to build Qt application for macOS without owning Apple HW?

I need to build and try macOS distribution of my **Qt/C++ based** OSS application, but I don't own macOS machine. I need to build it every 2-3 months and it takes me \~1 to 5 days. Therefore I'm looking for a suggestion for a macOS (VM) hosting with a reasonable plan. Ideally one you have experience with and/or you use. CI is not enough as I need to also test whether app works as expected.

I want to make sure that if I rent a machine, I will be able to set up Qt and C++ development environment (or it will be there), I will have sufficient rights & connectivity and it won't be problem to build the application there w/o spending a lot of $.

If you have any other solution, how to solve this problem, please share it. Thank you in advance for your suggestions!

https://redd.it/f7pa7u
@r_devops

GKE, IAP and your own user permissions

I'm building an app which is running on Google Kubernetes Engine. We're using Identity-Aware Proxy as a means of authenticating a user

However, IAP only lets you know if a user has a particular IAM role - IAP-Secured Web User. You then receive a JWT containing their email, but that's all.

So we have a need to resolve this to a user in our database and retrieve a bunch of app level permissions.

Ideally I'd like a service at the front door on an incoming request to, decode the incoming jwt, get the user deets+perms, put it all in a jwt and proxy the request to the responsible service (ie for handling API requests).

But, I can't think of a nice way of doing this - I feel like I want something that augments the ingress. Any ideas? 😐

https://redd.it/f8d6gh
@r_devops

Looking for a Python Library to Process Large Files

Hi,

I'm looking for a Python library to process large files without performance problems. For example I want to transfer large files with hash calculating. Python faces memory problems with such processes. Is there any library that helps? (such pandas in data science)

Any help or documentation would be appreciated.

Thanks

https://redd.it/f88vgw
@r_devops

Versioning, change tracking in microservices environment

We are in the middle of migrating out of our data center and into AWS. We are also completely re-architecting the system to use mostly lambdas and microservices. We used to deploy the software after every sprint and all code was tagged with the release version (like "TheSystem_2.19.1").
The new system is spread across more than 100 individual projects/git repos. Each project is deployed to production individually and potentially multiple times per sprint. We are having trouble tracking what has changed between specific periods of time and knowing what exactly was in production at a specific time.
I'm guessing that we need to write some scripts to pull change logs across all the projects where we past in the start and end timestamps as parameters?
And should we be using a similar script to apply a tag across all the projects too?
And advice is appreciated.

https://redd.it/f800t4
@r_devops

Securing API tokens in scripts

Hey all,

​

I think I'm missing something on using APIs securely, particular in scripts like Lambdas or similar. I understand the basics of making a request and using a token to authenticate to the API endpoint, but how do I secure those when they are on the system?

I understand they can be loaded and referenced as an environmental variable so they are not hardcoded into the script, but what's setting that variable and from where? Is this something Hashicorp Vault or AWS Secrets Manager handles? Do I call the secret store from the script and set that variable in the script?

Here's my use case: I have multiple APIs I want to get data from and use my script to correlate and combine the data in a simple python script. These endpoints might be internal API endpoints, Google G Suite Admin, Okta, or other apps.

On a similar note, as I'm testing my API tokens and calls in something like [Postman](https://www.postman.com/), how do I ensure all my tokens are not synced to their servers? I'm a little paranoid of being "that guy" who syncs all my secrets to Github or similar. I see plenty of really basic "how to call an API" info out there, but less so on how to do it securely.

https://redd.it/f8fdhy
@r_devops

Advice on how to explain your previous DevOps experience to your hiring manager.

When ever this question pops up, I blank really hard.
Can anyone tell me how they’d answer this? ( I’ve Basic DevOps experience of 1 year which includes GIT,Jenkins, Puppet, AWS, Docker, Nagios, Python etc). TIA.

https://redd.it/f81ous
@r_devops

Azure CosmosDB vs MongoDB Atlas. Any experience?

Does anyone have experience with these two technologies?


My company is currently in the midst of a full migration Azure and we are at a crossroads on what would be the best solution for our use case.

​

We currently run on prem mongodb clusters. We were debating between Azure Cosmosdb with mongo API or MongoDB Atlas. They both fit our use-case and it really is a matter of deciding which offers even a slightly marginal benefit over the other.

Some considerations that we are debating

\- We are currently using Mongodb thus it would be nice to continue without re-architecting

\- CosmosDB has a smaller max document size which would require some re-architecting

\- CosmosDB API is wrapper and it would be preferable to use the true Mongo API

\- CosmosDB is the Azure solution for a document DB so it would be a natural choice for going all in on Azure

\- Seems as though if you don't properly account for your throughput with Cosmos that you run the risk of going over RUs and have to take this into account.

\- We are doing ongoing research into differences in sharding

\- Pricing differences (anyone have any experience with either of these techs?)

​

This is relatively early in our research. I am mostly just looking to see if anyone here has any experience with these technologies, or transitioning to them.

https://redd.it/f83wby
@r_devops

Why is Splunk still very popular? Its user interface is super ugly to say the least.



https://redd.it/f8jgo9
@r_devops

($?) Storing ~480GB of data with MongoDB

Hey all!

I'm in the need to store \~480GB of data in a MongoDB database.

If I go the Linode/DO route, it should cost \~$160/mo?
Does that seem like a good price for this amount, and if so, are there any recommendations on scaling back costs by a bit?


Thanks all!

\-P.

https://redd.it/f8kt7x
@r_devops

Can you describe your first two weeks at your new DevOps role? Do you start utilizing all required skills right away? Building CI/CD, Ansible automation, AWS, etc? Or are you given some time to understand your companies new architecture before you get your hand's dirty? How were your first weeks?

I'm suffering from imposter syndrome, even though I know deep inside that I know all the required stuff, and it's giving me terrible anxiety during the interview process.

https://redd.it/f8kprq
@r_devops

Prometheus – Looking for a good guide to learn Prometheus

HI, I have inherited a Prometheus server (with alert manager and Grafana) and for the most part I have a basic/ok understanding on its configuration, however I find Prometheus documentation to be very lacking / hard to understand.

So in terms of my knowledge I know how to add additional scraping jobs, drop metrics from scraps etc.

There are a number of things I just don’t understand why you would do it or only have a basic understanding what it is, such as:
* Why would you perform a relabel on a label?
* What the best practise is for splitting up scrap jobs – ie if all nodes have a node exporter should I have a job just for node exporter or a job for each kind of service and scrap all explorer these servers run (ie node exporter and haproxy exporter).
*alerting
* Remote read and writes
* Scaling / HA – I just assume deploy another instance to scrap everything
* etc. I find the documentation to be very lacking for example the “remote_read” under the configuration document only states “Settings related to the remote read feature”. It took be a bit of digging though to find out the basics of what remote_read are used for.

I also have a book form the humble bundle a while ago and reading thought it I also find it just assumes you know what things are without explain them as well.

So does anyone have a good guide on explaining Prometheus in a more easy to understand manor?

Ps: I know the issue of my understanding of Prometheus docs might just be down to me as my English is not very good

Thanks

https://redd.it/f8hxs2
@r_devops

Looking for feedback on LogDNA

We are currently in the process of adding a log management solution to our production servers (bare metal), and are comparing many options that are available in the space. We have narrowed it down to running our own ELK stack, or using a managed solution - among which LogDNA caught our eyes. I have not been able to find much feedback on that particular tool in here, despite its seemingly wide adoption.

Have you had experience with LogDNA? How does it compare to ELK in terms of ease of setup/effort/benefits? How does it compare to other centralized logging tools?

Thank you.

https://redd.it/f8eos1
@r_devops

What are the best practices to secure the production MongoDB on cloud (Azure/GCP/AWS)?

It'll be of great help because there has been a breach recently in our production Database. Can you guys help me with this? I'm totally new to DevOps and have been handling things in my company smoothly until this happened.

​

Thanks!...

https://redd.it/f8ng1w
@r_devops