devopsdays nyc 2020 tickets are now on sale (earliest bird pricing!)

Check out our March 3-4, 2020 event in NYC!

[https://www.eventbrite.com/e/devopsdays-nyc-2020-tickets-83067223301?ref=estw](https://www.eventbrite.com/e/devopsdays-nyc-2020-tickets-83067223301?ref=estw)

We have tickets priced at our lowest, $150

https://redd.it/e57g7q
@r_devops

Unique Christmas ideas for my team?

I manage a team of DevOpsy folks who've done really well this year and would like to get them all a nice holiday gift. Thinking something personalized (perhaps laser engraved?) and tech-related. Any ideas? Looking to spend about $50-100 each.

One thought was a nice USB power pack with our team/company logo, their name, and their start date engraved on it. Maybe that's a bit cliche, but looking for other ideas that fall into that type of realm.

https://redd.it/e58ign
@r_devops

Netdata release v1.19

Hey all,

Introducing a major rewrite of our web log collector, [cmocka](https://www.facebook.com/hashtag/cmocka?source=feed_text&epa=HASHTAG&__xts__%5B0%5D=68.ARDKJpuwODv-hPc236C7fM2r9vCYx9vTJeH93a18odECvej08YhKL8yikqtkH5YC0qIPbakDgFmmf15tnogjIj3cf_Wya85PQYEzfw9Y2OUmmX9y_0GfH_ikrxDZS109LzcjMTWo8zoUXSy893FBfR0_AxyBw6IFCcIkBObQiTOtCIjrxws3vt14_Jg5s_H61VJwB_KoiG9hPRXpuA6vV5Vm9rUSZUwM0KpucD6_6XrlRI2xwd7Sz5M_3Arnkqr4DckHC5NXNGGTVMUYJgz2H3mq1KcdwguQvlsigL2Pz4k6QW1ZCCYCvG21xk1UzcDAEEr15Yqxx-AEGa1AlNoxRqzZc8MM&__tn__=%2ANK-R) unit testing, improvements on our [Unbound](https://www.facebook.com/hashtag/unbound?source=feed_text&epa=HASHTAG&__xts__%5B0%5D=68.ARDKJpuwODv-hPc236C7fM2r9vCYx9vTJeH93a18odECvej08YhKL8yikqtkH5YC0qIPbakDgFmmf15tnogjIj3cf_Wya85PQYEzfw9Y2OUmmX9y_0GfH_ikrxDZS109LzcjMTWo8zoUXSy893FBfR0_AxyBw6IFCcIkBObQiTOtCIjrxws3vt14_Jg5s_H61VJwB_KoiG9hPRXpuA6vV5Vm9rUSZUwM0KpucD6_6XrlRI2xwd7Sz5M_3Arnkqr4DckHC5NXNGGTVMUYJgz2H3mq1KcdwguQvlsigL2Pz4k6QW1ZCCYCvG21xk1UzcDAEEr15Yqxx-AEGa1AlNoxRqzZc8MM&__tn__=%2ANK-R) collector and even more! Check out our blog post for full details or read our release notes below.

Release v1.19.0 contains 2 new collectors, 19 bug fixes, 17 improvements, and 19 documentation updates. Full release notes can be found [here](https://github.com/netdata/netdata/releases/tag/v1.19.0).

**At a glance**

We completed a major rewrite of our web log collector to dramatically improve its flexibility and performance. The [new collector](https://github.com/netdata/go.d.plugin/pull/141), written entirely in Go, can parse and chart logs from Nginx and Apache servers, and combines numerous improvements. Netdata now supports the LTSV log format, creates charts for TLS and cipher usage, and is amazingly fast. In a test using SSD storage, the collector parsed the logs for 200,000 requests in about 200ms, using 30% of a single core.

This Go-based collector also has powerful custom log parsing capabilities, which means we're one step closer to a generic application log parser for Netdata. We're continuing to work on this parser to support more application log formatting in the future.

We have a new tutorial on [enabling the Go web log collector](https://docs.netdata.cloud/docs/tutorials/collect-apache-nginx-web-logs/) and using it with Nginx and/or Apache access logs with minimal configuration. Thanks to [Wing924](https://github.com/Wing924) for starting the Go rewrite!

We introduced more **cmocka unit testing** to Netdata. In this release, we're testing how Netdata's internal web server processes HTTP requests—the first step to improve the quality of code throughout, reduce bugs, and make refactoring easier. We wanted to validate the web server's behavior but needed to build a layer of parametric testing on top of the CMocka test runner. Read all about our process of testing and selecting cmocka on our blog post: [Building an agile team's 'safety harness' with cmocka and FOSS](https://blog.netdata.cloud/agile-team-cmocka-foss/).

Netdata's **Unbound collector** was also [completely rewritten in Go](https://github.com/netdata/go.d.plugin/pull/287) to improve how it collects and displays metrics. This new version can get dozens of metrics, including details on queries, cache, uptime, and even show per-thread metrics. See our [tutorial](https://docs.netdata.cloud/docs/tutorials/collect-unbound-metrics/) on enabling the new collector via Netdata's amazing auto-detection feature.

We [fixed an error](https://github.com/netdata/netdata/pull/7220) where **invalid spikes** appeared on certain charts by improving the incremental counter reset/wraparound detection algorithm.

Netdata can now send [**health alarm notifications to IRC channels**](https://docs.netdata.cloud/health/notifications/irc/) thanks to [Strykar](https://github.com/Strykar)!

And, Netdata can now monitor [**AM2320 sensors**](https://docs.netdata.cloud/collectors/python.d.plugin/am2320/), thanks to hard work from [Tom Buck](https://github.com/tommybuck).

**Improvements**

* **New Collectors**
* AM2320 sensor collector plugin

* Added parsing of /proc/pagetypeinfo to provide metrics on fragmentation of free memory pages
* The unbound collector module was completely rewritten, in Go [go.d.plugin/#287](https://github.com/netdata/go.d.plugin/pull/287)

**Collector improvements**

* **We rewrote our web log parser in Go, drastically improving its flexibility and performance**
* The [Kubernetes kubelet collector](https://docs.netdata.cloud/collectors/go.d.plugin/modules/k8s_kubelet/) now reads the service account token and uses it for authorization. We also added a new default job to collect metrics from https://localhost:10250/metrics
* Added a new default job to the [Kubernetes coredns](https://docs.netdata.cloud/collectors/go.d.plugin/modules/coredns/) collector to collect metrics from [https://kube-dns.kube-system.svc.cluster.local:9153/metrics](https://kube-dns.kube-system.svc.cluster.local:9153/metrics)
* apps.plugin: Synced FRRouting daemons configuration with the frr 7.2 release
* apps.plugin: Added process group for git-related processes-apps.plugin: Added balena to the container-engines application group
* web\_log: Treat 401 Unauthorized requests as successful
* xenstat.plugin: Prepare for xen 4.13 by checking for check xenstat\_vbd\_errorpresence
* mysql: Added galera cluster\_statusalarm
* **Metrics Database**
* Netdata generates alarms if the disk cannot keep up with data collection
* **Health**
* Fine tune various default alarm configurations
* Update SYN cookie alarm to be less aggressive
* Added support for IRC alarm notifications
* **Installation/Packages**
* Corrected the Makefile.am files indentation, to prevent unexpected errors
* Rationalized ownership and permissions of /etc/netdata
* Made various improvements to the installer script netdata-installer.sh
* Include go.d.plugin version v0.11.0
* **Other**
* Improve Travis build warnings
* cmocka testing for http requests
* CI/CD: Prevented nightly jobs from timing out

https://redd.it/e54dql
@r_devops

Best way to automatically update many containers when new image version is available in my repository?

So as the title says, i am wondering what would be the best way to store my docker images and update my running containers on, lets say, a lot of different machines. I was thinking about using gitlab and watchtower. Is there a better way?

https://redd.it/e56wtr
@r_devops

How Do You Get Out of Your Rut?

You've been there - just going through the motions at a job you don’t love anymore.

You’re feeling stuck - but you don’t know why.

You probably think a new job will fix it.

And it might, for a while.

But when that doesn’t last, what will you do?

Getting to the underlying reason that you’re stuck will help.

**How do you do that?**

https://redd.it/e5c2ec
@r_devops

How can I logically link two branches in two separated git repos for the creation of a temporary frontend app that should query its backend?

Our devs wants to have a temporary web app running whenever they push a feature branch to the repo. I got this part going already, I'm using Heroku for the infra and GitLab for CI/CD.

On the API repo they do the same thing, they push a branch and they get a temporary version of said push.

The problem is that I don't know how to logically link both of these automatically. The temporary frontend should talk to the temporary backend.

I know that I have to set and environment variable to point the front to the back, but I'm stuck in the part of how to link both of these independent branches together programmatically in the pipeline, if that makes sense.

I thought about using the same git tags on both branches but didn't get around it yet. I also thought about using Terraform in the pipeline to spin both apps and then make it somehow query its state to link both up.

Any help?

https://redd.it/e55t9z
@r_devops

Tech Professionals - Challenges with Fitness

I know that many Tech Professionals are struggling with losing weight, getting in shape and building healthy eating habits. For all those that are struggling with getting in good shape, what are some of your biggest challenges in this area of your life ?

What keeps you from reaching your fitness goals & What bothers you the most about ?

https://redd.it/e4t7pc
@r_devops

How to upgrade Ruby Bundle version in AWS Opsworks

I have a project running on AWS Opsworks, which has current bundler version of ***1.5.3*** on ***Ruby 2.2.2***.

i want to upgrade the bundle version to ***1.17.3***

but when i try to do it i am getting this error

Compiled Resource:
------------------
# Declared in /var/lib/aws/opsworks/cache.stage2/cookbooks/opsworks_bundler/recipes/default.rb:2:in `from_file'

gem_package("Installing Bundler 1.17.3") do
provider Chef::Provider::Package::Rubygems
action [:install]
retries 0
retry_delay 2
package_name "bundler"
version "1.17.3"
cookbook_name "opsworks_bundler"
recipe_name "default"
gem_binary "/usr/local/bin/gem"
end



[2019-12-03T07:24:08+00:00] INFO: Running queued delayed notifications before re-raising exception
[2019-12-03T07:24:08+00:00] ERROR: Running exception handlers
[2019-12-03T07:24:08+00:00] ERROR: Exception handlers complete
[2019-12-03T07:24:08+00:00] FATAL: Stacktrace dumped to /var/lib/aws/opsworks/cache.stage2/chef-stacktrace.out
[2019-12-03T07:24:08+00:00] ERROR: gem_package[Installing Bundler 1.17.3] (opsworks_bundler::default line 2) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of /usr/local/bin/gem install bundler -q --no-rdoc --no-ri -v "1.17.3" ----
STDOUT:
STDERR: ERROR: Error installing bundler:
"bundle" from bundler conflicts with /usr/local/bin/bundle
---- End output of /usr/local/bin/gem install bundler -q --no-rdoc --no-ri -v "1.17.3" ----
Ran /usr/local/bin/gem install bundler -q --no-rdoc --no-ri -v "1.17.3" returned 1
[2019-12-03T07:24:08+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)


[Opswork configuration](https://i.stack.imgur.com/eUaxe.png) the bundler version was previously 1.5.3 i changed it to 1.17.3 and thats when the errors started to come

when i run bundle -von the server i am getting bundler version as **1.5.3.** what ever i do i cannot change the bundle version from that.

how to add configuration in either the chef recipe or anywhere else to update the bundle version ?

any suggestion will help me a lot. i have no experience working with chef or opsworks i have good working experience in working with EC2s and linux bash.

Our devops team doesn't know how to work with either. i am trying to solve an issue on a client's application which was launched on 2014 and hasn't ever been touched.

https://redd.it/e5g3jf
@r_devops

Apache Archiva in Front of Maven

I thought this would be no-brainer, but it's a little more difficult than I was led to believe by the [quick start documentation](https://archiva.apache.org/docs/2.1.1/quick-start.html).

I added the URL to the local Archiva server as a [mirror](https://archiva.apache.org/docs/2.1.1/userguide/using-repository.html) in settings.xml, and I know it has an effect because the maven build now breaks. The first thing was that the missing version stamps on the helper plugins went from a warning to an error:

[INFO] Scanning for projects...
[WARNING]
[WARNING] Some problems were encountered while building the effective model for com.github.jitpack:maven-simple:jar:0.2-SNAPSHOT
[WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-source-plugin is missing. @ line 35, column 15
[WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-javadoc-plugin is missing. @ line 49, column 15
[WARNING]
[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
[WARNING]
[INFO]
[INFO] ------------------< com.github.jitpack:maven-simple >-------------------
[INFO] Building Simple Maven example 0.2-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.512 s
[INFO] Finished at: 2019-12-01T20:57:26-06:00
[INFO] ------------------------------------------------------------------------
[ERROR] Error resolving version for plugin 'org.apache.maven.plugins:maven-source-plugin' from the repositories [local (/home/tad/.m2/repository), archiva.default (https://localhost:8080/archiva/repository/internal/)]: Plugin not found in any plugin repository -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] https://cwiki.apache.org/confluence/display/MAVEN/PluginVersionResolutionException


But I fixed that by adding current version numbers to each plugin mentioned in the pom.xml.

Now it is complaining about missing pom.xml files for each of those plugins:

[INFO] Scanning for projects...
[INFO]
[INFO] ------------------< com.github.jitpack:maven-simple >-------------------
[INFO] Building Simple Maven example 0.2-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[WARNING] The POM for org.apache.maven.plugins:maven-source-plugin:jar:3.2.0 is missing, no dependency information available
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.127 s
[INFO] Finished at: 2019-12-01T20:59:09-06:00
[INFO] ------------------------------------------------------------------------
[ERROR] Plugin org.apache.maven.plugins:maven-source-plugin:3.2.0 or one of its dependencies could not be resolved: Failure to find org.apache.maven.plugins:maven-source-plugin:jar:3.2.0 in https://localhost:8080/archiva/repository/internal/ was cached in the local repository, resolution will not be reattempted until the update interval of archiva.default has elapsed or updates are forced -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the erro

rs and possible solutions, please read the following articles:
[ERROR] [Help 1] https://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException

I'm guessing that there's more to do than just starting Archiva and pointing to it, but searching for "maven archiva howto" or for "maven archiva missing pom.xml" doesn't find me very much that seems relevant.

What I'm wanting to do is to use Archiva as a local mirror for dependencies when working down a narrow-bandwidth pipe, and also to add a few dependencies we have that aren't available across the internet. I thought that this should be the sort of thing that Archiva does out of the box, but the guide is 5 years old, so may be there's some other bit of configuration that is needed that I'm missing...?

Thanks for any thoughts!

https://redd.it/e4sm9w
@r_devops

Question about local Docker options

I am needing some advice about how to handle supplying Docker instances locally for our developers.

As a company we are just now getting into "devops" but we have several projects running on our current k8s platform. We have several hundred devs and only a select few are using docker and k8s currently but it looks like the majority of them are going to want to use the platform eventually. Specifically my question is:

What are good options for providing local docker instances for our devs to work on? We are a windows shop mostly so all desktops are running win10. We have explored Docker for Windows but we've seen multiple problems with it conflicting with vmware Horizon. The other option is to create linux vm's with docker installed there but considering the amount of devs we have, this isn't ideal either because of the new work that creating 300-400 new vms would bring, including supporting a whole new environment like that. Is there a better option? What are other mature companies doing?

Thanks for the help!

https://redd.it/e5jsgz
@r_devops

Is testing considered important in devops?

In my org, our devops team doesn't seem to test their changes to our infrastructure as it relates to the software we support and it tends to be an afterthought. Typically things seem to get thrown over the wall to QA and our engineering team. Is this commonplace at most companies?

https://redd.it/e5g1m3
@r_devops

Terraform: Confusion on provider inside module referencing.
https://www.reddit.com/r/Terraform/comments/e5m7ci/confusion_on_provider_inside_module_referencing/

https://redd.it/e5m96r
@r_devops

EKS on Fargate! World's first fully managed EKS option?

I am so excited about this, many of us knew this will happen, but it is finally here. Of course it has limitations, but still.. anyone tried it yet? Opinions?

https://aws.amazon.com/blogs/aws/amazon-eks-on-aws-fargate-now-generally-available/

https://redd.it/e5n6dt
@r_devops

How much do you guys earn and what location?

I recall the stackoverflow survey said devops and sre were some of the highest paid. Anyone willing to share some data?

https://redd.it/e5ks7p
@r_devops

Going Docker, Swarm and Kubernetes Production Like a Pro

This is a 45 minute talk from GOTO Berlin 2019 by Bret Fisher, Docker captain, DevOps trainer and consultant. I've dropped the link to the talk along with the full talk abstract below for a read before diving into the talk:

[https://youtu.be/5zY5\_iTGIsU?list=PLEx5khR4g7PKMVeAqZdIHRdOwTM1yktD8](https://youtu.be/5zY5_iTGIsU?list=PLEx5khR4g7PKMVeAqZdIHRdOwTM1yktD8)

Learn fast from my years of being a container consultant and Docker implementer. Come join me for a jam-packed session of decisions you need to make and key technical factors you should know. No fluff, all practicals. Updated for 2019 and based on my 3 years of top-10 DockerCon talks.

**You should show up if:**

* You are planning or involved with building/using a Docker production system.
* You are thinking of using Swarm and/or Kubernetes (but not required).
* You like random 80's/90's video game trivia thrown at you.

DevOps in the Real World is far from perfect, yet we all dream of that amazing auto-healing fully-automated micro-service infrastructure that we'll have "someday." But until then, how can you really start using containers today, and what decisions do you need to make to get there?

This session is designed for practitioners who are looking for ways to get started now with Docker and container orchestration in production. This is not a Docker 101, but rather it's to help you be successful on your way to Containerizing your production systems. Attendees will get tactics, example configs, real working infrastructure designs, and see the (sometimes messy) internals of Docker/Kubernetes in production today.

**Session Topics**

* Preventing scope creep in your project planning (what you can throw out)
* Easy Dockerfile anti-patterns
* OS and Kernel choice
* Container image choice
* Swarm and Kubernetes architecture designs
* Infrastructure layers to maybe outsource
* Infrastructure "stacks", layer the solution from low-level OS choices to high-level cluster GUI's

https://redd.it/e5kylz
@r_devops

Is this a career I should pursue?

I am 42 years old. I was historically an Electrical Engineer and firmware developer. I moved from the big city into a smaller city for a job. That company closed down. I was laid off. I got another job doing industrial controls. That company also downsized.

I took a CS50 course and got a software QA job. That company had a 25% layoff, myself included.

After 3 layoffs in 4 years, I am trying to figure out what the hell I am supposed to do with my life. At my last job, the devOps guy appeared to be on top of things and was happy with his job. This made me wonder if I should look into that.

I need some honest input. Is devOps something that a 42 year old guy who's only experience is python, .net, C, C++ would be able to break into....or is my resume going to continue getting sideways looks from employers?

\-Is it something that can be done from an Entry level position?

\-Is it something that you are generally happy with, or do you not enjoy it?

\-Is it something that can be done remotely?

\-I'm getting burned out of constantly learning new things. Is there a high barrier to entry for it? (Am I going to be going to school for another year?)

&#x200B;

Any input would be greatly appreciated. My career no longer exists where I live, but I still need to support my family. I'm in a pretty tough spot right now.

https://redd.it/e5ot9f
@r_devops

Assistance Securing EKS Cluster

Is anyone using EKS in production managed to secure the cluster?
My current environment uses AWS transit gateways and sub accounts and as a result, it’s difficult to setup the DNS resolver for a private API endpoint.

Would anyone have experience with this, or would the IAM permissions be sufficient to keep the cluster secure?

Thanks!

https://redd.it/e5r96u
@r_devops

Bitbucket pull request notifications for triggering builds

For those of you who use Bitbucket, how do you handle pull request notifications?

We were using a plugin, but after a Bitbucket upgrade, it all stopped working. Looking for a more permanent solution.

The plugin offered tons of parameters such as the email of the author of the pull request and the commit hash of the branch, but Bitbucket's hooks don't include that functionality.

https://redd.it/e5qz23
@r_devops

Dev image customizations?

With dev, build, test, and prod running the same image, I was wondering how devs using those images handle customization. Like dotfiles and stuff.

What do you guys do?

https://redd.it/e5q2nn
@r_devops