Our team started using webpack with Rails. Referencing/Sharing asset information from webpack to Rails is the problem being solved in this post. We want to use the webpack assets that are exposed in our Rails app. Not setting up a new plugin for uploading the generated assets would be a plus.

Our team used webpack to compile javascript and related assets. (If you are trying to learn webpack, note that it has a very steep learning curve). Once you get the hang of it, you should be able to integrate/debug any other webpack plugin easily. I followed survivejs book.

After moving our Angular project including CSS, images and fixing compatibility issues. We had assets in Rails which we cannot move because they’d come from other gems. The application uses asset_sync to sync data to CDN. The final step was to use webpack assets in the existing rails app. For this we need to have a link point. The asset-manifest generated by sprockets has become our sync point.

We have a webpack-manifests file, rails manifest file. We need merge the webpack asset list into the Rails one so that we can do two things.

1. use webpack assets in Rails
2. upload webpack assets into the same storage service which is behind the CDN.

Here’s the code to get it done

Details

The rake task merges webpack asset files with rails before the asset:syncstep and after the assets:precompile step. We have enhanced the rake task to achieve this (last few lines). sprockets gem does not currently allow to add any custom files into the manifest. I raised another PR for allowing users customize the manifest file in sprockets.

The .manifest-random-string.json file is a hash. The two keys are files and assets. asset_sync gem uses the assets key which has file => file-fingerprint and uploads it. Uploads work great, but has a problem.

asset_sync chooses to upload only rails fingerprinted files as ‘Cacheable’ in S3 (or other fog-storage adapters). I have made a code change to allow this to be customised. Now you can pass in cache_asset_regexps to match the ones generated by webpack will solve the problem with ‘Cacheable’.

The regular expression usually matches these for webpack and rails

webpack assets -> vendor.[0-9a-f}{8}.js
rails_assets -> application.[0-9a-f]{32}.js 

You can identify if this is working by checking the headers from the CDN for webpack specific files. Browser should not be re-requesting them on every browser reload.

Its a long list of slides about Security with the NoSQL, SQL and other tools used . I thought I’d make a short notes, but its worth reading every slide of the presentation

Devoops Security Notes - How to avoid losing customer data and money

If you are installing 2.1.x or 2.2.x versions of ruby from 2014/2015, support is missing for some of them. gem install stops working with the default gem that installs via rvm when installed on Mac OS X 10.10 (OS X Yosemite)

This is the error I faced. You get the error

ERROR:  Could not find a valid gem 'rubocop' (>= 0), here is why:
          Unable to download data from https://rubygems.org/ - SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (https://api.rubygems.org/latest_specs.4.8.gz)

This worked for me

curl https://curl.haxx.se/ca/cacert.pem > /usr/local/etc/openssl/cert.pem

(found via https://github.com/rubygems/rubygems/issues/1800#issuecomment-270617750)

There are a series of steps here which you may want to try out. http://railsapps.github.io/openssl-certificate-verify-failed.html

These steps did not work for me

There is a suggestion to upgrade to the latest ruby gems by installing it from local after downloading. This results in the following error.

ERROR:  SSL verification error at depth 2: certificate has expired (10)
ERROR:  Certificate /C=BE/O=GlobalSign nv-sa/OU=Root CA/CN=GlobalSign Root CA expired at 2014-01-28T12:00:00Z
ERROR:  SSL verification error at depth 2: certificate has expired (10)
ERROR:  Certificate /C=BE/O=GlobalSign nv-sa/OU=Root CA/CN=GlobalSign Root CA expired at 2014-01-28T12:00:00Z
ERROR:  SSL verification error at depth 2: certificate has expired (10)
ERROR:  Certificate /C=BE/O=GlobalSign nv-sa/OU=Root CA/CN=GlobalSign Root CA expired at 2014-01-28T12:00:00Z
ERROR:  Could not find a valid gem 'rubocop' (>= 0), here is why:
          Unable to download data from https://rubygems.org/ - SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (https://api.rubygems.org/specs.4.8.gz)
ERROR:  SSL verification error at depth 2: certificate has expired (10)
ERROR:  Certificate /C=BE/O=GlobalSign nv-sa/OU=Root CA/CN=GlobalSign Root CA expired at 2014-01-28T12:00:00Z

installing via rvm was suggested too - rvm install 2.1.2 --disable-binary . It did not work.

The yearly or half yearly performance reviews are the things every one looks forwards to, loves and hates. This is the only process where people get emotional and the details are personal. There are companies which only do only self reviews and those who add a 360° feedback. On the other hand there are startups where this process is cranky and screwed up( err., not so well organised). We hate the tools the HRs choose and the way the tools get stuck to on the last hour when everyone is trying to submit/finalize their reviews.

The self review feedback where I have from couple of friends that the feedback is only between you and your manager. This usually is biased since its only important what your manager thinks and your alignment with him along with the goals. Your manager does this for all others in the company. Your manager “may” get feedback from others, but “may” not be included. This is how startups or small companies which care of deadlines usually think. Its good for the employees short term since they only see what their manager wants them to see. (*depending on the manager).

Introducing the 360° feedback, startups transition to this when they usually want to have “changes” in policy and when they feel that 1-1 / self review feedback has had enough biases and see reports people are unhappy or when someone in the chain is unhappy. There are downsides to the 360° feedback as well. This does not work so well for introverts since they may not like to work well with others especially on the communication part which is crucial. If you are a senior developer, you get lots of requests. Middle Managers and Product Managers usually tend to get much more than 40 since they deal with many people.

About the financial aspect of things, is where different companies try to show different tactics. Some have fixed component based on your “rating” across 5 levels and decides how much increment you are eligible for and the bonus part of it also changes based on the currently vesting stock options(from previous years that have been granted) and some stock vesting grant for the future. Some companies tend to payout as a “variable” bonus which the employees look forward a lot to.

(getting side tracked here, open “details” if you want to read more)

In the current generation of corporate sector, the people are in a state where they “feel bad” if someone your experience is getting a higher salary than you or worse if a fresher makes even a penny more than you. Companies are okay with employees moving around (the iteration) and they care more about money given to employees than the amount of money kept to retain them. (This is only about local companies established with a sense of money and not a damn about people like the standard service based companies which employ freshers and fake them as 5yr experience to clients who blindly accept them. Not any two companies are the same. some suck more than others)

The “screwed up” process based companies (where I have experienced in a proclaimed “furniture startup” which does e-commerce) tend to work that every employee is responsible for the revenues the company make and cut your bonus based on that. This is the SOP in Indian based service companies where your actual take home is 70% of your CTC (after taxes). Even though you are not responsible directly or even indirectly for your company’s sales or pace of growth and if you want to do something to increase it, you will have to spend your own time to do it since you are busy with by existing deadlines and last minute feature creeps which did not exist 1 hour ago. I will explain how choosing such a company has impacted me and how you can avoid making the same mistake. I think I can write a PRD about how a company should not be, but I think I will limit it a post.

The good parts about the feedback reviews at companies is that, I really liked the way it was done at Amazon. Amazon has 15 principles. Any business decision you do is usually backed by it. There are some conflicting principles, but they really really help cover many parts of what you tried to do. Amazon Leadership Principles. If you are a developer, some code/features or design changes you have done may not fall into any of the categories. Your code does not have meaning till it solves a business problem or solution.

Every company should have such or similar principles where you can link any business action you have done which gives you a clear idea on what is right and what may be wrong in the decision you have taken. There are points for being wrong or right, which means that there are points for “trying” and being responsible.

Amazon Leadership Principles

I don’t currently work at Amazon anymore, but I love their principles. The amount of clarity it brings to you when you are reflecting on your previous year and the actions you have done is immense. Even if you are very small company or a startup, you should start using your feedback reviews with your company’s principles. I don’t see startups doing this, but it’d make a hell lot of a culture stick to your employees on what the CXX thinks and the company aims for.

Culture is created by what you publicly reward, not what you say

Think of the tools like Makefile for C / C++ or yarn or nom in Javascript or bundle in Ruby or requirements.txt in Python or maven.xml in Java. Now the same applies to the operating system as well, in the operating system, you can add additional stuff outside a package manager which is hard to maintain or guess. You can do the same in the languages for libraries or ad-hoc code which was not intentionally to be present.

Miscellaneous code or libraries is to developers is the same as non-versioned package to Devops

Year 2005

The story starts from the day back in 2005, when I started writing PHP code. The code runs on the same box as its database. There is no firebug, there is no Chrome browser, (yes browser was only introduced after 2008 / 09).

The standard mechanisms for deploying code when developers start out with C code or PHP or HTML, the standard mechanisms are FTP, SCP.

Year 2009

Once you code starts becoming big, you would add version control. With version control, you will need to have a centrally accessible server to access code and authenticate. svn works well with ssh. With version control, the codebase has become distributable which serves 2 things, allow to collaborate and use the same to deploy. The external or implicit dependencies are managed in a Makefile or in a README to get dependencies installed. You start using package managers based on the distro(linux distrubtion).

Year 2011

With the advent of vagrant, the dependencies moved to a environment which can be managed independent of the code. All developers independent of platform can use the virtual environment and just mount their code and run on the Operating system that is being distributed across teams. This was useful when everyone in the team had different versions of everything and had problems compiling openssl or C based libraries like nokogiri or browser based capybara like gems which depend on specific versions of C / C++ libraries with dependencies on other dynamic libraries.

Vagrant like solutions became popular only after 2010. Before Vagrant, there were huge number of man hours wasted before that trying to find solutions in forums, experimenting. Some of these aspects are good because you get to learn, understand and ensure your Makefiles are better and avoid mistakes others have made.

How did other companies solve the problem before this? One of the companies I worked at, had systems in place to compile lots of code. Any package that is acceptable as per the norms of re-distributable, was imported as code, the dependencies were evaluated similarly and each of them had a alternative Makefile which would compile into what ever is required. The advantages of compiling from source and linking dependencies is that you have this huge graph of package dependency. The package manager used to compile any upstream dependencies after it was built to all supported platforms. When your applicationCode failed because of an OpenSSL new package that was imported, you will know that you cannot upgrade to that version, you will know before deploying or building packages. The package management itself is on a separate topic which is an amazing piece of software any developer would appreciate.

The difference between the initial approach and the second one is obviously a build step. Where ever you are planning to deploy the code, you need to build your package into a distributable format like deb or rpm where it can sit into its own files and when the daemon starts, it should start the application code.

Package your code along with its application code dependencies into a single distributable object or a tarball

Year 2012-13

If you are changing code of 5 services, you can test all the code independently, but never in sync on a development box. One of the downsides I have experienced with running several Vagrant setups (because of their dependency of VirtualBox or VMWare) is the hardware limitation. One of the teams I worked had 8 services. I could not run more than 3 at any time since my laptop would start heating up bad. You can solve the software part with distributing the API clients after agreement across the services, but manual testing is not a problem it solves.

Some companies solve this problem by providing multiple machines to the developer , One, where the developer can write code and probably browse along with communication or other business tools can run. Second where code can run without having to keep their laptop running at 8,000 rpm fan speeds.

Don’t compromise on hardware. The cloud is at your disposal.

All of this falls under having automated setup of new environments or a 1-click setup of a new environment in order to test integration scenarios or load testing the services in a company. This heavily depends on the setup of your infrastructure and the automation tools that are present. If you don’t have access to the cloud, as long as you can configure a vagrant or virtual box with your dependencies and distribute port numbers so that other services can connect etc., that will solve your problem. (Its really really hard to do it in a single shot or keep maintaining such configurations)

You need to chose the right tools and ensure your setup scripts work well so that you can make that new deployable environment in a single click(may be two clicks are fine). This is where you start looking at solutions like Chef or Puppet to manage several of these problems. The more generic scripts you make, the faster you can create the environments.

I intentionally skipped types, sizes of hosts and other hardware components.

Enter the new world of 201[5-7]

Docker is a container platform that is written in Go. If you can put your application code in a Docker, all your dependencies are taken care of. If your host OS can run docker, you don’t need to take care of dependencies on your host OS where you run this.

Kubernetes (also written in Go) allows you to automate deployment scaling along with management of containers of app code. There are some hardware redundancy which are not inbuilt into Kubernetes but, its possible to build them using the data from Kubernetes to ensure host machines are spread evenly across racks and data centres to reduce chances of failure.

If you are starting a new service and you need to scale to tens of thousands of requests every second, these are worth a look along with other projects from Hashicorp. Using docker like tools help you deploy and scale up faster. If you are not into maintaining services, you should consider Heroku or other PaaS offerings.

Premature scaling is considered evil. Don’t scale without having the need. When you use the best practices when building or distributing software alongside development, its very easy to scale later. Keep an eye on the database as well, not just the application code.

TL; DR

What ever method you use to distribute code, even if its scp, its better to document it in a README file . When you evolve to the next version or when others want to use your code, it would be easy for them to get started on the important part of coding and spend less time on setting up the environment for the coding. The new tools are getting famous for reducing operational burden like maintenance and management on Devops at the cost of fresh installation of everything they need.

Go is a cross platform language which supports a variety of operating systems and architectures. All new Devops tools are being written in Go. Docker, Kubernetes, Vault, Consul and others by HashiCorp along with many others. Checkout the language if you want to try out a new one.

It takes few years or you need to face the problem before you can appreciate the beauty of the problem and the solution. It does not apply when you are managing one or two hosts, but when there are 10s of them across data centres or multiple availability/redundancy zones.

Disclaimer

all the years I provided are a reference of when I got to know them, not necessarily when these technologies started.

There is lot more to cover in the topic…