A clean way of Deploying Code
Think of the tools like Makefile for C / C++ or yarn or nom in Javascript or bundle in Ruby or requirements.txt in Python or maven.xml in Java. Now the same applies to the operating system as well, in the operating system, you can add additional stuff outside a package manager which is hard to maintain or guess. You can do the same in the languages for libraries or ad-hoc code which was not intentionally to be present.
Miscellaneous code or libraries is to developers is the same as non-versioned package to Devops
Year 2005
The story starts from the day back in 2005, when I started writing PHP code. The code runs on the same box as its database. There is no firebug, there is no Chrome browser, (yes browser was only introduced after 2008 / 09).
The standard mechanisms for deploying code when developers start out with C code or PHP or HTML, the standard mechanisms are FTP, SCP.
Year 2009
Once you code starts becoming big, you would add version control. With version control, you will need to have a centrally accessible server to access code and authenticate. svn works well with ssh. With version control, the codebase has become distributable which serves 2 things, allow to collaborate and use the same to deploy. The external or implicit dependencies are managed in a Makefile or in a README to get dependencies installed. You start using package managers based on the distro(linux distrubtion).
Year 2011
With the advent of vagrant, the dependencies moved to a environment which can be managed independent of the code. All developers independent of platform can use the virtual environment and just mount their code and run on the Operating system that is being distributed across teams. This was useful when everyone in the team had different versions of everything and had problems compiling openssl or C based libraries like nokogiri or browser based capybara like gems which depend on specific versions of C / C++ libraries with dependencies on other dynamic libraries.
Vagrant like solutions became popular only after 2010. Before Vagrant, there were huge number of man hours wasted before that trying to find solutions in forums, experimenting. Some of these aspects are good because you get to learn, understand and ensure your Makefiles are better and avoid mistakes others have made.
How did other companies solve the problem before this? One of the companies I worked at, had systems in place to compile lots of code. Any package that is acceptable as per the norms of re-distributable, was imported as code, the dependencies were evaluated similarly and each of them had a alternative Makefile which would compile into what ever is required. The advantages of compiling from source and linking dependencies is that you have this huge graph of package dependency. The package manager used to compile any upstream dependencies after it was built to all supported platforms. When your applicationCode failed because of an OpenSSL new package that was imported, you will know that you cannot upgrade to that version, you will know before deploying or building packages. The package management itself is on a separate topic which is an amazing piece of software any developer would appreciate.
The difference between the initial approach and the second one is obviously a build step. Where ever you are planning to deploy the code, you need to build your package into a distributable format like deb or rpm where it can sit into its own files and when the daemon starts, it should start the application code.
Package your code along with its application code dependencies into a single distributable object or a tarball
Year 2012-13
If you are changing code of 5 services, you can test all the code independently, but never in sync on a development box. One of the downsides I have experienced with running several Vagrant setups (because of their dependency of VirtualBox or VMWare) is the hardware limitation. One of the teams I worked had 8 services. I could not run more than 3 at any time since my laptop would start heating up bad. You can solve the software part with distributing the API clients after agreement across the services, but manual testing is not a problem it solves.
Some companies solve this problem by providing multiple machines to the developer , One, where the developer can write code and probably browse along with communication or other business tools can run. Second where code can run without having to keep their laptop running at 8,000 rpm fan speeds.
Don’t compromise on hardware. The cloud is at your disposal.
All of this falls under having automated setup of new environments or a 1-click setup of a new environment in order to test integration scenarios or load testing the services in a company. This heavily depends on the setup of your infrastructure and the automation tools that are present. If you don’t have access to the cloud, as long as you can configure a vagrant or virtual box with your dependencies and distribute port numbers so that other services can connect etc., that will solve your problem. (Its really really hard to do it in a single shot or keep maintaining such configurations)
You need to chose the right tools and ensure your setup scripts work well so that you can make that new deployable environment in a single click(may be two clicks are fine). This is where you start looking at solutions like Chef or Puppet to manage several of these problems. The more generic scripts you make, the faster you can create the environments.
I intentionally skipped types, sizes of hosts and other hardware components.
Enter the new world of 201[5-7]
Docker is a container platform that is written in Go. If you can put your application code in a Docker, all your dependencies are taken care of. If your host OS can run docker, you don’t need to take care of dependencies on your host OS where you run this.
Kubernetes (also written in Go) allows you to automate deployment scaling along with management of containers of app code. There are some hardware redundancy which are not inbuilt into Kubernetes but, its possible to build them using the data from Kubernetes to ensure host machines are spread evenly across racks and data centres to reduce chances of failure.
If you are starting a new service and you need to scale to tens of thousands of requests every second, these are worth a look along with other projects from Hashicorp. Using docker like tools help you deploy and scale up faster. If you are not into maintaining services, you should consider Heroku or other PaaS offerings.
Premature scaling is considered evil. Don’t scale without having the need. When you use the best practices when building or distributing software alongside development, its very easy to scale later. Keep an eye on the database as well, not just the application code.
TL; DR
What ever method you use to distribute code, even if its scp, its better to document it in a README file . When you evolve to the next version or when others want to use your code, it would be easy for them to get started on the important part of coding and spend less time on setting up the environment for the coding. The new tools are getting famous for reducing operational burden like maintenance and management on Devops at the cost of fresh installation of everything they need.
Go is a cross platform language which supports a variety of operating systems and architectures. All new Devops tools are being written in Go. Docker, Kubernetes, Vault, Consul and others by HashiCorp along with many others. Checkout the language if you want to try out a new one.
It takes few years or you need to face the problem before you can appreciate the beauty of the problem and the solution. It does not apply when you are managing one or two hosts, but when there are 10s of them across data centres or multiple availability/redundancy zones.
Disclaimer
all the years I provided are a reference of when I got to know them, not necessarily when these technologies started.
There is lot more to cover in the topic…
