Introduction to DevOps This chapter introduces the concept of DevOps and its importance in software development. It explains how DevOps bridges the gap between development and operations teams, allowing for faster delivery of high-quality software.
"Waterfall Model" - Traditional Software Development Methodology "Waterfall Model" is a linear and sequential software development methodology that has distinct goals for each phase of development. This chapter discusses the various stages involved in waterfall model, such as requirements gathering, design, coding, testing, deployment, and monitoring. It also highlights some limitations of this approach.
Agile Methodology - Continuous Iteration Approach The Agile methodology promotes continuous iteration of development and testing throughout the software lifecycle. This chapter compares agile with waterfall model by highlighting its advantages like continuous feedback from testing iterations but also mentions one limitation: lack of continuity in deployment process.
Version Control Tools - Version control tools like Git and Subversion help manage source code, track commits made by developers, identify errors caused by specific commits, and revert back to previous versions. - Continuous integration involves building the application continuously after any developer makes a change. It includes compiling source code, validating it through various tests (unit testing, integration testing), and packaging the application. - Continuous delivery deploys the built application onto test servers for user acceptance testing before deploying it onto production servers.
Source Code Management - Source code management can be done using centralized or distributed version control systems. - Centralized version control uses a central server to store files and enables team collaboration. Developers have working copies of the repository on their local machines but need network connectivity for actions. However, if the central server crashes or gets corrupted, all data is lost. -Distributed version control allows each contributor to have their own local copy of the main repository. Changes can be committed locally without manipulating data on the main repository.
Git: A Distributed Version Control Tool - Git is a popular distributed version control tool that supports non-linear workflows in software development. - With Git's workflow diagram as an example: * Developers make changes in their working directory * They add these changes to staging area with 'git add' * The staged changes are then committed into their local repositories with 'git commit' * Finally they push these commits from local repositories into remote repositories
Creating a Remote GitHub Repository To create a remote GitHub repository, you need to initialize it with a README file and copy the SSH link. Then, use 'git remote add origin' command followed by the copied link to add the remote repository to your local one. Finally, pull any changes from the remote repository using 'git pull origin master'.
Continuous Integration 'Continuous integration' is when developers commit code changes multiple times a day into a shared repository. These commits trigger builds in tools like Jenkins that include compiling code, unit testing, and packaging applications into executable files.
Puppet: Configuration Management and Dynamic Scaling Puppet is a tool that helps in configuration management by continuously checking and confirming the required configurations. It also enables dynamic scaling up and down of machines. Puppet follows a master-slave architecture where slaves pull changes from the central server, allowing for centralized control over all configured machines.
Containerization with Docker "Containers" are lightweight alternatives to virtual machines that contain application binaries and libraries. Docker is a popular containerization tool that allows easy deployment of applications by packaging them along with their dependencies into containers. These containers can be easily replicated across different environments, ensuring consistent computing throughout the software delivery lifecycle.
Continuous Monitoring Tools Continuous monitoring tools detect system errors before they impact business productivity, helping maintain security, availability, troubleshoot performance issues,and plan infrastructure upgrades proactively.They integrate an organization's security tools to aggregate,normailze,and correlate data for analysis based on risk goals.These tools ensure minimal downtime through real-time response to identified risks
Building a Docker Image The application is a Hello World server written with Node.js. The Docker image is based on the official Node.js image, which already includes all dependencies. A Dockerfile is used to specify the contents and parameters of the image.
Using Jenkins for Automation - Jenkins is used to automate tasks such as cloning a repository, building a Docker image, testing it, and pushing it onto Docker Hub. - Credentials are saved in Jenkins for authentication purposes. - The pipeline script in the Jenkins file specifies each stage of automation: clone repository, build image using specified target username/repository name tag (draca1), test by printing 'test passed', push to specified URL on Docker Hub using credentials from Jenkins.
Version Control with Git and GitHub - Version control manages changes made in projects by creating snapshots or versions at different points in time. - Collaboration among developers becomes easier as conflicts can be avoided through shared workspaces provided by version control systems like Git and GitHub. - Storing versions becomes organized without unnecessary redundancy or confusion about what has changed between versions.
Version Control System Backup A version control system provides a backup of all project files by storing changes made in the past. Developers have local copies of the files and can easily retrieve them if needed, even if the central server crashes.
Project Analysis with Version Control Version control systems allow for easy analysis of projects by providing information on what has changed, when it was changed, and how much time it took. This helps identify areas for improvement and evaluate project evolution.
Nonlinear Development of Software Git supports nonlinear development by creating a tree graph from the index, allowing for branching and independent local branches. The master branch contains production quality code, while data storage is compressed to save space. Git's use of C makes it faster than other version control tools.
"Open Source" and Reliability [...] Linus Torvalds created Git as an open source alternative after being dissatisfied with paid options like BitKeeper. It provides reliability through backup copies in both local and central repositories, ensuring that files are not lost even if the central server crashes.
Economical Solution Git is released under the General Public License (GPL), making it free to download without any cost or subscription fees. Additionally, since most operations are performed locally before pushing changes to the central repository on GitHub or another host, expensive hardware requirements can be minimized.
Setting up the Local Repository To create a local repository, right-click on the desired folder and select 'Git Bash Here'. In the Git Bash terminal, use the command 'git init' to initialize an empty git repository. Then, link your local repository with a remote one using 'git remote add origin [repository URL]'. Finally, fetch all files from the central repository into your local machine using 'git pull origin master'.
Committing Changes in Local Repository 'Git add [file name]' adds files to be committed in your index. Use 'git status' to check which files are added or modified. To commit changes in your local repository, run 'git commit -m [commit message]'. Remember to perform a git pull before committing changes and avoid frequent commits.
Creating and Switching Branches To create a new branch, use the command 'git checkout -b [branch name]'. This will create a new branch with all the files from the current branch. To switch to the newly created branch, use 'git checkout [branch name]'. You can now make changes in this new branch without affecting the master branch.
Merging Branches 'Merging' is used to combine work from different branches into one. After creating and working on a separate feature or topic in a specific branch, it's important to merge that work back into the main/master brach when you're done. Use 'git merge [branch name]' while being checked out in your destination (master) brach.
Pushing Changes from Local to Central Repository To push changes from your local repository to the central repository, use 'git push origin branch-name'. This will create a new branch in the central repository with all the files from your local branch. You can verify this by refreshing the repository on GitHub.
Reverting Back to Previous Changes 'Git checkout commit-hash file-name' allows you to revert back to a previous version of a file. By copying and using only the first eight hexadecimal digits of the commit hash, you can go back and view an earlier version of that file.
Continuous Integration: The Nightly Build Approach Nokia adopted a process called nightly build, where developers commit changes to the source code in a shared repository. A build server compiles all the commits made throughout the day at night. However, this approach is not ideal as it requires checking all comments for bugs.
The Solution: Continuous Integration "Nokia adopted continuous integration as a solution", where developers commit changes to the source code and a continuous integration server pulls and compiles these changes immediately. This allows for easy bug detection by identifying which commit caused an error.
Jenkins Build Pipeline The video demonstrates how to create a build pipeline in Jenkins. It shows the process of pulling code from GitHub, performing unit tests, and deploying the application to production. The disadvantages of using one single Jenkins server are also discussed.
Jenkins Distributed Architecture "Jenkins distributed architecture" is introduced as a solution to the limitations of using one single Jenkins server. It consists of a master node for scheduling build jobs and multiple slave nodes for executing those jobs on different environments or handling heavy loads.
Workflow for Application Dependencies The workflow ensures that application dependencies are present throughout the software delivery life cycle, resolving issues where applications work in development but not in production. Docker is used to create a build environment with all dependencies and deploy it across different teams.
Case Study: Indiana University's Adoption of Docker - Before adopting Docker, Indiana University faced problems deploying applications manually and optimizing their environment for legacy Java-based apps. - They wanted to modernize their architecture by moving from monolithic to microservices. - Security was also a concern due to sensitive student data. - The solution was implementing Docker Data Center, which provided control over images, deployment automation, role-based access controls.
Installation Steps The installation process for Kubernetes involves several steps that need to be followed on both the master and slave ends. These steps include updating the repository, disabling swap space, updating hostname and hosts file, setting a static IP address for all nodes in the cluster, installing openssh server and docker components, as well as installing Q barium.
"Master" Configuration "Master" configuration requires at least 2GB of RAM and two core CPUs. The host name should reflect "k Master", with a static IP address set up using specific commands. Additionally, openssh server is installed along with Docker to support containerization in Kubernetes.
Kubernetes Environment Setup To establish the Kubernetes environment on both master and slave machines: curl is installed; packages are downloaded from a specified URL; an update command is run; then cubelet (kubelet), qbarium (kube-apiserver) ,and cubecdl(kubectl) are installed via apt-get install command
Setting up the Hosts File To set up the hosts file, add the IP address of the master and specify its name as 'kemaster'. Then, update the hosts file on both master and slave VMs.
Bringing Up The Cluster - Execute commands to initiate Kubernetes cluster - Install a pod network (Calico in this case) - Set up Kubernetes dashboard before nodes join
Benefits of Containers Containers package applications and isolate them from the host, making them fast, reliable, efficient, lightweight, and scalable. However, scaling up containers manually can be challenging.
Challenges with Scaling Up Containers - Managing a large number of containers becomes difficult. - Auto-scaling functionality is not available in regular containers. - Distributing traffic among multiple containers requires load balancing.
Kubernetes vs. Docker: Market Dominance "Communities" is currently prevailing in the market due to its early entry and strong community support, but it relies on Docker containers for functionality.
What Kubernetes Is - Kubernetes provides a robust and reliable cluster that ensures container availability. - It excels at scaling up containers compared to other platforms. - It offers a user-friendly GUI dashboard for application deployment.
Comparing Kubernetes with Docker Swarm - Installation and configuration are easier with Docker Swarm, but once set up, Kubernetes offers more flexibility and reliability. - Kubernets has better load balancing capabilities than Dockerswarm - Scaling may be faster in Dockerswarm initially, but overall strength of the cluster is stronger in kubernetes
Use Case: Pokemon Go Powered by Kuberentes 'Pokemon Go' used kubernetes to handle unexpected high traffic loads during its launch. The game's backend architecture included Google Bigtable as the main database and Java Cloud programs running on mapreduce & cloud dataflow.
How Pokemon Go Used Communities Pokemon Go used communities to handle three times the traffic they expected, which was a surprising success. The game broke stereotypes by encouraging physical activity and became incredibly popular.
Architecture of Kubernetes "Kubernetes architecture is simple: there is a Cube Master that controls everything, while containers run on nodes. Pods are logical groupings of containers running an application. Replication controllers ensure the desired number of pods are always running, and services provide load balancing."
Creating the Dashboard To create a Kubernetes dashboard with admin functionalities, a service account is created in the default namespace. The generated token is used to access the dashboard, which provides an overview of workloads, deployments, pods, and services.
Deploying Applications through the Dashboard - Using the Kubernetes dashboard's click functionality - Specify application name and image from Docker Hub - Set number of ports for containers - Enable external access to service on specified port number
Deployment via Yaman File To deploy using a Yaman file, you need to write your YAML or JSON code. In the code, specify the API version and kind of deployment. You can also define replicas and containers with their respective images and ports. Once done, save the file and upload it to Kubernetes using 'kubectl create -f [filename]'.
Introduction to Ansible 'Ansible' is a configuration management tool that helps solve problems related to managing multiple servers efficiently. It provides features like orchestration, provisioning, configuration management, application deployment, and security automation.
Benefits of Configuration Management Configuration management solves various problems such as managing multiple servers by organizing them based on tasks; scaling up/down infrastructure easily; improving work velocity for developers by automating server configurations.
Configuration Management Before configuration management, system administrators had to manually configure servers for testing, causing delays and hampering developer productivity. With configuration management tools, administrators can easily configure all servers at once, improving agility and collaboration between developers and administrators.
Rollback Problem "Rolling back" to a previous version of an application was time-consuming before configuration management. With the documentation provided by configuration management systems, rolling back becomes seamless with zero downtime for users.
Ansible Architecture Ansible is a simple and flexible automation tool that can be used to manage various machines. It does not always require SSH for connection, as other network authentication protocols like Kerberos can also be used. Ansible playbooks are written in YAML code, which serves as a data serialization language. Playbooks start with three dashes and consist of host machine lists, variables gathering facts, tasks executed in order, and handlers triggered by specific events.
Orchestration with Ansible "Orchestration" refers to the process of defining how different configurations on multiple systems interact with each other to maintain a seamless flow of applications. With ansible playbooks, you can specify the tasks that need to be carried out before others in order for your application deployments to work properly. This ensures proper installation orders (e.g., web server first before deploying HTML/PHP files) and coordination between front-end/back-end services.
Provisioning & Configuration Management In terms of provisioning using ansible playbook means ensuring all necessary software required for an application is installed across different environments within your infrastructure such as development/testing/production servers. Configuration management involves describing your infrastructure's configuration using human-readable yet machine-parseable data descriptions called ansible configurations or playbooks. With these configurations/playbook files stored in source control repositories (like Git), you can easily apply changes/configurations without needing any agent/client software installations on target hosts.
Provisioning Playbook for Lamp Stack In this chapter, we learn how to write a provisioning playbook to install the necessary software (Apache, PHP, MySQL) on our node machine. The playbook uses become and become user to escalate privileges and gather facts about the remote host. Each task in the playbook installs a specific package using Ansible's package module.
Creating Database and Tables "My SQL Modding" is an orchestration playbook that installs required software (MySQL Python module) on the note machine using pip. It then creates a new MySQL user with all privileges for all databases and creates a database named "edureka". Since there is no built-in module for creating tables in Ansible, it uses Command Module with custom MySQL queries to create tables inside the database.
Introduction to Configuration Management Configuration management is a process that helps manage changes in infrastructure. It solves problems like manual deployment, updates, and inconsistencies in computing environments.
"Infrastructure as Code" Concept "Infrastructure as code" means writing code for infrastructure. With this approach, you can easily configure large infrastructures and roll back to previous versions if needed.
Puppet: A Configuration Management Tool Puppet is a configuration management tool used for deploying applications, configuring nodes, managing services, and more. It uses master-slave architecture with SSL encryption for secure communication between the Puppet Master and agents.
Deploying MySQL and PHP using Puppet In this chapter, we learn how to deploy MySQL and PHP using Puppet. We download the predefined modules for MySQL and PHP from the Puppet Forge. These modules define classes that we need to declare in our manifest file. After declaring the classes, we run 'puppet agent -t' command on our puppet agents to pull the configurations from the puppet master.
Introduction to Configuration Management This chapter provides an overview of configuration management and its importance, using the New York Stock Exchange as a use case. It also introduces infrastructure as code and different approaches to configuration management.
The Need for Continuous Monitoring "Continuous monitoring" is explained in this chapter, highlighting three key reasons why it is important: preventing failures in CI/CD pipelines, identifying performance issues before release, and ensuring server health. The phases of continuous monitoring are defined along with popular tools like Amazon CloudWatch and Nagios.
Understanding Nagios for Continuous Monitoring This chapter focuses on Nagios as a tool for continuous monitoring. It explains how Nagios works with its server-agent architecture where plugins interact with local or remote hosts/services. The features of Nagios are highlighted including scalability, security, easy plugin writing capability,and support for redundant monitoring posts.
Continuous Monitoring Continuous monitoring is essential for industries to ensure the smooth functioning of their systems. It differs from traditional monitoring systems and involves various phases. Nagios is a popular continuous monitoring tool that allows remote server monitoring.
"NRP" - Nagios Remote Plugin Executor "NRP" enables Nagios to monitor local resources on remote Linux or Unix machines by installing an agent on those machines. This agent communicates with the local monitoring machine through a secure SSL connection, allowing it to check disk space, load, HTTP services, etc.
Case Study: Bitnetics Bitnetics faced challenges in remotely monitoring an online store located 500 miles away from their staff's location due to firewall restrictions and false positive notifications. They overcame these issues by adding SSH firewall rules, converting notifications into Nagios alerts with escalation logic, configuring event handlers for service restarts before notification, and using multiple servers at NOC facility.