Thursday, June 5, 2008

Deployment System, Batch Deployment and Monitoring Tool and Access Control System

1.0 Deployment System
The Deployment Service is a comprehensive tool that allows the operations team to deploy any application (or application components) on a set of selected servers in a server farm. The deployment is a scheduled activity and can be scheduled to be done immediately or at some future pre-programmed time. This tool also takes care of deployment on offline servers by scheduling or re-attempting the deployment on such servers for a later time. Essentially it ensures that all production servers are in sync with each other with respect to the product versions.


Fig 1. Deployment System Architecture

The Scheduler Interface provides the mechanism for operations engineers to schedule the upload of new or updated applications and configuration files to the live servers. The source files for deployment will be from the baselined versions in the CVS repository. The Deployment Engine ensures that all target servers selected are updated with the same version of the files being uploaded, including those that are not available on the network at the time of the scheduling. The Deployment Engine makes use of standard UNIX utilities for packaging and transferring the files to the selected servers. The scheduler interface also provides a comprehensive report on the status of all deployments including those that were scheduled in the past. The database is used for storage of deployment reports, besides other pieces of information.

Server components:
• Comprises of the Deployment Server Software (RDSS).

Client components:
• Comprises of the Package Management Applications (PMA)

Deployment functions
These functions are implemented in terms of scripts implementing individual functions.
• Automated software repository access.
• Packaging of components to be deployed into self extraction scripts or RPMs.
• Transfer of the packaged scripts / RPMs onto selected destination machines.
• Actual deployment of the new packages from the self extracting scripts or RPMs onto the destination servers.
• Scheduler – Main component which handles the scheduling for deployment, synchronization between the other components, keep track of offline servers, ensures all servers are in sync with each other.

Package Management Application (PMA)
The Package Management Application is a small footprint PERL script that executes on each of the selected destination server machines. This application performs the deployment steps instructed to it via a self extracting shell script.

2.0 Batch Deployment and Monitoring Tool
The Batch Deployment and Monitoring Tool (BDMT) is an intelligent, centralized batch (Cron job) deployment and monitoring tool. This tool controls and maintains the status of scheduled jobs running on all production servers spread across multiple server farms. This tool monitors the execution of these scheduled jobs and reports the corresponding status.

Multiple scheduled jobs need to deployed and executed on the production servers for regular maintenance kind of activities. These activities are critical to ensure the prolonged uptime of these servers, which is in turn critical from a business perspective. There are some standard jobs to be run on all the servers, besides specific jobs to be run for specific applications/servers etc.

The necessary intelligence is built into this framework. Prior to initiation of a batch job, checks are made to see if an earlier instance of the same job is running or is in some error state. Besides this availability of vital system resources are checked prior to execution. All error conditions are logged for subsequent action by the system administrator.

The BDMT has the capability of adding, deleting, modifying and controlling the execution of jobs on any production server via a user friendly interface.
BDMT generates comprehensive reports on job execution status with all the necessary statistics.


Fig 2. The BDMT Architecture

BDMT Components
The BDMT comprises of multiple components, each handling specific tasks. The components are as follows:

BDMT User Interface
The BDMT UI provides the following functionality
• Add /Modify /Control /Delete cron jobs
• Construction and display of reports.

Scheduler Module
This is an intelligent wrapper script which launches all required cron jobs which need to be monitored. Apart from job processing, it ensures that necessary checks are done prior to initiation, besides maintaining logs and smoothly handling all envisaged error conditions.

Cron Updater Module
This is a script which is present on each individual server to handle updates to the cron queue. This script is triggered remotely to add, modify or delete cron jobs.

Synchronization Module for report data
This module ensures transfer of status of individual jobs from the production servers back to the centralized server. This module operates on a scheduled basis at regular intervals in a day which is configurable; alternatively it can be triggered via the interface to retrieve the current set of reports immediately.

Report Builder Module
The status of execution of individual jobs on production servers is transferred back to the centralized server at a fixed location. Comprehensive reports are constructed from this data with the aid of this module.

3.0 Access Control System
The Access Control System is a gateway for access to all the production servers. Access to any of the servers is possible only through the Access Control System. This is a single point of entry which has an audit trail to log all user activity.

The system records all requests for access and approval. A temporary access id is provided using which the requesting user can logon to the permitted production servers. Apart from this, permanent ids also exist which are controlled and approved by the necessary authorities.

The system provides time based access to the users. The access is withdrawn once the time for which the access was authorized is over.

It has a facility for role based access. This restricts the command set that the user can use on the production servers. The system can also restrict the user to a specific set of directories on any given server.

The system has comprehensive reporting facilities and can provide a complete audit trail of all user requests for access, approvals and the access activity itself.