Automate web log compression using Analog logfile analyser tool

Write a script which compresses a series of web log files and confirms that basic data has in fact been correctly preserved. **Background:** We use the [Analog][1] logfile analyser tool to parse our web logs, and its "[cache file][2]" mechanism to compress ancient web logs. As noted in that documentation, this approach archives some, but not all, of the web log data. We need a script which automates the creation of these cache files, and then tests to confirm that the basic data which is supposed to be preserved has in fact been done so correctly. The intent is that we will (manually) gzip the cache files and remove the original files once this script confirms that all data has been correctly preserved. This script should *not* gzip or remove anything itself!! All web logs were created by Apache using standard methods. **Caution:** All syntax used below is from software written six years ago. Some things may (or may not) have changed. Please read the above documentation carefully to ensure that you understand current syntax. ## Deliverables The script **[login to view URL]** (**m**ake **an**alog **ca**che files) will do the following: Initialize counts of successful and failed compressions. If a set of one or more filenames is passed in the script's command line, loop through all of those files whose filename actually begins with "access." and perform the following steps: Otherwise, loop through all files in the current directory whose filename begins with "access." and perform the following steps: In this discussion, "$tail" indicates the portion of the filename which follows the initial component, "access.". Write to the standard output, "Processing access.$tail\n". Note that some, but not necessarily all, of the access.* files are compressed using gzip. Analog can parse these directly, and does *not* require you to gunzip them! e.g., A typical set of files might include: # ls access.03022{0,1}* [login to view URL] [login to view URL] [login to view URL] [login to view URL] [login to view URL] [login to view URL] [login to view URL] [login to view URL] [login to view URL] [login to view URL] [login to view URL] [login to view URL] [login to view URL] [login to view URL] [login to view URL] [login to view URL] [login to view URL] [login to view URL] If anca.$tail already exists, write, "anca.$tail already exists\n". Do not overwrite it, but do continue processing after the creation step below. Otherwise, create anca.$tail using a command similar to: analog +C"LOGFILE access.$tail" +C"OUTPUT CACHE" +C"CACHEOUTFILE anca.$tail" and write, "Created anca.$tail\n". The following steps currently use a configuration file named [login to view URL] with the following content: OUTPUT ASCII ALL OFF GENERAL OFF MONTHLY ON MONTHCOLS RPB MONTHGRAPH B If you can do the following *just* using command line parameters, without using a configuration file at all, that'd be far easier, as it'd keep the script "self-contained". Generate two reports from analog, one from each file, saving the output to temporary files, using commands similar to: analog +g/web/logs/[login to view URL] +C"LOGFILE access.$tail" >>/tmp/manca.before.$$ analog +g/web/logs/[login to view URL] +C"LOGFILE NONE" +C"CACHEFILE anca.$tail" >>/tmp/manca.after.$$ This will yield reports which contain lines similar to this: month: #reqs: pages: kbytes: --------: -----: -----: ------: Oct 2005: 14453: 1265: 29981: ++++++++++ Nov 2005: 41690: 3600: 92838: +++++++++++++++++++++++++++++++ If all of the numbers (i.e., three numbers per month) match for the two reports, write, "Compression confirmed.\n". Otherwise, write, "Compression FAILED!\n". Increment the appropriate count. After the loop is complete, write, "### files processed: ### confirmed, ### failed.\n". | ![][3] | PGP is **not** required for this project. | | ![][4] | An NDA is **not** required for this project. | We will retain full legal rights to the code and/or data upon completion. All code and/or data is and becomes *our* intellectual property! You will *not* have any rights to the code and/or data in any fashion. **Thank you for your consideration and for bidding through RentACoder!** * * *Copyright ? 2005, RentACoder Buyer [notecom][5]. **All Rights Reserved.** Reproduction without permission is strictly prohibited. * * * 1) Complete and fully-functional working program(s) in executable form as well as complete source code of all work done. 2) Deliverables must be in ready-to-run condition, as follows (depending on the nature of the deliverables): a) For web sites or other server-side deliverables intended to only ever exist in one place in the Buyer's environment--Deliverables must be installed by the Seller in ready-to-run condition in the Buyer's environment. b) For all others including desktop software or software the buyer intends to distribute: A software installation package that will install the software in ready-to-run condition on the platform(s) specified in this bid request. 3) All deliverables will be considered "work made for hire" under U.S. Copyright law. Buyer will receive exclusive and complete copyrights to all work purchased. (No GPL, GNU, 3rd party components, etc. unless all copyright ramifications are explained AND AGREED TO by the buyer on the site per the coder's Seller Legal Agreement). ## Platform Linux (required) and Solaris (optional)