TSM Policy vs Tape Library data

Monday, July 3, 2017

Crc32 Implementation in C#


Hi Folks,
Recently I was given task to port Java application to C# and I was stuck in below line of Code in Java


byte bytes[] = "HElloworld@123".getBytes();
Checksum checksum = new CRC32();
checksum.update(bytes,0,bytes.length);
long lngChecksum = checksum.getValue();
System.out.println( String.valueOf(lngChecksum).trim());



I'm unable to find a simple Code in C# on stackOverflow and other blog to achieve the same
like :

            Crc32 crc32;
            crc32 = new Crc32();
            String data = "HElloworld@123";
            crc32.AddData(System.Text.Encoding.ASCII.GetBytes(data));
            Console.WriteLine(crc32.Crc32Value.ToString());


I found following links to be useful to achieve the same and hope it can help you too

http://www.dotnetframework.org/default.aspx/DotNET/DotNET/8@0/untmp/WIN_WINDOWS/lh_tools_devdiv_wpf/Windows/wcp/Print/Reach/Serialization/Crc32@cs/1/Crc32@cs


How to achieve it:

1) Create a console Application in VS: ProjectName/NameSpace: Crc32Simple


2) add Crc32.cs as provided in above link
3) in Program.cs it should look like this:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Runtime.InteropServices;
namespace Crc32Simple
{
     class Program
    {
        static void Main(string[] args)
        {
            Crc32 crc32;

            crc32 = new Crc32();
            String data = "HElloworld@123";
            crc32.AddData(System.Text.Encoding.ASCII.GetBytes(data));
            Console.WriteLine("Crc32 Checksum for :" + data.ToString() + " is :" + crc32.Crc32Value.ToString());

            crc32 = new Crc32();
            data = "HElloworldu8h8uh87$^#^WGerdtrd$#T$%TTHDdfd576766565565456#ZerWTRET";
            crc32.AddData(System.Text.Encoding.ASCII.GetBytes(data));
            Console.WriteLine("Crc32 Checksum for :" + data.ToString() + " is :" + crc32.Crc32Value.ToString());
            
            crc32 = new Crc32();
            data = "HElloworld@123";
            crc32.AddData(System.Text.Encoding.ASCII.GetBytes(data));
            Console.WriteLine("Crc32 Checksum for :" + data.ToString() + " is :" + crc32.Crc32Value.ToString());

            Console.ReadLine();

        }        
    }
}



if above link for Crc32.cs is broken then the source code for Crc32 is:


using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Security.Cryptography;
using System.Security;
using System.Security.Permissions;

namespace Crc32Simple
{
    internal sealed class Crc32
    {
        #region Constructor

        ///  
        /// Constructs an instance of a Crc32 class. 
        ///
        public
        Crc32()
        {
            _crc32Value = Crc32StartValue;
        }

        #endregion Constructor

        #region Public properties

        ///
        /// The Crc32 value of the bytes calculated
        /// thus far.
        ///
         public UInt32 Crc32Value
        {
            get
            {
                return (~(_crc32Value));
            }
        }

        #endregion Public properties

        #region Public methods

        ///
        /// Adds the specified bytes of data to the 
        /// already calculated crc32 value.  This
        /// method can be called as many times as needed
        /// and the Crc32 will continually keep updating.
        ///
         ///
        /// A byte array of data. 
        ///  
        public void AddData(byte[] data)
        {
            _crc32Value = UpdateCRC32(data, _crc32Value);
        }

        #endregion Public methods

        #region Private static helper methods

        ///
        /// Updates the Crc32 value by adding the 
        /// supplied data bytes into the ongoing
        /// calculation. 
        ///
         ///
        /// The data bytes used to update value. 
        /// 
        ///
        /// The Crc32 value prior to this data
        /// being added. 
        /// 
        ///  
        /// A new 32-bit unsigned integer representing 
        /// the new Crc32 value.
        ///
         ///
        /// Critical    - This takes in raw bytes of a bit map
        /// TreatAsSafe - This does not keep the data it just uses it for a calculation.
        ///
         [SecurityCritical, SecuritySafeCritical]
        private
        static UInt32 UpdateCRC32(byte[] data, UInt32 oldCrc)
        {
            int bytes = data.Length;

            if (bytes == 0)
                return oldCrc;

            UInt32 newCrc = 0;

            for (int n = 0; n < bytes; n += 1)
            {
                UInt32 idx = (byte)(oldCrc ^ (UInt32)data[n]);
                newCrc = Crc32Table[idx] ^ ((oldCrc >> 8) & 0x00FFFFFF);
                oldCrc = newCrc;
            }

            return newCrc;
        }

        #endregion Private static helper methods

        #region Private data

        private UInt32 _crc32Value;

        #endregion Private data

        #region Private static data

        ///
        /// The starting Crc32 value. 
        ///
         static UInt32 Crc32StartValue = 0xFFFFFFFF;

        ///
        /// The Crc32 precalculated data table. 
        ///
        static UInt32[] Crc32Table = {
            0x00000000, 0x77073096, 0xee0e612c, 0x990951ba, 0x076dc419, 0x706af48f, 0xe963a535, 0x9e6495a3, 
            0x0edb8832, 0x79dcb8a4, 0xe0d5e91e, 0x97d2d988, 0x09b64c2b, 0x7eb17cbd, 0xe7b82d07, 0x90bf1d91,
            0x1db71064, 0x6ab020f2, 0xf3b97148, 0x84be41de, 0x1adad47d, 0x6ddde4eb, 0xf4d4b551, 0x83d385c7,
            0x136c9856, 0x646ba8c0, 0xfd62f97a, 0x8a65c9ec, 0x14015c4f, 0x63066cd9, 0xfa0f3d63, 0x8d080df5,
            0x3b6e20c8, 0x4c69105e, 0xd56041e4, 0xa2677172, 0x3c03e4d1, 0x4b04d447, 0xd20d85fd, 0xa50ab56b, 
            0x35b5a8fa, 0x42b2986c, 0xdbbbc9d6, 0xacbcf940, 0x32d86ce3, 0x45df5c75, 0xdcd60dcf, 0xabd13d59,
            0x26d930ac, 0x51de003a, 0xc8d75180, 0xbfd06116, 0x21b4f4b5, 0x56b3c423, 0xcfba9599, 0xb8bda50f, 
            0x2802b89e, 0x5f058808, 0xc60cd9b2, 0xb10be924, 0x2f6f7c87, 0x58684c11, 0xc1611dab, 0xb6662d3d, 
            0x76dc4190, 0x01db7106, 0x98d220bc, 0xefd5102a, 0x71b18589, 0x06b6b51f, 0x9fbfe4a5, 0xe8b8d433,
            0x7807c9a2, 0x0f00f934, 0x9609a88e, 0xe10e9818, 0x7f6a0dbb, 0x086d3d2d, 0x91646c97, 0xe6635c01, 
            0x6b6b51f4, 0x1c6c6162, 0x856530d8, 0xf262004e, 0x6c0695ed, 0x1b01a57b, 0x8208f4c1, 0xf50fc457,
            0x65b0d9c6, 0x12b7e950, 0x8bbeb8ea, 0xfcb9887c, 0x62dd1ddf, 0x15da2d49, 0x8cd37cf3, 0xfbd44c65,
            0x4db26158, 0x3ab551ce, 0xa3bc0074, 0xd4bb30e2, 0x4adfa541, 0x3dd895d7, 0xa4d1c46d, 0xd3d6f4fb,
            0x4369e96a, 0x346ed9fc, 0xad678846, 0xda60b8d0, 0x44042d73, 0x33031de5, 0xaa0a4c5f, 0xdd0d7cc9, 
            0x5005713c, 0x270241aa, 0xbe0b1010, 0xc90c2086, 0x5768b525, 0x206f85b3, 0xb966d409, 0xce61e49f,
            0x5edef90e, 0x29d9c998, 0xb0d09822, 0xc7d7a8b4, 0x59b33d17, 0x2eb40d81, 0xb7bd5c3b, 0xc0ba6cad, 
            0xedb88320, 0x9abfb3b6, 0x03b6e20c, 0x74b1d29a, 0xead54739, 0x9dd277af, 0x04db2615, 0x73dc1683, 
            0xe3630b12, 0x94643b84, 0x0d6d6a3e, 0x7a6a5aa8, 0xe40ecf0b, 0x9309ff9d, 0x0a00ae27, 0x7d079eb1,
            0xf00f9344, 0x8708a3d2, 0x1e01f268, 0x6906c2fe, 0xf762575d, 0x806567cb, 0x196c3671, 0x6e6b06e7, 
            0xfed41b76, 0x89d32be0, 0x10da7a5a, 0x67dd4acc, 0xf9b9df6f, 0x8ebeeff9, 0x17b7be43, 0x60b08ed5,
            0xd6d6a3e8, 0xa1d1937e, 0x38d8c2c4, 0x4fdff252, 0xd1bb67f1, 0xa6bc5767, 0x3fb506dd, 0x48b2364b,
            0xd80d2bda, 0xaf0a1b4c, 0x36034af6, 0x41047a60, 0xdf60efc3, 0xa867df55, 0x316e8eef, 0x4669be79,
            0xcb61b38c, 0xbc66831a, 0x256fd2a0, 0x5268e236, 0xcc0c7795, 0xbb0b4703, 0x220216b9, 0x5505262f, 
            0xc5ba3bbe, 0xb2bd0b28, 0x2bb45a92, 0x5cb36a04, 0xc2d7ffa7, 0xb5d0cf31, 0x2cd99e8b, 0x5bdeae1d,
            0x9b64c2b0, 0xec63f226, 0x756aa39c, 0x026d930a, 0x9c0906a9, 0xeb0e363f, 0x72076785, 0x05005713, 
            0x95bf4a82, 0xe2b87a14, 0x7bb12bae, 0x0cb61b38, 0x92d28e9b, 0xe5d5be0d, 0x7cdcefb7, 0x0bdbdf21, 
            0x86d3d2d4, 0xf1d4e242, 0x68ddb3f8, 0x1fda836e, 0x81be16cd, 0xf6b9265b, 0x6fb077e1, 0x18b74777,
            0x88085ae6, 0xff0f6a70, 0x66063bca, 0x11010b5c, 0x8f659eff, 0xf862ae69, 0x616bffd3, 0x166ccf45, 
            0xa00ae278, 0xd70dd2ee, 0x4e048354, 0x3903b3c2, 0xa7672661, 0xd06016f7, 0x4969474d, 0x3e6e77db,
            0xaed16a4a, 0xd9d65adc, 0x40df0b66, 0x37d83bf0, 0xa9bcae53, 0xdebb9ec5, 0x47b2cf7f, 0x30b5ffe9,
            0xbdbdf21c, 0xcabac28a, 0x53b39330, 0x24b4a3a6, 0xbad03605, 0xcdd70693, 0x54de5729, 0x23d967bf,
            0xb3667a2e, 0xc4614ab8, 0x5d681b02, 0x2a6f2b94, 0xb40bbe37, 0xc30c8ea1, 0x5a05df1b, 0x2d02ef8d 
        };

        #endregion Private static data
    }
}




Saturday, August 2, 2014

Web Utility to Capture TSM Policy Vs. Data Backed up

Hi Everyone,
I'm Saurabh Sharma and this is my first blog on IBM Tivoli TSM.

You can view my work at : www.tsmtools.co.nf  for TSM Free tutorials and Awesome Backup Simulator & Backup Calculator for Corporate Environment.

 Objective of writing this article is to share my work with the TSM community and to develop free open source tools/scripts for better TSM Administration and Management. I'm a TSM Administrator for last 2 years and one of my crucial role is to define Data Management policy for Backup Landscape.
Suppose Backup architecture comprises of TSM server, Various clients(Win,AIX,Linux,HP-UX), LTO5 Tape Library and different backup profiles includes
  • File Level Backup
  • TSM for Mail Backup
  • TSM for MSSQL/Oracle Backup
  • AIX/Linux - file & DB2 Backup

This is an Utility which I've developed to understand the impact of Copy Group(TSM policy) parameters and amount of Data backed up. The algorithm that runs behind the scene take input as Initial Size of Filesystem/Database and it's growth rate(added,modified,deleted files) and TSM Copy Group parametrs

  • Version Data Exist
  • Version Data Deleted
  • Retain Extra Version
  • Retain Only version
  • Initial Filesystem/DB Size* GB
    % Data Added/Day* % (> 0)
    % Data Changed/Day* % (> 0)
    % Data Deleted/Day* % (> 0)
    Versions Data Exists* (1-9999 )
    Versions Data Deleted* (1-9999 )
    Retain Extra Versions* (1-9999 )
    Retain Only Version* (0-9999 )
    Backup Frequency*
    Time Interval*
    Time Span* Days (< 1000)
    *Results shown in the right pane depends on the Input provided which require thorough understanding of the system & environment and heuristic analysis of Data Change in future.
    Growth Of Data on Tape Library

    Sample Case Study of File level Backup

    Suppose We want to backup a Folder that contain log files(varying size, 17 KB to 1 GB). Retention period of each file is 7 days. If size of folder remains constant, then we can assume that 1/7th files in the folder are deleted daily and almost same no. of files are created daily, no files are modified. We are also backing up DB backup file ABC_DB.bak which is created and deleted daily in the same folder

    Assume following Data:

    Initial Filesystem Size: Log folder size :150 GB, Database Backup Folder Size : 380 GB, Total: 530 GB Data Added Every Day : 30 GB for Log Files And 380 GB for DB backup :((30+380)*100)/530 = 77% of initial file system Data Changed every day : 0 assuming files are not modified by different application = 0% Data Deleted Every Day : 30 GB for Log files and 380 GB for DB backup as retention period for Log files = 7 days and DB backup = 1 day : ((30+380)*100)/530 = 77% of initial file system As per Policy : Versions Data Exists: 9999 (NO_LIMIT) Versions Data Deleted: 9999 (NO_LIMIT) Retain Extra Versions: 30 Retain Only Version: 9999 (NO_LIMIT) Backup Frequency : Daily Now use above data and analyze the data growth on Tape, following observations were found:

    • Data on Tape at the end of 3 month is 41.3 TB (by Backup Calculator) and 'ACTUAL' data present on Tape for this scenario is 49 TB, Timespan: 3 months
    • Data is not deleted even when 'Retain Extra Versions' = 30 because REV comes into picture when data is changed and subsequent 'TSM PROGRESSIVE' backup created active and inactive copies of data, but here data is not changed
    • The Reason No_LIMIT(9999) is set to VDE and VDD is because of the need of Point in time Recovery
    • If Retain Only Version(currently : No-LIMIT) is set to 60 days then data will be stablized to 25 TB indefinately