AllFusion Harvest Change Manager UDPs

Document ID : KB000055903
Last Modified Date : 14/02/2018
Show Technical Document Details

Introduction

Processing large lists with a User-Defined Process (UDP) can be a challenging task depending on which operating system you use. AllFusion Harvest Change Manager offers different ways to handle the input into a UDP so that your process results will not be affected by the operating system limitations to the command line. This article effectively describes how to invoke and write UDPs that can handle all kinds of large lists.

Using Variables in UDPs

The use of UDPs as pre-link or post-link processes is very common in AllFusion Harvest Change Manager (referred to simply as Harvest). Additionally, stand-alone UDPs are widely used across our customer base to enhance the native capabilities of Harvest and to meet the expectations and needs of the users. The most common type of UDP is the notify process, which in Harvest is a special UDP type.

The power of Harvest is to create generic UDPs that later, during execution, are adapted to the specific context that the UDP is running in. To make UDPs generic, Harvest passes parameters in an argument list to the command line program. These parameters vary depending on the context set in Harvest. For example, if a UDP uses the [package] variable, this variable is replaced on execution with the name of the package that is used. When promoting a package, the user may want to send an email automatically to managers to notify them that a package has been promoted to the next state.

Most of the variables return a single value, for example, [broker] or [project]. Other variables return a list of values, for example, [version], [file], and [package], depending on the selection by the user. For example, if a user promotes 10 packages to the next state, the [package] variable will be replaced with the names of the 10 packages.

The most problematic variables are [version] and [file]. These variables can be replaced with a very lengthy list. A check out pre-link UDP that uses the [version] variable to process the list of versions will become very lengthy if a user executes a complete check out of the repository attached to a project. Problems are more likely to occur on big projects, such as Java projects where the files are small but numerous and the view paths are very long (Java Package naming convention). The UDP that handles the list may fail with the following error message:

---------- Begin  <ARG Test>  Process ---------------
E03020132: Error: Couldn't find UDP program : perl.exe "C:\Harvest\harvest\CCC_Harvest\arg.pl"
"\WSAD Webcast\Param\true.log;0" "\WSAD Webcast\Param\Build.bat;0" "\WSAD
Webcast\Param\Sql\SqlDb\DBUnit.h;0" "\WSAD Webcast\Param\Sql\SqlDb\U
E03060019: ERROR: Process Execute failed. Process Name: ARG Test .
---------- End  <ARG Test>  Process ---------------

For a list of variables and their descriptions, see the appendix "System Variables" in the Administrator Guide.

Argument List Limitations

The argument list length varies by operating system. Harvest interacts with the operating system and is dependent on the operating system capabilities.

The UNICODE_STRING structure limits the maximum argument list length for the CreateProcess function to 32,767 characters. CreateProcess is the core function for creating processes, so if you are using Win32 that is the only limitation concern. If using the CMD.EXE command processor, you are additionally subject to an 8,192 character command line length limitation imposed by CMD.EXE. If you want to pass more than 32,767 characters of information to a process, you need to use a method other than the command line. Some options will be discussed later in this document.

UNIX has a larger limitation than the Unicode limitation, but it is not unlimited. Limitations on UNIX can vary and the maximum is 4 MB. In the case of a large Harvest project with numerous versions and items, this 4 MB limitation is insufficient. Some users execute UDPs with version lists that may reach 20 MB. All UNIX and Linux operating systems can be adapted, but all of them share the same problem that occurs on very lengthy lists.

For example, Sun UNIX limits the length of the command line. In the file,
/usr/include/limits.h, the limitations are defined. See the line:

#define MAX_INPUT       512     /* max size of a char input buffer */

By changing these values, the Sun UNIX administrator may allow longer lists, but an UNLIMITED setting is preferred.

Despite all this, the use of arguments is not the best method for handling very lengthy lists.

Differences Between [version], "[version]" and ["version"]

The variable ["version"] is used once in the UDP setup that is described in the section Recreating the Problem. The variable [version] is used in the UDP setup that is described in the section Solving the Problem. By using both variables, you may better understand how to handle these variables and the difference between them. The following table lists and describes the version variables:

["version"]Quotes located in brackets expand the system variable list members quoted individually: ["version"] evaluates to "version name 1" "version name 2" ...
"[version]"Quotes located outside the brackets expand the system variable list members as one quoted list: "[version]" evaluates to "version name 1 version name 2 ..."
[version]With no quotes located outside the brackets, the system variable list members expands as a simple list: [version] evaluates to version name 1 version name 2 ...

Handling Very Lengthy Lists in UDPs

In this section, you will recreate the problem of lengthy lists and then fix the problem.

Recreating the Problem

The following sample scenario recreates the problem of lengthy lists:

  1. Create a Harvest project and repository.


  • Attach the repository to the project.


  • Check in more than 20,000 files.


  • Create a pre-link UDP to the check out process that passes the ["version"] variable. This UDP takes all versions and writes them to a file.
    1. Enter the following sample Perl code in the Input field of the UDP Properties dialog:
      #!/usr/bin/perl  
      open(ARGUMENTS, ">VERSION.TXT");
      foreach $arg ( @ARGV )
      {
      print ARGUMENTS " $arg\n";
      }
      close (ARGUMENTS);
      exit(0);
    2. Set up your UDP Properties dialog as follows:

  • Select 20,000 files and try to check them out. This results in the failure message:
    ---------- Begin  <ARG Test>  Process ---------------
    E03020132: Error: Couldn't find UDP program : perl.exe "C:\Harvest\harvest\CCC_Harvest\arg.pl"
    "\WSAD Webcast\Param\true.log;0" "\WSAD Webcast\Param\Build.bat;0" "\WSAD
    Webcast\Param\Sql\SqlDb\DBUnit.h;0" "\WSAD Webcast\Param\Sql\SqlDb\U
    E03060019: ERROR: Process Execute failed. Process Name: ARG Test .
    ---------- End  <ARG Test>  Process ---------------
  • Solving the Problem

    The Harvest UDP dialog has an Input field. Harvest variables can be used in this field and Harvest will parse the variables specified in the Input field to <STDIN>. Instead of using the arguments as we did in the previous example, use <STDIN>.

    The following sample scenario fixes the problem of lengthy lists:

    1. Create a pre-link UDP to the check out process that passes the [version] variable. This UDP takes all versions and writes them to a file.
      1. Enter the following sample Perl code for the usage of <STDIN>:
        #!/usr/bin/perl
        open(ARGUMENTS, ">VERSION.TXT");
        # loop through lines of stdin and print it out
        while (<STDIN>)
        {
          next if /^\s*$/;  # ignore blank lines
          s/^\t//;    # trim the leading tab
          chomp;    # trim the trailing EOL
              # version, file or package name is now in $_ 
          push(@STIN, $_);  # push the $_ into a Array
        }
        foreach $stin ( @STIN )
        {
        print ARGUMENTS " $stin\n";
        }
        close (ARGUMENTS);
        exit(0);


    2. Set up your UDP Properties dialog as follows:

  • Select 20,000 files and try to check them out. Harvest succeeds with the message:
    ---------- Begin  <ARG Test 2>  Process ---------------
    I00060052: ARG Test 2  execution was successful.
    ---------- End  <ARG Test 2>  Process ---------------
    A sample of the output file VERSION.TXT follows:
    \WSAD Webcast\TEMPLATE\Ad\PKG0087.DOC;0
    \WSAD Webcast\TEMPLATE\Ad\PKG0072.DOC;0
    \WSAD Webcast\TEMPLATE\Ad\PKG0127.DOC;0
    \WSAD Webcast\TEMPLATE\Ad\PKG0107.DOC;0
    \WSAD Webcast\TEMPLATE\Ad\PKG0066.DOC;0
    \WSAD Webcast\TEMPLATE\Ad\CSTEST4.DOC;0
  • Sample Java Code to Handle STDIN

    The following code is a sample for <STDIN>:

    vnReader = new BufferedReader(new InputStreamReader(System.in));
    interLine = vnReader.readLine();
    vnLine = "";
    while (interLine != null) {
    vnLine = vnLine + interLine;
     interLine = vnReader.readLine();
      }
     vnSt = new StringTokenizer(vnLine, "\"", false);
     while (vnSt.hasMoreTokens()) {
     fullItemPath = vnSt.nextToken().trim();
     lastSeparator = fullItemPath.lastIndexOf('\\');
     lastComma = fullItemPath.lastIndexOf(';');
     itemPath = fullItemPath.substring(0, lastSeparator);
     itemName = fullItemPath.substring(lastSeparator + 1, lastComma);
      if (myHar.getResultVector() != null) {
       stopLoop = myHar.getResultVector().size();
       for (startLoop = 0; startLoop < stopLoop; startLoop++) {
       myHarRow = (HarRow) myHar.getResultVector().get(startLoop);
     System.out.println("Element " + itemName ");
            }
              }

    Conclusion

    The inability of handling lengthy lists is an operating system based problem involving the length of the ARGV (Argument buffer). Harvest interacts with the operating system and is dependent on the operating system capabilities. With the use of <STDIN>, users will be able to handle an almost unlimited number of items in the variable. A test using the STDIN approach with more than 700,000 items executed successfully. With this information, you will be able to handle very lengthy version, file, or package lists.