Introduction

The modern world requires organizations to cooperate more closely with each other. Supply chains must be more efficient and this requires fast and reliable information and data flow. Organizations expect an immediate response from their partners - as if they were one company. This may be difficult when organizations use different computer systems. 

The typical approach to integrating such systems is using an existing API. In this ideal world, one system exposes API (application programming interface) and another consumes it. The system consuming an API will also have some kind of adapter that converts between system data models. Once the integration is complete, systems can to “talk to each other”, limiting human interaction to high-level supervision or maintenance. There are many ways to integrate systems via REST or SOAP API, database, file exchange on the FTP server, or even via email. 

In this article, however, we will study an approach where human interaction is necessary, and how this effort can be minimized to optimize repetitive tasks and introduce business process automation. 

A practical use case

As the introduction may seem a little generic, we will introduce a practical use case. Apart from custom software development, our company - 3e - offers support services based on Service Level Agreements (SLA). One of the most common parameters in SLAs is the time needed to acknowledge an issue and the time to provide a solution. This process will range from minutes, for critical/emergency issues, to hours or days for routine/low priority tickets. At 3e, we use the Redmine ticketing tool to organize both development and support projects. However, corporate clients are often reluctant to use the supplier’s system and will stick to their current solution. 

This is the case we faced with one of our larger clients, who required us to use their internal and custom Support Ticketing System (STS). 

Our first thought was to integrate Redmine with STS via API. Unfortunately, this could not be achieved for various reasons (STS not exposing API and the necessity of security hardening). To maintain this support, we had to create a support team that manually updated issue statuses in both systems. This, however, required time and effort and created an actual couple of minutes delay for every ticket acknowledgment and solution. Hundreds or even thousands of tickets per year would have a significant impact on both labor costs and service quality. But as a software company, we didn’t give up but looked for alternatives to system integration. 

Requirements

The first step was to diagnose the most time-consuming tasks and formulate requirements. The analysis yielded the following results:

• the most time consuming and mundane tasks require manual data synchronization
• synchronized data includes user/developer comments and file attachments
• STS and Redmine tickets are from different data models, and in particular, have different statuses and categories

To speed up the process:

• each new ticket in STS should be introduced automatically or semi-automatically in the Redmine system
• STS tickets status updates should be synchronized to Redmine (and vice versa) without manual effort
• tickets statuses and categories should be assigned automatically

Vision and technology stack

To simplify the solution, we assumed that integration would be tackled from the front-end only. Each essential required step will be processed semi-automatically. It should start with user action (i.e. after button click event) and allow the user to verify synchronized data before final submission. This is still robotic process automation, however, with an attended software agent being used.

We chose the Tampermonkey extension for Chromium-based browsers (compatible with Chrome and also Edge). Tampermonkey, a descendant of the even older Greasemonkey plugin for Firefox, is a versatile extension that allows users to create custom JavaScript scripts (known as user scripts) that are executed when the page URL matches the pattern. Typically, user scripts can automate logon, remove ads, alter page behavior or layout, or even in some cases, skip CAPTCHA. Both extensions are quite mature, created initially in 2004 and 2010, and should be regarded as battle-proven and reliable. However, please note that user scripts have access to local storage, layout, cookies, and user session. Moreover, it is essential to understand the security risks and install external scripts only after a thorough examination. 

Implementation details

Full implementation required 28 kb of code in three scripts. As this article is not a detailed tutorial, it will focus on the essentials only. Once the extension is installed, it is time to create a new script - to do so, click on the extension icon and choose the appropriate option.

Demonstration

Here’s a short screencast with the solution in place. We start with an STS page. After enabling the Tampermonkey extension, orange buttons appear. Usually, the extension is enabled all the time. By clicking the “add to Redmine” button, incident data is saved in local storage, Redmine is opened in a new tab, and data is used to create a new ticket. The whole process is optimized with two clicks.

Script declaration

Creating a new script opens Tampermonkey’s built-in editor. One should start with defining script parameters:

// ==UserScript==
// @name STS2Redmine
// @namespace http://tampermonkey.net/
// @version 0.8
// @description sync STS with Redmine
// @match https://sts.clientdomain.com/incident*
// @match https://redmine.3e.pl/projects/clien*/issues/new
// @match https://redmine.3e.pl/issues/*/edit
// @grant GM.setValue
// @grant GM.getValue
// @grant GM_xmlhttpRequest
// @grant GM.xmlHttpRequest
// @run-at document-idle
// ==/UserScript==

The script header starts with a metadata block with various keys and values. While some are obvious such as name, version, or description, others will require a little explanation. 

@match

The @match key defines patterns for URLs where the script is executed. It is crucial to narrow the pattern and be extra careful with scripts that run everywhere (*). Please note that our script uses three URLs: the STS incident page, the Redmine page for adding new issues, and a Redmine page that allows issue edition. Typically, the script should run on a single URL within a single domain. However, our script will move data between systems using local storage. Therefore, it must include more locations as data storage is restricted to a single script.

@grant

The @grant keys explicitly allow the script to use additional special-purpose functions. In our example, we use GM.setValue/GM.getValue to access extension local storage. This is a typical property bag, where objects can be serialized via the JSON.stringify function.

The script will also use the GM.xmlHttpRequest function that will be used to send custom requests to Redmine. Of course, JavaScript has a native XMLHttpRequest function, and jQuery can simplify its usage even more. However, native requests are subject to Cross-origin resource sharing (CORS) policies and not always will yield expected results. Therefore, sending an extension level request with higher privileges and works outside CORS policy is much safer.

Core setup

Now it is time to set up core objects, modules and split our script into two domains. 

/* globals jQuery,uploadAndAttachFiles */
(function() {
  'use strict';
  var $ = jQuery;
  var inc = ''; // an STS incident number
  var issue = ''; // a Redmine issue number
    (function ( $ ) {
      $.fn.scrollTo = function(elem, speed) {
        var $this = jQuery(this);
        var $this_top = $this.offset().top;
        var $this_bottom = $this_top + $this.height();
        var $elem = jQuery(elem);
        var $elem_top = $elem.offset().top;
        var $elem_bottom = $elem_top + $elem.height();
        if ($elem_top > $this_top && $elem_bottom < $this_bottom) { return; }
        var new_scroll_top;
        if ($elem_top < $this_top) { 
          new_scroll_top = {scrollTop: $this.scrollTop() - $this_top + $elem_top}; 
        }
        else { 
          new_scroll_top = {scrollTop: $elem_bottom - $this_bottom + $this.scrollTop()}; 
        }
        $this.animate(new_scroll_top, speed === undefined ? 100 : speed);
        return this;
      };
    }( jQuery ));

    (function ( $ ) { $.fn.filled = function() { 
      return this.css("background-color", "#FFDDCC"); }; 
    }( jQuery ));

  if (window.location.host == "sts.clientdomain.com") 
  {
    // ...
    return; 
  }
  if (/https:\/\/redmine.3e.pl\/projects\/client.*\/issues\/new/.test(window.location.href))
  {
    // ...
    return;
  }
  if (/https:\/\/redmine.3e.pl\/issues\/\d+\/edit/.test(window.location.href)) 
  {
    // ...
    return;
  }
})();

The first line declares the global variables that will be used in the script. This will turn off syntax warnings for missing declarations. The first one is jQuery, which is luckily available in STS (however not under typical $ variable), and the second is a Redmine fronted function for attachments upload. Since we are going to heavily alter and use DOM elements, jQuery seems like the right tool to ease the effort. If jQuery is not used by the page, one can use @require keys to include any script.

Next, two custom jQuery functions are declared. The first one will scroll the browser window to the selected element. This will be used every time the script will set values for input fields. The second is just a simple function that changes the background color. This function is executed on each modified input element to visually mark changed elements.

Finally, the most important part is dividing the script into three separate parts executed on different URLs. Again, this is necessary as the single script will access the same extension local storage. This may look strange, and one could argue that scripts should be further divided. However, local storage would have to be replaced by an external service or clipboard. 

Synchronizing data

Data synchronization occurs in several steps. First, we have to gather all the necessary data. This will require opening Chrome DevTools (or Developer Toolbar) and finding necessary DOM elements, and identifying them by some kind of jQuery selector. For example, to identify the issue, our user script should include:

inc = $("#sys_displayValue").val();

Luckily for us, there is a simple hidden input with an incident number. Looking for the right elements may require some time; however, once the information is visible in the browser, it can be safely scrapped using jQuery.

We continue by gathering incident information and creating a link/button that will copy it to Redmine.

// add link to copy issue to Redmine
var $ad = $(`<a id=’add' class='baton' href=#>add ticket to Redmine</a>`); 
$("div.navbar-right span[id^=section_head_right]:first").prepend($ad);
$($ad).click( ()=>{
    // set values
    var ticket = {
        inc : inc,
        des : $("textarea[name='incident.description']").val(),
        imp : $("input[name='sys_display.incident.u_service']").val(),
        trt : $("input[name='incident.u_target_resolution_time']").val(),
        att : jQuery("li.attachment_list_items a.content_editable").get().map(v=>({url:v.href,name:v.innerHTML})),
    }
    GM.setValue("STSincident", JSON.stringify(ticket))
        .then( v=> {
        window.open("https://redmine.3e.pl/projects/client-support/issues/new", '_blank').focus();
    }, reason => {
        console.error("STS2Redmine ::: GM.setVal error ",reason);
    });
});

A simple div with a single anchor link is created (id set to `add`). An event is attached that creates a ticket object, serialized as a STSincident internal value. The ticket properties include an incident number, description, impacted service, resolution time, and finally, a list of attachments. This list will include the names and URLs of those files. Finally, a window is open, and the same userscript will be executed but in the Redmine section, as explained above. 

We should add links/buttons copying incident comments or updates in a similar fashion. However, as there may be many comments, a class should be established for all the links (a.kdr selector in the example below).

Adding new issue

We will not cover here how data is introduced in Redmine. But, to summarize - a new issue form is opened, and once the userscript reads a STSincident value with GM.getValue, all necessary form fields are filled and their background color altered to signify the change. Again, we have assumed that a person is already logged in Redmine.

A failsafe - Checking for already synchronized issues

Finally, we can check just in case if the issue has already been transferred to Redmine. We won’t be using the Redmine API in the code below but will send a simple request to search form and scan all of the links in the results. 

GM.xmlHttpRequest({
    method: "GET",
    url: `https://redmine.3e.pl/search?all_words=1&issues=1&titles_only=1&q=${inc}`,
    onerror: ()=>{
        $("a.kdr").remove(); // remove all links copying statuses to Redmine 
    },
    onload: function(response) {
        var rt = response.responseText;
        var $ar = $(`a:contains(${inc}):first()`,$(rt));
        if ($ar.size()>0) {
            var href = $ar.prop("href");
            issue = href.split('/').pop(); // (/issues\/(\d+)/.exec(href)||['','']).pop()
            var status = (/\(([\w\s]+)\)/.exec( $ar.text() )||['','?']).pop();
            // is the issue digits only
            if (/^\d+$/.test(issue)) {
                $("#add").remove(); // issue found, remove link adding it to Redmine
            }
        }
    }
}).then( ()=>{
    if (issue=='') $("a.kdr").remove(); // no Redmine issue found, remove status copy links
});

Once the request gets a response, it is parsed. If the issue for the incident has already been introduced, we remove the appropriate button to avoid duplication. In the case that it has not been introduced, we should remove buttons copying status updates and comments. 

Alternatives

Apart from semi-automatic front-end scripts, one could also use a fully automated browser. Such tools are typically used for testing purposes where lots of manual tests are recorded and then played back in a containerized web browser. The most common solutions are: 

• Puppeteer - a headless browser that is programmed in Nodejs
• Cypress - a JavaScript framework for end-to-end testing

Hypothetically, this alternative solution would require setting up a web browser periodically checking for new tickets or status updates and automatically syncing the data between the systems. However, this may raise security concerns as the web browser would have to run with actual support member credentials.

Summary

We have shown how to speed up manual data entry and speed up system integration using Tampermonkey scripts. Feel free to contact us if you require any help in this area.