Application Migration from IBM DOC 3.x to IBM DOC 4.x

A detailed guide for anyone considering migrating an application developed with IBM DOC (Decision Optimization Center) 3.x to 4.x.

Why upgrade IBM DOC 3.x Applications to IBM DOC 4.x?

Organizations running optimization applications built on IBM DOC (Decision Optimization Center) 3.x are being strongly encouraged to migrate to DOC 4. DOC 3.x relies on technologies that are no longer being enhanced or supported, such as Websphere Application Server (WAS). Another, hopefully, stronger reason to migrate is to take advantage of the benefits that the new DOC 4.x platform offers. These include:

• DOC 4 is Cloud-ready: DOC 4.x leverages technologies like Docker and Kubernetes, making it easy to deploy and scale your applications and run them on the cloud. Kubernetes is an open-source technology that automates the management and deployment of applications on any cloud using the concept of “containers.” Of course, this is most relevant if you have the desire to move your application to the cloud (which could be a private cloud). Most organizations are shifting their apps to the cloud in order to reduce application management effort and better meet performance and functionality needs in different environments, but ultimately only you can decide if a transition to the cloud is right for you.
• DOC 4 Enables Better Scalability and Cost Control: DOC 4.x enables out-of-the-box allocation of CPU/Memory resources on-demand so that optimization runs in worker services can have dedicated resources allocated. Those workers can start and stop on-demand so that the user is only paying for cloud resources used.
• DOC 4.x supports the latest CPLEX Versions: Optimization applications built with DOC 3.x do not support the latest versions of CPLEX. DOC 3.9 fp2 is the latest version that supports CPLEX 12.10. Given that DOC typically has ~4 new releases per year, new versions of CPLEX should be supported by DOC 4.x and later versions within 90 days of the CPLEX version release date.
• DOC 4 Uses Open Source Standards: Most organizations are trying to increase their adoption of open-source technology in order to benefit from the diverse community of contributors whose collective wisdom generally outperforms proprietary technologies. This also makes it easier to find people with the right skills to maintain and enhance the applications. Both the front-end and back-end components of DOC 4.x applications are built on technology stacks that are widely known and used, like:- Java – for backend development (business logic, optimization models, etc)- Python – an alternative to Java for backend development- Spring Boot – for micro-services creation- Typescript/Angular- for front end UI development

  – RabbitMQ – task lifecycle monitoring

  – Keycloak – user authentication

• DOC 4 is Actively Evolving: Thanks to the modern technology stack on which DOC 4.x sits, a growing customer base, and active investment in the platform, DOC 4.x benefits from at least 4 new releases per year. New features come from both the DOC development team and from the wider Open Source community, which also helps ensure potential security vulnerabilities are addressed quickly.
• DOC 4.x Applications Can Be Deployed Much Faster: DOC 4.x applications rely on a Web User Interface which means that no installation is required in the end-user environment. DOC 4.x applications are published as a set of container images that gives access to simple and standard deployment technologies (for example, helm charts).

Upgrading your application to DOC 4.x can reduce costs and risk while enabling you to more easily adapt your application to meet the needs of tomorrow.

DOC 3.x vs DOC 4.x

DOC 3.x Development and Deployment Approach

DOC 3.x comes with different installers. One for development and one for deployment.

The development environment includes a version of the Eclipse IDE (Integrated Development Environment) and a set of artifacts that are used to compile and run the application.

The deployment environment contains three predefined components:

• A data server
• An optimization server
• A desktop application

The first two components must be installed on the IBM WebSphere platform and require two main databases. The last component must be installed on the computer of each end-user of a DOC application.

DOC 3.x applications are built as a set of Java libraries and configuration files that are all bundled together for deployment (odmapp file). This bundle is then deployed on top of the predefined components through a process that is partially automated and partially manual.

DOC 4.x Development and Deployment Approach

DOC 4.x is a platform one uses to develop optimization-powered applications, such as planning or scheduling applications. As such, the platform itself does not need to be ‘deployed.’ Only applications that are generated by DOC 4.x can be deployed. In this sense, DOC 4.x can be seen as an application generator that produces a set of artifacts (UI, runtime server, data connections, etc) that are ready-to-use in the generated application.

DOC 4.x does not come with an IDE but can be used with any modern IDE like JetBrains Intellij for example. All the artifacts required to generate the application and to compile and deploy it are available in dedicated registries for download when they are needed. Those artifacts are java libraries, node modules and docker images.

A DOC 4.x application is built as a set of Java libraries, javascript modules and micro-services packaged in containers (Docker images). Those Docker images must be published to a container repository (Nexus for example) and can then be deployed in production in an environment that supports containers.

The deployment environment can be as simple as a pure Docker setup (using Docker-Compose for configuration) or a Kubernetes cluster (using helm charts for configuration).

DOC 3.x vs DOC 4.x – Side-by-Side Feature Comparison?

Notable breaking changes are highlighted in orange. Benefitials changes are highlighted in green.

Technology

Development Environment

1. DOC 4.x will work with the data model definitions from DOC 3.x, so porting an existing DOC 3.x data model is straightforward.

Backend Features

2. This feature requires a proprietary database and sophisticated “copy on change” technology. This makes scenario duplication fast, but the downside is that it’s not possible to clean the database and is extremely difficult to back-up.

Frontend Features

DOC 4 Data Table Comparison

DOC 4 Scenario Comparison

Frontend Features – Widgets

DOC 4 Dashboard Configuration

DOC 4 Job Monitoring

Integration

3. Keycloak provides various methods to integrate with LDAP.

Deployment

How complex is the DOC Migration?

To assess the effort to migrate an entire DOC 3.x application, each of these application components should be assessed:

• Data processing (modeling, Integration with external systems, pre/post-processing)
• Optimization Engine
• User Interface
• Deployment
• Migration of existing scenarios
• Users & Authentication

Following are some of the specific considerations that one should consider before undertaking a migration.

Data Processing

Data Model Migration

In DOC 3.x, the application data model is defined interactively using the Data Perspective of the IDE. This definition is then used to define the schema of the application scenarios. This data model definition is stored in a DBM file.

In DOC 4.x, the application data model is defined manually in a textual format (JDL). A migration tool is available to automatically generate this JDL file from the DBM file. Contact DecisionBrain if you’d like a copy of this tool.

Data Processing

In DOC 3x., two APIs are available to handle data using the Java language:

• The Data Object Model (DOM) which is a set of java classes automatically generated by DOC 3.x that allows handling scenario data using java classes.
• The IloTable API which is a set of IloXXX classes that allow handling scenario data using relational concepts (tables, rows, columns).

DOC 4.x supports a DOM with a Java API that is fully compatible with the one that is included in DOC 3.x. No IloTable API is available in DOC 4.x

As a result, code that uses the DOM is fairly straightforward to migrate. Code that uses the IloTable API will need to be migrated to use the DOM, but is usually straightforward. It should, however, be validated through unit-testing.

Integration with External Systems

In DOC 3.x, integration with external systems is performed using the Java language. It is usually a combination of data processing (see the previous section) and of calls to java libraries (like JDBC for example).

DOC 4.x relies on the same principles, and provides a data integration API to ease the integration with external systems.

Optimization Engine

Some work may be required to adjust the import and export of the data to and from the engine.

Optimization Engine Encapsulation

In DOC 3.x, an optimization engine is run within a Custom Task inside the Optimization Server. This requires writing a Java class that receives the scenario and then transforms the scenario data into the format that the optimization engine expects.

In DOC 4.x, a similar mechanism is available through the encapsulation of the optimization engine into a so-called “Worker”.

Data Mapping

In DOC 3.x, an optimization engine could use:

– The automated OPL Data Mapping:

• The engine is written in OPL with a mapping defined interactively in the Eclipse perspective
• Scenario data is automatically passed into the OPL tuples and the solution is automatically extracted from the OPL tuples to be stored back into the scenario.

– Manual Data Mapping:

• The engine is written using the Java API of a library (CPLEX, OPL, other)
• Scenario data is mapped into the library data structures using manually written Java code. Similarly, the solution is extracted from the library data structures and written back into the scenario.
• Scenario Data is accessed using either the DOM API or the IloTable API (see above).

DOC 4.x does not provide “automated OPL Data Mapping” which means that some Java code must be written to implement a DOC 4.x “Manual Data Mapping”.

If a “Manual Data Mapping” was used, it must be ported to become a DOC 4.x “Manual Data Mapping”.

User Interface

Architectural Changes

DOC 3.x provides a user interface based on a Java Desktop application. The features supported by the user interface are tightly linked with this “heavy-client” architecture. When a view is opened, data is loaded locally in the client and the end-user is allowed to interact with the data locally without performing any synchronization with the back-end of the application. This lack of continual synchronization can cause issues, particularly with the scenario database.

DOC 4.x solves this issue by using a modern web-based user interface. This way, data changes are immediately synchronized between the web browser and the back-end. The scenario database always reflects what is displayed in the web user interface.

This architectural change means that some features in DOC 3.x cannot be exactly replicated in DOC 4.x:

• Undo/redo

Technology Changes

The DOC 3.x user interface is built on top of Java Swing, which enables integration of complex widgets from IBM/ILOG JViews (Gantt Chart, etc.).

The DOC 4.x user interface is built on top of Angular, which is a popular open-source platform, originally from Google, for developing mobile, desktop, and web-based applications. Many UI libraries for Angular are available and are much more widely used than JViews. Depending on the features needed, additional licensing for certain libraries may be required.

User Interface

DOC 4.x provides an implementation of most of the standard DOC 3.x widgets. Most standard DOC 3.x views can be migrated without coding through an interactive configuration of dashboards.

OPL Widgets

DOC 4.x does not provide a tight integration with the OPL language. The following views are not supported out of the box:

• Requirements view
• Goal view

If needed, time should be allocated to build these views through customization as part of the migration process.

Pivot Table for Large Tables

DOC 4.x provides a simple pivot table widget that is not meant to serve data coming from tables that exceed 10k rows. If the DOC 3.x application has larger tables and requires pivot table representations, then integrating an external library will be required (ag-grid enterprise is a good candidate for this – www.ag-grid.com).

JView Extensions (Gantt Chart, etc.)

As described earlier, if the DOC 3.x application relies on complex widgets coming from the JViews library, then external libraries will need to be used to provide similar functions.

Derived Columns

DOC 3.x has a “derived columns” feature that allows one to set up an automated recomputation of some columns when others are edited. DOC 4.x provides an equivalent feature that is called “computed fields”.

Scenarios

The level of effort to migrate scenarios is based on the following:

• # of scenarios kept in the application
• # of tables
• # of rows and columns in the biggest table
• Typical scenario size
• The exchange format (crf, CSV, database …)

It may also be necessary to re-implement the data read/write functions for scenarios to work in 4.x because the data format may no longer be consistent with the expected data model in DOC 4.0.

In order to improve long-term maintainability and supportability, it is strongly recommended to consider adopting the standard file export/import format used by DOC (Excel for example) to avoid maintaining the legacy file exchange formats.

Deployment

The level of effort to migrate deployments depends on the following factors:

• Number of environments (preprod, integration, test, prod)
• What is done on each environment
• Sizing of each environment
• Application criticality (usually, the more close to execution, the more critical)
• Who is responsible for the deployment and first-level support (for example if the customer’s IT team is responsible, that will reduce the cost of services from DecisionBrain)

Users and Authentication

Migrating the users and re-setting up authentication configuration typically takes just a few days. Factors that impact the level of effort are:

• do they integrate to a specific SSO system (LDAP …)
• # of user types/groups
• Access differences between the types/groups

Summary & Migration Support Services

Migrating a DOC 3.x application to the new DOC 4.x platform can be a significant undertaking, both in terms of the breadth of skills required (optimization, DevOps, front-end, back-end, data integration, etc) and total time and effort, which in our experience ranges from a few person-weeks of effort to several months. That said, the benefits in terms of cost savings, increased performance and improved adaptability and maintainability of the application will usually more than justify the effort, especially when you consider the cost of downtime of your critical planning, scheduling, or logistics applications.

DecisionBrain has successfully migrated several DOC 3.x applications, large and small, for various clients. We have also developed utilities to streamline portions of the migration. Feel free to contact us to discuss your migration project. We offer a free, no-obligation estimate and report of what will be involved in migrating your specific application. We are available to assist with migrations as well as provide training on DOC 4.x.

Check the latest DOC 4.x online documentation.

At the end of the migration effort, you will have access to a modernized application based on the latest technology stack for simplified deployment and maintenance.

Learn more in our YouTube channel