Enterprise applications are inherently complex, and performance problems are unfortunately a common phenomenon. This article describes some of the most common performance issues, and provides insights to help you identify problems and eventually optimize the performance of your apps. The content here is tailored for the Joget platform, but the information provided can be applied generally for any enterprise environment.

Joget is an open source no-code/low-code application platform that empowers business users, non-coders or coders with a single platform to easily build, deliver, monitor and maintain enterprise applications. Getting apps built and deployed quickly is important, but having them perform well is essential. At the platform level, Joget has been optimized and streamlined as outlined in the article Need for Speed: How We Optimized Performance in the Joget Workflow v6 Platform. The Joget platform already takes care of a lot of things under the hood for running your apps. However, your apps are not immune to performance issues that can occur in enterprise apps, especially once you start introducing huge amounts of data, integration with external services and potentially erroneous custom code. Typically, there could be application level bad practices, bottlenecks and code errors that would greatly affect performance.

Thankfully Joget, especially the next generation Joget DX, provides built-in Application Performance Management (APM) features to help you identify and troubleshoot these performance issues.

Part 1: Most Common Performance Issues

The most common performance issues described here can be grouped into 4 main categories:

Memory
Database
Integration
Infrastructure

Do note that these issues are typical of any enterprise web application regardless of platform so you can also apply this knowledge outside of the Joget platform. For each issue there will be some recommendations that will be elaborated upon in a later section on How to Optimize App Performance.

1. Memory Issues

One of the most common problems in a Java environment, which Joget platform runs on, is running out of memory. Java uses a type of memory called heap memory within a Java Virtual Machine (JVM) which has a predefined maximum limit. There is automatic garbage collection to free up unused memory, but if the limit is reached and no more memory can be allocated or freed then the system throws an Out-of-Memory error and becomes unresponsive.

1.1 Insufficient Java Heap Memory Allocation

The default maximum JVM heap limit is usually rather small (¼ of total system memory in many JVM implementations), so it is unlikely to be enough for a real production environment. The Joget downloadable installers are actually pre-configured with low system requirements (less than 1GB maximum heap) for development use on a laptop.

The actual amount of memory actually required would vary depending on many factors e.g. the number and complexity of apps, size of data being processed, number of users, etc. So for example if you have many large apps that require at least 6GB of memory, and your maximum heap limit is only 4GB, then you are very likely to hit Out-of-Memory errors that would bring your system to a halt.

Recommendations: Optimize Memory and Server Configuration

1.2 Spikes in Traffic or Load

Sometimes, although a configured memory limit might seem to be sufficient, a sudden spike in application traffic or load might suddenly incur higher memory use, which can then trigger Out-of-Memory errors.

Recommendations: Optimize Memory and Server Configuration

1.3 Resource Leaks Due to Custom Code Errors

Any code that uses resources (e.g. creates objects, database connections, network connections or file I/O) must release them in a try-finally block. Any unreleased resources are considered leaks, which would mean that the memory usage would continually increase over time. If there are resource leaks, there will never be enough memory allocated as it will always be used up and eventually cause Out-of-Memory errors.

At the platform level, Joget has been carefully optimized and tested to ensure that there are no resource leakages. However, custom application code in custom plugins or BeanShell scripts are potentially major culprits, and this is actually one of the most frequent causes of reported performance issues so you will want to be very careful in your custom code.

Recommendations: Eliminate Resource Leakage

2. Database Issues

Databases are an integral part of any enterprise application, and the performance of the database plays a big role in how fast your application responds. If there are problems at the database level, the system would in turn be slow or unresponsive.

2.1 Slow Queries

In enterprise apps, each page request typically contains dynamic data that requires multiple database calls. If there are database queries which are slow, then by extension the response time of the application would be affected.

In many reported cases, slow queries due to complex unoptimized queries or very large datasets are the cause of performance bottlenecks so this is one main area you should focus on.

Recommendations: Minimize and Optimize Queries, Employ Caching

2.2 Large Query Results

Related to slow queries, database queries that return large result sets would require massive processing to process the results and to transfer via network I/O back to the application. This would incur high CPU, disk and network usage thus greatly delaying the response time of an application. Extremely large database tables on unoptimized database servers may even cause deadlocks.

For example, if a page in the application executes a query that retrieves many thousands of customer records at one time, that page will definitely take a long time to process. This increases the overall load of the system which also affects response times in the application as a whole.

Recommendations: Minimize and Optimize Queries, Employ Caching

2.3 Insufficient Database Connections

Joget and other enterprise systems utilize database connection pools to efficiently manage connections to the database. There are limits to the number of connections allowed, both in the connection pool and the database server itself, so if the limit is reached then requests requiring another connection will have to wait for previous connections to be freed.

In the event where there is high load and a large number of concurrent database connections are required exceeding the limit, then there is a bottleneck at the database connection limit. Worse still if there is a database connection leak (connections used in custom code which are not freed after use), which means that the connection will definitely be exhausted eventually.

Recommendations: Eliminate Resource Leakage

3. Integration Issues

Enterprise apps would typically involve some level of integration to external systems. Inefficient integration to these external systems could affect the response times of your apps.

3.1 Slow Service Calls

It is very common to integrate to external systems such SMTP servers to send emails, LDAP or Active Directory servers for user authentication, external database systems, as well as other REST, JSON or SOAP web services.

Depending on the scenario, such calls could potentially take a long time, from several seconds or even minutes in some cases. For example, if a page request in your app is actively invoking a web service call that takes a couple of minutes, then the user would have to wait for it to complete before getting a response which seriously affects the app user experience. Another example affecting performance is when the app retrieves user directory information from an LDAP server using an unoptimized LDAP query that takes many seconds to process and returns large results.

Recommendations: Use Background / Asynchronous Processing, Minimize and Optimize Queries, Employ Caching

3.2 Slow Transactions

Related to the slow single service calls, there could also be cases of slow transactions consisting of many individual tasks. For example, you could have a process that executes many small individual tasks sequentially. Although each task does not take a large amount of time, the entire transaction does, and so affects the actual response time for a page request.

Recommendations: Use Background / Asynchronous Processing, Minimize and Optimize Queries, Employ Caching

4. Infrastructure Issues

The Joget platform and apps run on enterprise servers, which would be an operating system on a physical host machine, a virtual machine or containers, whether in the on-premise or in the cloud. If there are problems at the infrastructure level, the performance of your apps would be affected.

4.1 Server Capacity Issues

A problem in server resources (for example insufficient CPU or physical memory, hardware faults or corrupted software services, etc) could cause slowdowns in your apps. If there are shared resources in a host, especially in a cloud or virtualized environment, then there could be the likelihood of other virtual guests or software processes hogging the CPU, memory or disk. In that scenario, there might not be enough resources for your apps to run properly.

Recommendations: Optimize Memory and Server Configuration, Load Test your Apps, Introduce Clustering

4.2 Network Connectivity Issues

Another infrastructure issue to consider is the network connectivity. Bandwidth congestion or a misconfiguration in the network services could cause slowness in application response times, either between the application server and the client browser, or even between the application and database servers.

Recommendations: Optimize Memory and Server Configuration, Load Test your Apps, Introduce Clustering

Part 2: How to Identify and Troubleshoot Performance Issues

Now that we have covered the most common performance issues, let’s see how we can identify and troubleshoot issues whenever a problem is encountered. Before any solution can be found, it is extremely important to know what the actual problem is. As an analogy, you can’t treat an illness or injury until you diagnose what is wrong, otherwise you could end up treating an arm when a leg is broken! So keep calm, and read on to see how you can troubleshoot performance issues.

1. Identify Problematic Pages and Actions

The first thing to keep in mind is to be precise about the problem and to narrow down the issue. It is no use to just say “the app is slow”, and expect for someone to magically resolve the problem. An app is unlikely to just consist of one URL, so first you will need to identify which URLs are problematic.
If only specific pages are slow or unresponsive, try to isolate the problem to specific database or integration issues. Record down the specific steps or actions taken in order to reproduce the issue.
Once specific pages or actions have been identified, start to pinpoint parts of your app (e.g. specific pages, forms, menu items, etc) as the potential cause. You can use the Joget Performance Analyzer and Application Performance Monitoring features (described later) to help identify the culprits.
If the entire application cannot be accessed at all, i.e. every single page cannot be loaded, then it is likely to be due to Out-of-Memory errors, infrastructure issues, or exhaustion of database connections due to connection leaks.

2. Utilize Application Performance Monitoring and Alerts

Joget DX introduces new built-in Application Performance Monitoring and Alerts. The monitoring is done at runtime, so it works even in actual production environments.
View Important metrics (e.g. Java VM memory usage, physical memory usage, as well as system and process CPU loads) in a graphical format over a selected period of time.
Configure alerts for the various metrics including errors, so that email notifications are sent when thresholds are exceeded. When configured, these notifications provide warnings so that proactive action can be taken before any service interruption is encountered.
Drill down into response times and throughput by requests for each application.
Whenever slow traces are encountered, detailed information will be available for troubleshooting including code level insights like database query performance and thread profiles.

3. Identify Bottlenecks with the Performance Analyzer

The Performance Analyzer is a feature in Joget that allows administrators and app designers to analyze the performance of their apps and to identify possible bottlenecks.
When enabled, the Performance Analyzer identifies and highlights elements within your app where the processing time exceeds a specific threshold.
Hover over a highlighted element to display details of affected method calls and/or database queries.

4. Check Logs

Monitoring of logs is an important way of finding and identifying issues. The location of log files depends on the application server used. For example, in Apache Tomcat there are several log files in the “apache-tomcat/logs” directory.
Access the logs directly from the application server, or download logs from the Joget web console. Joget DX also provides real-time streaming of logs in the web console for easier viewing and accessibility.

Look out for database connection leak warnings in the logs. Joget Workflow v6 introduced connection leak detection, which provides a warning mechanism that checks for potential unclosed JDBC connections that may cause leaks. This detection works for both BeanShell scripts and custom plugin code. When a possible leak is detected, a warning will be captured in the log file, e.g.

WARN 22 Aug 2016 12:06:10 org.joget.commons.util.Analyzer - Possible unclosed DB connections: 4; URL: /jw/web/userview/dbtest/v/_/form1_crud

5. Check Server Resources

Check the CPU, memory and disk usage on the application and database servers
If the Java process running the application server has consistently high CPU, then there could be very heavy processing in custom plugins or scripts. Running out of memory or errors due to resource leakages would also max high CPU usage in the Java process.
If the database process has consistently high CPU or disk activity, then it is likely to be one of the database issues.
If there are any other processes hogging server resources, then you should check on these misbehaving 3rd party applications.

6. Monitor Java VM and Take Thread / Heap Dumps

Check the heap memory usage of the Java Virtual Machine (JVM) . You can use JDK monitoring tools (jps and jstack utility) or VisualVM. You should see memory usage increasing and decreasing over time, as objects are created and garbage collected. If you see memory usage constantly increasing, then it is a sign of a memory leak in your apps.
If there is a memory leak, you can perform a heap dump to analyze the composition of objects, which might help to identify which portions of your app might need to be looked at.
If you are currently encountering slow page requests, perform thread dumps to find out what is currently running. Try to generate the thread dump files at the point of high memory usage or near crash. If the thread dump ran during normal operations, we might not see any abnormalities in the dump. If say the slow request takes 5 seconds during form submission, then submit that form and within that 5 seconds take a few thread dumps. Each thread dump is a "snapshot" at a point of time. This is to see what's happening at a different points in that period.

7. Check Database for Slow Queries, Running Queries, and Connections

Check for database issues by looking for slow queries. Different database servers would have ways to do this, for example in MySQL you can use the Slow Query Log.
You can also check running queries at a certain point in time to see if there are any long running queries currently being executed by the database server. For example in MySQL you can use the SHOW PROCESSLIST command.
If your apps are totally unresponsive, check the number of database connections to ensure that the database connection limit is not hit. In MySQL you can the SHOW STATUS command. If the limit is reached without connections being freed, there is likely to have unreleased connections in your apps.

Part 3: How to Optimize App Performance

Before wasting any effort on optimization, it is important to identify where the specific problems or inefficiencies are, following the steps in the previous section. The saying from renowned computer scientist Donald Knuth goes:

“We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%.”
Once you have identified the causes of performance issues in your apps, you can then employ the following optimization techniques as required.

1. Eliminate Resource Leakage

This is a critical consideration for performance, and is one of the most frequent causes of reported performance issues.
If the Java VM heap memory usage doesn't stabilize over time, it might indicate a memory or resource leak in your apps or plugins. If there is a leak, no matter how much memory is allocated, it will be taken up eventually.
Review your custom plugins and BeanShell scripts to make sure all resources are properly closed and released.
For example, you might be creating and using database connections without closing them. This would cause your database connection pool to be exhausted, and eventually cause the system to grind to a halt. When using custom Java or JDBC code in a BeanShell script or custom plugin, database connections must be closed in a try-catch finally loop e.g.

Connection con = null;

try {

// get connection

con = datasource.getConnection();

// more processing

} finally {

// close connection

try {

if (con != null) {

con.close();

}

} catch (Exception e) {

}

Likewise, any custom code that creates objects, network connections or file IO must release such resources in a try-finally block. An indication of unreleased memory would manifest itself in the Java VM memory usage, which would continually increase without reducing over time. If there is enough memory allocated without resource leaks, the memory usage should appear rather stable over time, as shown in the screenshot below.

2. Minimize and Optimize Queries

Slow database queries are another common cause of performance issues. It is therefore important to ensure that database queries are optimized.
It is significant to minimize the number of database calls that are made in the app. Check through your app to ensure that any database calls (or processing) are really necessary, if not remove them.
This also applies not just to the main userview pages, but also in the userview menus. For example, displaying a count of items in the menus incur a potentially slow database query, so you need to make a decision on whether or not to display it. You can test the performance of a page with and without menus by using the Userview Embedded Mode.
Optimize/tune the relevant database tables by adding appropriate indexes to speed up the queries. This will depend on the actual tables and queries, so you might want to consult a database administrator for this. For example refer to some optimization tips for MySQL.
Separate out the menu categories into different userviews if you have too many menus in a userview.
Over time, the database might have store redundant data that can be archived or deleted. Check for unused data that can be removed, including cleaning unused or completed processes in the process engine.
Optimization of queries also applies to other external services, for example when integrating to LDAP or Active Directory, ensure that your LDAP search queries are optimized.

3. Use Background / Asynchronous Processing

Sometimes it is unavoidable to have slow queries, service calls, or have long transactions with multiple activities in your apps. In this scenario, you can use background or asynchronous processing to prevent these from blocking responses to user requests.
When running long transactions or many activities in a process, make use of asynchronous deadlines to run these activities in the background.
If you have long running custom code in a plugin or BeanShell script, make use of Java threads to execute the code in the background.

4. Employ Caching

Caching is the storing of data in memory to reduce the need for database calls. When applied correctly, caching can greatly improve app performance and scalability.
Joget provides automatic support for optional caching for all userview pages. The Userview Caching feature allows for pages to be easily cached by selecting a scope and duration.
It is quite common to have large data sets when loading options in a form field (e.g. in a select box), so you can take advantage of the Form Options Caching feature.
If there are unavoidable slow and complex database or service calls, you can consider to temporarily store pre-calculated results into separate tables, so that your app can access using much faster queries.

5. Optimize Memory and Server Configuration

An important system configuration would be the Java VM heap memory allocation, in addition to database and application server tuning.
If this is too low, the system will run out of memory. However, if the setting is too high, there might be quite a large overhead in garbage collection. To get an optimum setting might require a bit of trial and error sometimes, depending on the usage environment. Tune the Java VM to suit your environment.
Tune your database server with the help of your database administrator (DBA). For example, in MySQL you can configure buffer pools and query caches to speed up queries considerably. Different releases of a database server may also have an impact, e.g. MySQL 5.7 provides a significant performance boost over MySQL 5.5.
Tuning your application server can also increase performance and throughput. For example in Apache Tomcat you can increase the maximum number of threads.

6. Introduce Clustering

When you have optimized your apps, you can increase your server capacity as appropriate to handle increasing load (vertical scaling). You can also start to consider performing horizontal scaling i.e. to cluster or load balance your installation. This can not only handle increased load, but also offer high availability. Read more on this at the Server Clustering Guide.
For modern application deployments, consider using cloud native technologies such as containers and Kubernetes that allow for easier scaling for apps. An example of this for the Red Hat OpenShift Container Platform can be found in the OpenShift blog post How to Automatically Scale Low Code Apps with Joget and JBoss EAP on OpenShift.
Once you start deploying and managing multiple apps, it becomes difficult to monitor, troubleshoot and scale apps independently. In this scenario, you would want to start distributing apps into separate clusters. You will also likely integrate with a directory server (e.g. LDAP or Active Directory) to centralize user management.

7. Load Test your Apps

There are many factors involved in determining the servers or clusters needed to run your apps effectively. These are some of the non-conclusive factors:

Total number of users
Maximum expected concurrent users
Number of apps running on the platform
Complexity of each of the apps
Amount of data generated in each app
Network infrastructure

For example, an environment with a small number of users running a heavily-used complex app might require more resources than an environment with large number of users running some simple apps only once a day.
To determine the actual requirement for your app and usage, it is best to perform load testing in your environment. There are many free and commercial load testing available, there is an article using the open source Apache JMeter tool at Joget Workflow Clustering and Performance Testing on Amazon Web Services (AWS)

Search This Blog

Joget Blog

How to Solve Your Enterprise App Performance Problems