Hardware Sizing for Java/Java EE Products

For doing a hardware sizing on Java/Java EE; especially where indexing or search frameworks like Solr/Lucene are involved - many more attributes add to the final sizing tabulation. [This entry does not include anything on Database Sizing]

Before we get ahead with a sizing exercise, we need to understand that the following will impact the accuracy of the metrics.

1. Decision on Exact Version of Runtime, Used Frameworks, Servers,

2. Experience of the Senior Engineer/Architect doing the Sizing Exercise

3. Understanding the Functional Characteristics of the System being Built

4. Agreeing upon the Non-Functional Characteristics of System being Built

5. Appreciating Sizing Environments [Development, Testing, Production, UAT,...]

6. The Future Extensions or Possible Lifeline of the System Being Built

The following are the most important standard guidelines and criteria for hardware or server sizing. Please note that these guidelines are for the server that hosts the Application. It does not contain any Database Sizing guidelines.  

•  Hardware Component should operate at no more than 80% Utilization 
•  Processor and Memory Resources should be allocated for Maximum User Load 
•  User Think times and Network Latency should be taken into Account 
•  Number of Potential Users and Number of Concurrent Users 
•  Service Time and Average Response Time of your Application

If you are using Solr/Lucene type of indexing or disk-based frameworks; then it is important that you estimate the entire possible index size by deducing the number of documents, number of indexed fields and number of stored fields and also the average size of each document. By considering some buffer, you may be able to compute, almost accurately, the Disk Space. While computing the Estimated Memory Requirements for Solr/Lucene; additionally; the number of Unique Terms per Field also need to be considered. In the references below, I have provided a sheet (that has been made publicly available by 'Lucidworks'). It will also provide you all the attributes that you 'tune' to your Memory Sizing and Disk Space Requirements for Solr/Lucene, especially with respect to 'Caching of Query Terms'.

While doing sizing exercise, you can provide various tabulated forms as the result for each of the possible environments.  Alternatively, you may choose to present a single tabulated result (mentioning the environment for which you are providing this sizing). You may mention, the additional constraints that may be applicable across environments. It is important the buffer may be added to each of the computed attributes; keeping in mind the cost and future extensibility. Most of them come to a conclusion that "Hardware is Inexpensive these Days - We can Recommend something that is Beyond the Best Possible Maximum Load". Though this may work almost always, we may not be able to come out with a "Possible Minimum Estimate with Least Cost". Coming out with the the estimates; keeping latter in mind, will equip us to better understand the future issues that various functional and non-functional aspects may cause. This is especially if we want to achieve maximum efficiency under the constraints for all possible 'Loads'. For example, if we were to achieve this ('latter') in the 'Development, Testing or User Acceptance Testing' environments; we may be able ot point out that Memory Leak that would have manifested itself due to an Incorrect Development Practice or Deployment Strategy. Sometimes, we may also end giving a "Inflated Estimate" for an otherwise "Size-S System"; the resources which may always lie unused - if we go with the former approach.

Before I take you to the tabulation there are a few terms, that need definition (from the text book). They are very often assumed and the slight difference in their actual meanings may be better to know as it is.

User Think Time:  The time the user is not engaged in actual use of the processor (The time between Requests). This is used interchangeably with User Wait Time. In absolute real-life however, this has a slightly different impact as it involves the 'Time Required by an User for thinking and performing his next action in the application either due to the response or otherwise'.
Response Time: The response time measured at the client under load. (Average of Time). 
Concurrent Users:  The number of users measured on the server, taken in snapshots from the Server Status or Server Console.
Service Time:  The elapsed time to complete the operation measured for a single user.
Maximum User Load: The maximum number of concurrent users that may be expected or the system is tested for.

User Wait Times: The time elapsed between actions or clicks for a given user. This is used interchangeably with User Think Time. In absolute real-life however, this has a slightly different impact as it involves the 'Time Required by an User for analyzing or reading data received between request and also performing other tasks such as reading email, using the telephone, and chatting with a colleague or on other Applications simultaneously Running'. If we were to go deeper into Software Testing and Performance - Both of these may be put to great use to improve user experience and/or performance.
CPU Utilization: Average of the Total CPU Utilization as a Percentage.

The final tabulated Hardware Sizing Recommendation for the Java/Java EE Product will look like the following: (One Table is shown here for 'Development' environment and consideration for 'Production/UAT' environments provided below).

The Load Balancing, Data Clustering, Failover Strategy and Backup Strategy are not planned for, due to the nature of the System.

FIELD NAME	FIELD TYPE
Type of Environment	Development [/Testing]
Type of Machines	Physical [/Virtual]
Number of Servers	1x
Operating System	Red Hat Enterprise Linux - Linux X.Y.ZZ-AAA.BB.C.eRR.xpp_bb OS
Application Server	Weblogic ??c (Weblogic ??.?.?*)
Load Balancing	[NONE]
Data Clustering	[NONE]
Failover Strategy	[NONE]
Database Connections	10 [maxActive], 02 [maxIdle]
Backup Strategy	[NONE]
Processors	4 Cores
Concurrency	~500 Concurrent Users  [Including Think Times]
Memory / RAM	4GB
Garbage Collection	Generational Garbage Collector [-XX:UseG1GC]
Disk Capacity [Reasons]         Lucene Indexing	~10GB SSD [/HDD]   [Logs, Indexes, Dependencies, +Buffer]   ~300MB [Worst Case, +Buffer]
Java Heap Size    Lucene    Second Level Caching	Dedicated Machine [-Xms=??g -Xmx=??.?g]   ~100MB [Worst Case, +Buffer]   ~000MB [NONE]

 

This recommendation is for the Development Environment. It is best that the above is used / emulated for any of Development or Testing. For Production or User Acceptance Testing environments, the considerations (with our recommendations in brackets) related to Storage Capacity [500GB SSD], Storage Redundancy [RAID], Processor Cores [08+], Total Memory (RAM) [08GB+], Application Failover Strategy [Active-Active with 4x Physical Servers] should best match with Other Organizational or Hardware Tier Standards.

I am giving you the Following Links, which can be used as Reference to Get the Best Results:

https://docs.oracle.com/html/A90444_01/sizing.htm#1032856
https://technet.microsoft.com/en-us/library/cc181325.aspx
http://lucidworks.com/blog/2011/09/14/estimating-memory-and-storage-for-lucenesolr/
http://lucidworks.com/blog/2009/09/02/scaling-lucene-and-solr/
https://svn.apache.org/repos/asf/lucene/dev/trunk/dev-tools/size-estimator-lucene-solr.xls 

Happy Hardware Sizing for Java/Java EE Products!

[Note: I am a Software Development Architect, working for a US based Software Product Company and this write-up is based on the work done as part of Special Product Customization for a Big Logistics Customer, as well as for later use in the Product Itself]. 

 

 

Java Memory Architecture (Model), Garbage Collection and Memory Leaks

Java Memory Architecture (Java Memory Model)

The above is the Java Memory Model for the Heap as well as the PermGen for any Java Application running in the Java Virtual Machine (JVM). The ratios are also provided to get a fair understanding of how the distribution of allowed memory is done across each of the generation types. All of the above is completely applicable up to Java release 1.7 (inclusive). The above is also known as the 'Managed Area' of the Memory Model.

In addition to the above, there is a Stack Area, which can be configured use the -Xss option. This area holds the references on the heap, native references, pc registers, code cache and local variables for all threads. This is also known as the 'Native Area' of the Memory Model.

Managed Area of the Java Memory Model (Java Memory Architecture)
[Young Generation/Nursery] Eden Space

All new objects are first created in the Eden Space. As soon as it reaches an arbitrary threshold decided by the JVM, a minor garbage collection (Minor GC) kicks in. It first removes all the non-referenced objects and moves referenced objects from the 'eden' and 'from' into the 'to' survivor space. Once the GC is over, the 'from' and 'to' roles (names) are swapped.

[Young Generation/Nursery] Survivor 1 (From)

This is a part of the survivor space (You may think of this a role in the survivor space). This was the 'to' role during the previous garbage collection (GC).

[Young Generation/Nursery] Suvrivor 2 (To)

This is also a part of the survivor space (You may think of this also a role in the survivor space). It is here, where during the GC, all the referenced objects are moved to, from 'from' and 'eden' .

 

[Old Generation] Tenured

Depending on the threshold limits, which can be checked by using -XX:+PrintTenuringDistribution, which shows the objects (space in bytes) by age - Objects are moved from the 'to' Survivor space to the Tenured space. 'Age' is the number of times that it has moved within the survivor space. There are other important flags like, -XX:InitialTenuringThreshold, -XX:MaxTenuringThreshold and -XX:TargetSurvivorRatio which lead to an optimum utilization of the tenured as well as the survivor spaces. By setting -XX:InitialTenuringThreshold and -XX:MaxTenuringThreshold we allow an initial value and an maximum value for 'Age' while maintaining the percentage utilization in the 'Survivor (To)' as specified by the -XX:+NeverTenure and -XX:+AlwaysTenure, as they suggest are used to either never tenure an object (risky to use) and the opposite usage is to always tenure, which is to always use the 'old generation'. The garbage collection that happens here is the major garbage collection (Major GC). This is usually triggered when the heap is full or the old generation is full. This is usually a 'Stop-the-World' event or thread that takes over to perform the garbage collection. There is another type of GC named as the full garbage collection (Full GC) which involves other memory areas such as the permgen space. Other important and interesting flags related to the overall heap are -XX:SurvivorRatio and -XX:NewRatio which specify the eden space to the survivor space ratio and old generation to the new generation ratio.

[Permanent Generation] Permgen space

The 'Permgen' is used to store the following information: Constant Pool (Memory Pool), Field & Method Data and Code. Each of them related to the same specifics as their name suggests.

Garbage Collection Algorithms

Serial GC (-XX:UseSerialGC): GC on Young Generation and Old Generation

Use the simple mark-sweep-compact cycle for young and tenured generations. This is good for client systems and systems with low memory footprint and smaller cpu.

Parallel GC (-XX:UseParallelGC): GC on Young Generation and Old Generation

This used N threads which can be configured using -XX:ParallelGCThreads=N, here N is also the number of CPU cores. for garbage collection. It uses these N threads for GC in the Young Generation but uses only one-thread in the Old Generation.

Parallel Old GC (-XX:UseParallelOldGC): GC on Young Generation and Old Generation

This is same as the Parallel GC, except that it uses N threads for GC in both Old and Young Generation.

Concurrent Mark and Sweep GC (-XX:ConcMarkSweepGC): GC on Old Generaton

As the name suggest, the CMS GC minimzes the pauses that are required for GC. It is most useful to create highly responsive applications and it does GC only in the Old Generation. It creates multiple threads for GC that work concurrently with applications threads, which can be specified using the -XX:ParallelCMSThreads=n.

G1 GC (-XX:UseG1GC): GC on Young and Old Generation (By Dividing Heap into Equal Size Regions)

This is  a parallel, concurrent and incrementally compacting low-pause garbage collector. It was introduced with Java 7 with the ultimate vision to replace CMS GC. It divides the heap into multiple equal sized regions and then performs GC, usually starting with the region that has less live data - Hence, "Garbage First".

Most Common Out of Memory Issues

The most common out of memory issues, which all Java Developers should know, so as to start debugging in the right earnest are as follows:

• Exception in thread "main": java.lang.OutOfMemoryError: Java heap space
  This does not necessarily imply a memory leak - as it could be due to lesser space configured for the heap. Otherwise, in a long-lived application it could be due to unintentionally references being mentioned to heap objects (memory leak). Even the APIs that are called by the application could be holding references to objects that are unwarranted for. Also, in applications that make excessive use of finalizers, sometimes the objects are queued into a finalization queue. When such an application creates higher priority threads and that leads to more and more objects in the finalizaton queue, It can cause an Out-of-Memory. 
• Exception in thread "main": java.lang.OutOfMemoryError: PermGen space
  If there are many classes and methods loaded or if there are very many string literals created, especially through the use of intern() (From JDK 7, interned strings are no longer part of the PermGen) - then this type of error occurs. When this kind of error occurs, the text ClassLoader.defineClass might appear near the top of the stack trace that is printed.
• Exception in thread "main": java.lang.OutOfMemoryError: Requested array size exceeds VM limit
  This again happens when the requested array size is greater than the available heap size. It may usually  occur due to programmatic errors during runtime, if an incredibly large value is requested for an array size.
• Exception in thread "main": java.lang.OutOfMemoryError: request <s> bytes for <r>. Out of swap space?
  It may usually be the root cause for a memory leak. It happens when either the Operating System does not have sufficient swap space or when Another Process hogs all the available memory resources on the system. In simple terms, it was unable to provide the request space from heap due to exhaustion of space. The message indicates the size 's' (in bytes) of the request that failed and the reason 'r' for the memory request. In most cases the <r> part of the message is the name of a source module reporting the allocation failure, although in some cases it indicates a reason.
• Exception in thread "main": java.lang.OutOfMemoryError: <reason> <stack trace> (Native method)
  This indicates that a Native method has met with an allocation failure. The root cause was that the error occurred in JNI rather than in the code executing inside the JVM.
  When the native code does not check for memory allocation errors, then the application crashes instead of going out of memory.

 

Definition of Memory Leak

"Think of memory leakage as a disease and the OutOfMemoryError as a symptom. But not all OutOfMemoryErrors imply memory leaks, and not all memory leaks manifest themselves as OutOfMemoryErrors. "

In Computer Science, a memory leak is a type of resource leak that occurs when a computer program incorrectly manages memory allocations in such a way that memory which is no longer needed is not released. In Object-Oriented Programming, a memory leak may happen when an object is stored in memory but cannot be accessed by the running code.

Common Definitions of Memory Leak in Java:  

A Memory Leak occurs when Object References that are no Longer needed are Unnecessarily Maintained.

Memory Leak in Java is a situation where some objects are not used by Application any more, but GC fails to Recognize them as Unused. 

A Memory Leak appears when an object is no longer used in the program but is still referenced somewhere at a location that is not reachable. Thus, the garbage collector cannot delete it. The memory space used for this object will not be released and the total memory used for the program will grow. This will degrade performances over time and the JVM may run out of memory.

In a way, Memory Leak would occur when No Memory can be Allocated in the Tenured Space.

Some of the Most Common Causes of Memory Leaks are:

1. ThreadLocal Variables
2. Circular and Complex Bi-Directional References
3. JNI Memory Leaks
4. Static Fields that are Mutable (Most Common)

I recommend the usage of Visual VM bundled with the JDK to start Debugging your Memory Leak Issues.

Common Debugging of Memory Leaks

1. NetBeans Profiler
2. Using the jhat Utility
3. Creating a Heap Dump
4. Obtaining a Heap Histogram on a Running Process
5. Obtaining a Heap Histogram at OutOfMemoryError
6. Monitoring the Number of Objects Pending Finalization
7. Third Party Memory Debuggers

The common strategies or steps for going about debugging memory leak issues include:

• Identify Symptoms
• Enable Verbose Garbage Collection
• Enable Profiling
• Analyze the Trace

Wishing Happy Times, Fixing Java Memory Issues!

 

Java Object Memory Layout/Anatomy [Deep Size of an Object]

Even to the mind of the most experienced Java Developer, the way JVM organises or allocates memory for an Object may not come intuitively. This also includes activities like measuring the size of an Object.

When we use 'new' to instantiate an Object, there is much more data size allocated than is required to just hold the value. For example, if you choose to create java.lang.Integer, the actual int value will be only 1:4 parts of the Object. The remaining is used to hold the metadata - The metadata includes the following:

1. Class: A pointer to the type of the class. In our case, a pointer to the java.lang.Integer class.

2. Flags: A collection of flags that describe the state of the Object, including the hash code of the Object and the shape of the Object (If it is an Array or not)

3. Lock: The synchronization information of the Object, including if it is synchronized currently.

The Object metadata is followed by the actual Object data. In this specific case, an int value.

Example layout of a java.lang.Integer object for a 32-bit Java process

[Reference: http://www.ibm.com/developerworks/library/j-codetoheap/]

I tried to explore all possibilities to measure the Object Memory usage as per the above understanding of the Object Allocation. There are many tools, blogs and write-ups on the internet, but all of them provide approximation, which may not be reliably usable. Also, the values which may be returned by each of these will not concur across these tools. 

The only tool that I came across that seems to be reliable and has the source code public is the Java Agent for Memory Measurements [https://github.com/jbellis/jamm]. The author also seems to be the committer for the Cassandra project, and it is being used in Cassandra - Hence, it is a safe bet and should be close to as exact as possible. I went through the source and the logic seems to concur with the memory layout above, Also, since it is being used in very important (and widely used) projects, [i skipped trying it out myself]. In all, this should be your safest [reliablity and performance] bet if you want to use it to introduce custom cache measurements, custom object profiling, server profiling or simply memory profiling tools.

[I plan to write my own tool to measure Object Sizes in Java as reliably as possible - using the above understanding of Java Memory Allocation . Keep checking this Blog to Download the Source and Binary]