Starting Search with Apache Lucene 5.3.x/5.4.x

Before we delve into Apache Lucene, the following are the most important terms that you need to be familiar with. This will also help you clarify a few terms before getting into 'search' or 'information retrieval':

Let us get ahead with Apache Lucene 5.3.x/5.4.y; The most important aspects of Lucene is mentioned under each of the headings.

Fig. 0 : Lucene - Search/Process Architecture (Usage)

1. Lucene Introduction (Usage)

• Apache Lucene is a high-performance, full-featured text search engine library written entirely in Java. It is a technology suitable for nearly any application that requires full-text search, especially cross-platform.
• Apache Lucene is an open source project available for free download
• Scalable, High-Performance Indexing
• Powerful, Accurate and Efficient Search Algorithms
• Cross-Platform Solution
 

2. LuceneTerms (Concepts)

• Inverted Index
  Inverted Index is used to get traverse from the string or search term to the document id's or locations of these terms. If we were to visualize this in terms of an 'index' - it would be 'inverted', as we would be using the term as a handle to retrieve 'id' or 'locations' - reverse of the popular usage of an index.
  
  
• Index
  Index is an handle (information) that can be used to get further related information from a file, database or any other source of data. Usually, Index is also accompanied by compression, check-sum, hash or location of the remaining data. Index contains multiple Documents.
  
  
• Document
  Document is a collection of Fields and the Values against each of the Fields. It is more like saying "Employee Name" - "Sumith Puri" | "Employee Desingation" - "Software Architect" | "Employee Age" - "33" | "Employee ID" - "067X" forms a document. The Lucene Indexing Process adds multiple documents to an Index. The entire set of Documents is called the Corpus.
   
• Field
  Field contains Terms and are simply 'Sets of Tokens' of information. The Lucene Indexing process take care to Identify (or Process) Fields and Index them. Fields belong to a Document always.
  
  
• Terms
  Terms are nothing but a 'Token' or 'String' of Information. This 'Term' is the smallest piece of Information that will be Indexed to form the Inverted Index. Set of Distince Terms is called the Vocabulary.
  
  
• String
  String is simply a 'Token' or the English Language String.
  
  
• Segment
  Segment is a fragmented or chunked part of the entire Index, for better storage and faster retrieval.

3. Lucene Segment (Indexing)

Each segment index maintains the following:
 
Field Names: This contains the set of field names used in the index.

Stored Field Values: This contains, for each document, a list of attribute-value pairs, where the attributes are field names. These are used to store auxiliary information about the document, such as its title, url, or an identifier to access a database. The set of stored fields are what is returned for each hit when searching. This is keyed by document number.   
 
Term Dictionary: A dictionary containing all of the terms used in all of the indexed fields of all of the documents. The dictionary also contains the number of documents which contain the term, and pointers to the term's frequency and proximity data.
 
Term Frequency Data: For each term in the dictionary, the numbers of all the documents that contain that term, and the frequency of the term in that document, unless frequencies are omitted (IndexOptions.DOCS_ONLY)
 
Term Proximity Data: For each term in the dictionary, the positions that the term occurs in each document. Note that this will not exist if all fields in all documents omit position data.
 
Normalization Factors: For each field in each document, a value is stored that is multiplied into the score for hits on that field.
 

Term Vectors: For each field in each document, the term vector (sometimes called document vector) may be stored. A term vector consists of term text and term frequency. To add Term Vectors to your index see the Field constructors

Deleted Documents: An optional file indicating which documents are deleted. 



4. Lucene Internals (Architecture)

    Fig. 1: Lucene Architectural Layers [Non-Copyrighted Image]

Enforce Size Constraints in Infinispan - Java Instrumentation API!

Infinispan does not provide size constraints out of the box, just like almost all of the other standard caching frameworks. You may use the Java Instrumentation API as follows to enforce cache size constraints for your product or project that uses Infinispan.

1. Introduce Instrumentation API in your code
a. Develop a class that has the following method in it. The VM will automatically insert an implementation of the Instrumentation API - once it encounters this method.

 public class ServerInstrumentation {  
      private Instrumentation instrumentation = null;   
      private static final Logger LOGGER = Logger.getLogger(ServerInstrumentation.class);  
      private static ServerInstrumentation serverInstrumentation;  

      public static void premain(String args, Instrumentation pInstrumentation) {  
           serverInstrumentation = new ServerInstrumentation();  
           serverInstrumentation.instrumentation=pInstrumentation;  
      }  
      public static Instrumentation getInstance() {  
           return serverInstrumentation.instrumentation;  
      }  
}

b. You may choose to add the following EJB (optionally) to make this object available throughout your Java EE application. Re-Implement the methods that you need to use from the Instrumentation API. For example, below we have implemented the sizeOf() which actually internally invokes Instrumentation.getObjectSize().

 @Remote  
 public interface ServerInstrumentationInterface {  
      public long sizeOf(Object toBeSized);  
 }  

 @Singleton  
 @Startup  
 public class ServerInstrumentationEJB implements ServerInstrumentationInterface {  
      private static Instrumentation instrumentation;  
      {  
           ClassLoader classLoader=this.getClass().getClassLoader().getParent();  
           try {  
                Class clazz = classLoader.loadClass(ServerInstrumentation.class.getName());  
                instrumentation=(Instrumentation)clazz.getDeclaredMethod("getInstance").invoke(null);                   
           } catch (Exception e) {  
                System.out.println("FFFFFF");  
           }  
      }  
      public long sizeOf(Object toBeSized) {  
           return instrumentation.getObjectSize(toBeSized);  
      }  
 }  

Package this code as 'agent.jar' with the following contents in MANIFEST.MF

 Premain-Class: ServerInstrumentation  
 Agent-Class: ServerInstrumentation  

c. You should place this in the normal deployment location (standalone\deployments) of Wildfly and also mention it as the -javaagent during server startup.The following contents should be in Wildfly standalone.bat

 "%JAVA%" %JAVA_OPTS% "-javaagent:%JBOSS_HOME%\standalone\deployments\agent.jar" ^  
  "-Dorg.jboss.boot.log.file=%JBOSS_LOG_DIR%\server.log" ^  
  "-Dlogging.configuration=file:%JBOSS_CONFIG_DIR%/logging.properties" ^  
   -jar "%JBOSS_HOME%\jboss-modules.jar" ^  
   -mp "%JBOSS_MODULEPATH%" ^  
    org.jboss.as.standalone ^  
   "-Djboss.home.dir=%JBOSS_HOME%" ^  
    %SERVER_OPTS%  

At the end of this, the Instrumentation methods are available throughout your Java EE application for all instrumentation purposes. The EJB will act as a facade over this functionality. Make sure that the following settings exist in your standalone.conf.bat

set "JAVA_OPTS=%JAVA_OPTS% -Djava.net.preferIPv4Stack=true -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Xbootclasspath/p:d:\wildfly\modules\system\layers\base\org\jboss\log4j\logmanager\main\log4j-jboss-logmanager-1.1.0.Final.jar;d:\wildfly\modules\system\layers\base\org\jboss\logmanager\main\jboss-logmanager-1.5.2.Final.jar" 
set "JAVA_OPTS=%JAVA_OPTS% -Djboss.modules.system.pkgs=org.jboss.byteman,org.jboss.logmanager"

2. Use this in your Actual Application (Client)
a. The lookup code is provided below to perform methods using Instrumentation, such as 'Object Size'. This may be used for multiple purposes in your Java EE products or solutions, apart form caching constraints.

 // let's do the lookup           
ServerInstrumentationInterface cacheInstrumentationService = (ServerInstrumentationInterface)   
               initialContext.lookup("java:global/agent/" + ServerInstrumentationEJB.class.getSimpleName());  
cacheInstrumentationService.sizeOf(new String("Java and Infinispan Caching Innovation")));  

b. The following method can be implemented in a different class (CacheInstrumentation.java) to perform the following function everytime you add an element to the cache, to check whether the Allocated Size is exceeded. This will have the above EJB Reference/Instance encapsulated in it.

 public boolean isEntityCacheLimitExceeded(Cache cache) {  
           boolean exceeded=false;  
           long currSize = cacheInstrumentationService.sizeOf(cache);  
           long allocSize = 10*1000*1024;  // 10 MB = 10 * 1000 * 1024 bytes
           if(currSize >= allocSize) {  
                exceeded=true;  
           }  
           return exceeded;  
}  

3. Use ConfigurationBuilder to Introduce Constraints
a. The following method will be used to setConfiguration() which will help to notify Infinispan that the configuration has changed, so that it can perform eviction dynamically.

 public class CacheManagerUtilities {  
      public static Configuration setConfiguration(int maxNumberofEntries) {  
           return new ConfigurationBuilder().eviction().maxEntries(maxNumberofEntries)  
                     .strategy(EvictionStrategy.valueOf("LIRS")).expiration()  
                     .lifespan(2000)  
                     .wakeUpInterval(500)  
                     .maxIdle(1000).build();  
      }  
 }  

4. Request/Enforce Eviction using EvictionManager [defineConfiguration()]

a. Everytime you put/add an element in your cache, you can use the sizeOf() from the EJB to check if the size constraints are exceeded. If they are, then dyanmically change the configuration to make the max-entries limit to the currently existing max-entries. We will now rely on Infinispan's Eviction Algorithms to kick-in and do its duty.

 public void put(K key, V value) {            
           while (true) {  
                if (!cacheInstrumentation.isEntityCacheLimitExceeded(cache)) {  
                     cache.put(key, value);  
                     break;  
                } else {  
                     // set the configuration to the infinispan cache programmatically  
                     Configuration configuration = CacheManagerUtilities.setConfiguration(cache.size());  
                     cache.getCacheManager().defineConfiguration(cache.getName(), configuration);  
                    
                     // start eviction manually  
                     EvictionManager evictionManager = cache.getAdvancedCache().getEvictionManager();  
                     evictionManager.processEviction();  
                     
                     // set the configuration to the infinispan cache programmatically - large number such as 9999  
                     configuration = CacheManagerUtilities.setConfiguration(9999);  
                     cache.getCacheManager().defineConfiguration(cache.getName(), configuration);  
                }  
           }  
}

The above usually performs three things:

1. Expire all elements that have reached expiration time (Data Container)
2. Expire all elements that have reached expiration time (Cache Store)
2. Remove all elements as per the eviction policy - Prunes to max-entries

You may also test the above using Arquillian to try out whether the eviction works by creating data within test cases that adhere to LIRS, LRU and Size Constraints.

Note that the above code will also make sure that only 'Serializable' objects are allowed to be Cached. Suitable for all Java/Java EE products and solutions that need to Enforce Size as a Constraint mechanism for Caching. Happy Caching!

Spring, Hibernate, EhCache Recipe

A simple scenario explaining the usage and performance, when using EhCache along with Hibernate in a Spring environment. The performance results are taken using mySQL as the database.

[Download Sample Code] Please be informed that the size is about 10MB; as i have provided all the dependencies.

In this example, I need to retrieve close to 5,000 records in a single fetch and then cache this information. As usual, setup Spring contexts in your Spring configuration file. I have just one class, which is a Hibernate DAO, HibernateDoctorDAO.java. The dependency injection hierarchy is dataSource > sessionFactory > hibernateTemplate. hibernateTemplate is then injected into the HibernateDoctorDAO.java at runtime by the Spring Framework.

The implementations of each of these are:

dataSource > org.apache.commons.dbcp.BasicDataSource

sessionFactory > org.springframework.orm.hibernate3.LocalSessionFactoryBean

hibernateTemplate > org.springframework.orm.hibernate3.HibernateTemplate

The results clearly shows the difference in performance with EhCache enabled, even in this simple example:
 

PERFORMANCE COMPARISON (In Seconds)
==============================
QUERY FETCH TIME (INITIAL): 0.599
QUERY FETCH TIME (HIBERNATE CACHE): 0.212
QUERY FETCH TIME (2ND LEVEL CACHE): 0.091

Version Reference > Spring-2.5, Hibernate-3.0, EhCache-1.4, mySQL-5.0

 

Instructions: Unzip the file and start by creating the database. Any mySQL database that is compatible with the MySQL Connector 5.0.8 is fine for this example. You will have to create the database data_explosion (USERNAME: root, PASSWORD: architect.2012) and the table DOCTOR_TABLE. You can use the provided doctor.sql as reference. Then run the java standalone com.sumsoft.spring.orm.hibernate.dataload.JDBCDataLoader to create and load a large number of arbitary records into the DOCTOR_TABLE. You can ignore or remove every invocation that is to PATIENT_TABLE, as this is only additional for this example. Then use spring_hibernate_cache.bat to run the application. Though not tested on UNIX, you may write a script similar to the batch file to run this application.