How to handle Stop Words in Hibernate Search 5.5.2 / Apache Lucene 5.4.x?

The Stop Words like ["a", "an", "and", "are", "as", "at", "be", "but", "by", "for", "if", "in", "into", "is", "it", "no", "not", "of", "on", "or", "such", "that", "the", "their", "then", "there", "these", "they", "this", "to", "was", "will", "with"] and the existence of them in terms or database or files that are to be indexed/searched by lucene can lead to any of the following:

1. Stop Words being Ignored/Filtered during the Lucene Indexing Process

2. Stop Words being Ignored/Filtered during the Lucene Querying Process

3. No Result for Queries that Include, Start With or End With any Stop Word

The way to solve this problem or to handle them during both indexing and searching process is as follows. The method explained here is specially suitable if you are using Hibernate Search 5.5.2 which in turn is using Apache Lucene 5.3.x/5.4.x

1. Define your Custom Analyzer, Adapted from the Standard Analyzer

You need to include only the two filters - 'LowerCaseFilterFactory' and 'StandardFilterFactory' as part of the Tokenizer definition. The filter factory that we have not included here is the 'StopFilter'. This allows Stop Words to be considered as other normal English Words and they are indexed.

@Entity 

@Indexed 

@Table(name="table_name", catalog="catalog_name") 

@AnalyzerDef(name = "fedexTextAnalyzer",
   tokenizer = @TokenizerDef(factory = StandardTokenizerFactory.class), 

   filters = {
     @TokenFilterDef(factory = LowerCaseFilterFactory.class),
     @TokenFilterDef(factory = StandardFilterFactory.class)
})

2. Mark the Field with Relevant Annotations (@Analyzer on @Field)

Along with the @Field Annotation on every Entity's or Table's Column Field, declare the Analyzer that we have defined above.

@Column(name="fedex_cs_product_name", nullable=false, length=100)

@Field(index=Index.YES,analyze=Analyze.YES, store=Store.NO, analyzer=@Analyzer(definition = "fedexTextAnalyzer"))

public String getFedexCsItemName() {

   return this.fedexCsItemName;

}

3. Use WhitespaceAnalyzer to Query so that Stop Words are 'Processed' by Default

Although the official documentation says that if we use 'StandardAnalyzer' by passing in the argument for Stop Words as CharArraySet.EMPTY_SET I found that the Query was still not able to retrreve any result. On Analysis with Luke, I found that for Queries such as 'Computer Science Books for Beginners', the 'for' was being ignored. Strange! I replaced it with WhitespaceAnalyzer, I found that it works for all 'Stop Words' and all 'Cases'.

 
I have found that the above is the best/minimal way to fix this issue. Also, our QA has verified that it works for all 'Stop Word' cases! Hope this helps you.

Starting Search with Apache Lucene 5.3.x/5.4.x

Before we delve into Apache Lucene, the following are the most important terms that you need to be familiar with. This will also help you clarify a few terms before getting into 'search' or 'information retrieval':

Let us get ahead with Apache Lucene 5.3.x/5.4.y; The most important aspects of Lucene is mentioned under each of the headings.

Fig. 0 : Lucene - Search/Process Architecture (Usage)

1. Lucene Introduction (Usage)

• Apache Lucene is a high-performance, full-featured text search engine library written entirely in Java. It is a technology suitable for nearly any application that requires full-text search, especially cross-platform.
• Apache Lucene is an open source project available for free download
• Scalable, High-Performance Indexing
• Powerful, Accurate and Efficient Search Algorithms
• Cross-Platform Solution
 

2. LuceneTerms (Concepts)

• Inverted Index
  Inverted Index is used to get traverse from the string or search term to the document id's or locations of these terms. If we were to visualize this in terms of an 'index' - it would be 'inverted', as we would be using the term as a handle to retrieve 'id' or 'locations' - reverse of the popular usage of an index.
  
  
• Index
  Index is an handle (information) that can be used to get further related information from a file, database or any other source of data. Usually, Index is also accompanied by compression, check-sum, hash or location of the remaining data. Index contains multiple Documents.
  
  
• Document
  Document is a collection of Fields and the Values against each of the Fields. It is more like saying "Employee Name" - "Sumith Puri" | "Employee Desingation" - "Software Architect" | "Employee Age" - "33" | "Employee ID" - "067X" forms a document. The Lucene Indexing Process adds multiple documents to an Index. The entire set of Documents is called the Corpus.
   
• Field
  Field contains Terms and are simply 'Sets of Tokens' of information. The Lucene Indexing process take care to Identify (or Process) Fields and Index them. Fields belong to a Document always.
  
  
• Terms
  Terms are nothing but a 'Token' or 'String' of Information. This 'Term' is the smallest piece of Information that will be Indexed to form the Inverted Index. Set of Distince Terms is called the Vocabulary.
  
  
• String
  String is simply a 'Token' or the English Language String.
  
  
• Segment
  Segment is a fragmented or chunked part of the entire Index, for better storage and faster retrieval.

3. Lucene Segment (Indexing)

Each segment index maintains the following:
 
Field Names: This contains the set of field names used in the index.

Stored Field Values: This contains, for each document, a list of attribute-value pairs, where the attributes are field names. These are used to store auxiliary information about the document, such as its title, url, or an identifier to access a database. The set of stored fields are what is returned for each hit when searching. This is keyed by document number.   
 
Term Dictionary: A dictionary containing all of the terms used in all of the indexed fields of all of the documents. The dictionary also contains the number of documents which contain the term, and pointers to the term's frequency and proximity data.
 
Term Frequency Data: For each term in the dictionary, the numbers of all the documents that contain that term, and the frequency of the term in that document, unless frequencies are omitted (IndexOptions.DOCS_ONLY)
 
Term Proximity Data: For each term in the dictionary, the positions that the term occurs in each document. Note that this will not exist if all fields in all documents omit position data.
 
Normalization Factors: For each field in each document, a value is stored that is multiplied into the score for hits on that field.
 

Term Vectors: For each field in each document, the term vector (sometimes called document vector) may be stored. A term vector consists of term text and term frequency. To add Term Vectors to your index see the Field constructors

Deleted Documents: An optional file indicating which documents are deleted. 



4. Lucene Internals (Architecture)

    Fig. 1: Lucene Architectural Layers [Non-Copyrighted Image]

Resolving BCEL Issue on JiBX 1.2.5 with JDK 1.8 on Apache Maven [Error loading class java.lang.CharSequence]

So, I spent some time on resolving this issue with a combination of the following:

1. JiBX 1.2.5 / JiBX 1.2.6

2. JDK 1.8

During the jibx-maven-plugin:bind phase of the Maven Build, you may get an error when you upgrade from JDK 1.x to JDK 1.8;The issue will manifest in the following most common form/error:

 

Failed to execute goal org.jibx:jibx-maven-plugin:1.2.5:bind (bind-compile) on project common: Error loading class java.lang.CharSequence: Error reading path java/lang/CharSequence.class for class java.lang.CharSequence

You can modify your pom.xml, to make these changes to fix this error.

1. Include the Repository for BCEL 6.0

2. Check the version of JiBX [Choose any of 1.2.5 or 1.2.6]

3. Exclude default 'BCEL' from the Build

4. See the modifications for the plugin 'jibx-maven-plugin'

5. Voila!

[Off-Beat] Software Architect's Guide to Knocking-In a Cricket Bat!

So, Start of the Year 2016. Happy New Year to All Readers!  I start off with an off-beat article on Knocking-In a Cricket Bat - Software Architect's Version.

I have represented U-13 District, School, College, Club and Corporate as a Cricket Player. I am an Opening Batsman, Leg-Spinner and mostly a Mid-On/Mid-Off Fielder. But, I have been away from any 'representative/professional' form of cricket since 2003. In 2016, I have decided to get back to my old passion - at least to a certain measure.

In that regard, I bought a new SG Kashmir Economy Cricket Kit and got an (exchanged) SG Kashmir Willow Cobra Gold Cricket Bat. So, I went through multiple YouTube Videos on Knocking-In (Primarily from English and US/Indian Cricket Merchants). Also, I read a few articles here and there on Knocking-In. Seeing that there are contrasting opinions on various aspects of Knocking-In, I have come up with my own version of Knocking-In. Hope it helps you as well.

I will use the outcomes of this, for my next cricket bat (sometime later this year) - The current choice being SS Ton (Any) English Willow Cricket Bat.

Requirements

SG Kashmir Willow (English Willow) Cricket Bat 

SG Toe Guard Kit - Toe Guard, Sand Paper, FeviBond

SG Bat Mallet
Sharp Scissors (Big Size Preferred)

SG Ball Mallet

SG Bat Face Tape

SS Bat Care Kit - Oil, Toe Guard, Anti-Scuff Tape Roll, Bat Face Tape

SG/SS Old Cricket Ball (Optional)

SS Swinger New Ball (SG Club Preferred)

SS True Test New Ball (SG Club Preferred) 
SG Bat Grip Cone
SG Bat Grip

A Cricket Companion (Preferred Medium Pacer)

Process

1.Unpack your New Cricket Bat. Apply a Coat of the SS Bat Oil to every exposed part of the willow. You may use a soft tissue paper here to apply the oil. Something like the one given in restaurants or used as face wipe. Here, every exposed part of the willow (front, top and back) should be applied with at least 1 or 2 coats of oil. You can choose to apply length-wise; so that the oil spreads well and evenly. That too alternatively, first starting from top to bottom (on all sides) and then bottom to top. This is to make sure that nowhere the oil accumulates in a 'larger/uneven' quantity. 

2. Leave the bat in a horizontal resting position with the face upright. You may also choose to keep this under 'sunlight'. The total rest period can be anywhere from 12-18 hours.

3. Now, apply another coat of Oil in the same way as (1) and then follow what you did in (2), this time you may choose to keep it for 18-24 hours.

4.Now Apply the Toe Guard. Apply lots of 'FeviBond' to the base edge of the bat. Also, apply lots of FeviBond to Toe Guard. You may leave it for 15 minutes. Preferred that you use an SG Toe Guard on a SG Bat (Seeing the Brand Name creates some 'Professionalism' and 'Motivation'). Now, apply one more coat of 'FeviBond' to each of the bottom edge of bat and the Toe Guard. Leave it for 2 minutes only - Then, press the Toe Guard against the bat edge so that it sticks together. Make sure that the centre of the Toe Guard is pressed against the edge. Now hit with the Bat Mallet so that the Toe Guard is put in place and bonds perfectly. Also, since there will be excess Toe Guard on both length and breadth - plaster it (bat tape or the useful anti-scuff tape) upwards to the edges and the face of the bat. This will make sure that the guard and the edge bond well together. Leave this overnight. In the morning, remove the plaster and you would find that the toe guard and the edge have bonded really well. Now use a sharp scissors to cut off the excess Toe Guard from all around the edge. [Now, you may use a sand paper - very mildly all across the edges, so that the edges of the Toe Guard and the Bat become uniform] At the end of this bat, you may be very satisfied with the look of your bat. Yay! Toe Guard.

5. Time for Further Action - Now use the Bat Mallet. Start by using the handle of the bat mallet to smoothen (roll) the edges of the bat. The Edges here mean the actual sharp edges of the face of the bat. You can apply a very 'heavy' pressure with your bare hands to smoothen and flatten the edges. This is to ensure that the edges don't chip when you either knock-in or play. After doing this for around 15-20 minutes, You can start hitting the edges at 45 Degree Angle using the face of the Bat Mallet. Hit it along the entire length of each edge on the face of the bat. You may do this for the first 20 minutes. [40 Minutes]

6. Now, leaving the bottom Six Inches, Hit the Bat Mallet at 90 Degree right on the face of the bat. You should start off very softly so that you do not damage the bat. Hit every part of the face where a ball can hit. Once through with about 5 to 10 minutes, start by hitting harder at 90 Degrees. This should be the next 20 minutes [20 Minutes]. Leave your bat to rest, somewhere where the willow is exposed to direct sunlight. I did this to make sure all excess oil is dried.  

7. Iterative Action - Again start off with the edges as mentioned in (5), You should aim to spend only 20 minutes on the edges. Then, follow it up with (6) for 20 minutes. In this setting, you should also start off with the bottom six inches. Do not ever hit at 90 Degrees on the bottom six inches. Hit at 45 Degrees in a way that the ball would bounce or roll off the bat. Start by hitting softly and then harder. Try to cover every part of the front face of the bottom six inches.  Spend 20 minutes on this.  [Iterate (7) 2 or 3 times - Next, Apply Oil as mentioned in (1) and leave for 12-18 hours. Apply only 1 coat.]

[At this point, we are halfway through. You may notice some cracks on the surface, especially along the grains. All of internet says that this is a pointer to a good quality bat that is being seasoned correctly!]

8. Iterative Action - Let us start with Ball Mallet. This is my adaptation of the Knocking-In Process. Do this only because I told you. I may not find any supporter for this - neither a cricket merchant nor another player. Hit with the seam upright in the same way as (6) and (7); each for 30 minutes. Make sure that you also use the handle of the ball mallet in the same way as (5) for about 5-10 minutes for the edges of the cricket bat. [Iterate (8) 3 or 4 times - Next, Apply Oil as mentioned in (1) and leave for 24-48 hours. Apply only 1 coat]
[At this point, we are almost through. You may notice that the cracks on the surface are more prominent, especially along the grains! Also, you would have spend around 6-7 hours in total with the bat mallet and the ball mallet.]

9. Now, get ahead with your Bowling Companion. Preferably a medium pacer to throw or bowl with New Balls to you. Make sure you show the full face of the bat with shots in the V. This is atleast for the first net session. Come back home and use the bat mallet for 20-30 minutes in the same way, followed by ball mallet. Oil once more and rest willow horizontally with face up and facing the sunlight for 24 hours. Now, apply Anti Scuff Tape on both the edges followed by a Bat Tape on the full face of the cricket bat. Yes, I also used the SG Bat Grip Cone to put an additional grip to my cricket bat. This is for better shock absorption and better control.

Next time on wards, you may start off with a few horizontal bat shots with a cut or pull. After this, use it freely and you may also use it for a match. Note that when you hit with the bat mallet and ball mallet, hit as hard you can, progressively - because if you won't do it, the ball will still do it later and may even cause damage. Also, you will be able to identify the sweet-spot(s) of the bat during this time - Note them and re-affirm them. Also, you will notice the sound of the willow change as you progress through this process. Though some say that finger nails marks will not be processed once the willow is ready, I can only confirm that the intensity or marks of finger nails prints reduce to a good extent - but they still do appear. Also, Some say that the seam marks of the ball stop appearing as the willow gets seasoned - This I can confirm from my own experience. If deep finger nails are still appearing or seam marks are appearing on the cricket bat, then still your bat needs more work (more knocking-in).

This is my finished SG Kashmir Willow Cobra Gold Cricket Bat. 

 

 


Okay, Now for the motivators for this start of cricket term from 2016. Enjoy!

I will mention as comment to this post, all throughout, the scores that are note-worthy (30+) for organized friendly or representative matches, all scored from the bat seasoned using the above process! All the Best to You!

Also, If you are in Bangalore and want to Join  the concept or the team started by me - Bangalore Software Engineers XI, You are most welcome - http://www.meetup.com/bangalore_software_engineers_xi/ This is a Meetup Group with an eventual aim to be a Registered Society/Club/Organization in India and to participate in Club Cricket in Bangalore. We start off with weekend Net Practice Session, Friendly Matches (2-Day/T20/ODI) against other local Corporate or Club Team. The first preference is for folks who have already played earlier for any of State, Zone, District, University, School, College, Club or Corporate!