Stop SOLR converting "plc" to "PLK"?

mpicker · November 17, 2015, 1:09am

Hi,

Is there a way to get SOLR (Pro) to stop converting the search term “plc” to “PLK”? I’m using both 1.2.11 and 1.3.2 of SOLR Pro.

We have an index with 3 records that should be returned for the search term “plc” (PLC = programmable logic controller) but SOLR is returning these 3 records plus one record relating to Bricklaying/Blocklaying (as the number 1 result). It appears to be a result of phonetic fields. One of the fields being searched is a metaKeyword field and it seems to be this is one of the main (but not only) reasons the bricklaying course is being returned and also at the top of the list. I’ve come to this conclusion by removing the phonetic option for this field in the index and now bricklaying is being returned as the fourth result. The keyword field has the same boost value as all other fields.

The keyword field for this bricklaying course only contains “block,BLOCKLAYING,brick,BRICKLAYING” so I’m not sure what “PLK” has to do with any of these words.

Just as an experiment I tried adding “plc” to both the Synonyms and Protected Words list to see if that had any impact but it did nothing.

I’m hesitant to remove the all phonetic fields as I see them as useful but at the same time returning a bricklaying course as the number one result for “plc” isn’t. Any suggestions on the best way to deal with this?

I’ve included the debug output from SOLR below.

Regards
Mark

<str name="rawquerystring">(plc) AND (typename:wiCourseInfo) AND (fcsp_sitename:test_wi_internet_7) AND fcsp_benablesearch:true</str>
<str name="querystring">(plc) AND (typename:wiCourseInfo) AND (fcsp_sitename:test_wi_internet_7) AND fcsp_benablesearch:true</str>
<str name="parsedquery">+(+DisjunctionMaxQuery((bitsmart_string_stored:plc | bitschooltrainee_string_stored:plc | bittwcconnectedclassroom_string_notstored:plc | bittwcanywhere_string_stored:plc | bittraineeship_string_stored:plc | careeropportunities_phonetic_notstored:PLK | bitfeehelp_string_notstored:plc | bittwcblock_string_notstored:plc | bitmainstream_string_notstored:plc | fcsp_rulecontent:plc | bittvet_string_stored:plc | specialisation_phonetic_notstored:PLK | bitcomm_string_stored:plc | learningproductidentifier_text_stored:plc | specialisation_text_stored:plc | coursedescription_phonetic_stored:PLK | bittwcmobile_string_notstored:plc | careerpathways_text_notstored:plc | coursedescription_text_stored:plc | coursename_string_notstored:plc | careeropportunities_text_notstored:plc | coursename_phonetic_notstored:PLK | bittwcworkshop_string_notstored:plc | industries_text_stored:plc | bitisgovsubsidcourse_string_stored:plc | bittwconline_string_notstored:plc | bittwcposted_string_notstored:plc | qualification_phonetic_notstored:PLK | metakeywords_phonetic_notstored:PLK | fcsp_rulecontent_phonetic:PLK | qualification_text_stored:plc | bittafeplus_string_notstored:plc | bitvetfeehelp_string_stored:plc | trainingpackage_text_notstored:plc | nationalcode_text_stored:plc | bitenrolmentsallowed_string_notstored:plc | bitapprenticeship_string_stored:plc | locations_text_stored:plc | bittwc_string_stored:plc | metakeywords_text_notstored:plc | objectid:plc | coursename_text_stored:plc | careerpathways_phonetic_notstored:PLK | bittwcwebconference_string_notstored:plc)) +typename:wiCourseInfo +MultiPhraseQuery(fcsp_sitename:"(test_wi_internet_7 test) wi (internet testwiinternet) 7") +fcsp_benablesearch:true)</str>
<str name="parsedquery_toString">+(+(bitsmart_string_stored:plc | bitschooltrainee_string_stored:plc | bittwcconnectedclassroom_string_notstored:plc | bittwcanywhere_string_stored:plc | bittraineeship_string_stored:plc | careeropportunities_phonetic_notstored:PLK | bitfeehelp_string_notstored:plc | bittwcblock_string_notstored:plc | bitmainstream_string_notstored:plc | fcsp_rulecontent:plc | bittvet_string_stored:plc | specialisation_phonetic_notstored:PLK | bitcomm_string_stored:plc | learningproductidentifier_text_stored:plc | specialisation_text_stored:plc | coursedescription_phonetic_stored:PLK | bittwcmobile_string_notstored:plc | careerpathways_text_notstored:plc | coursedescription_text_stored:plc | coursename_string_notstored:plc | careeropportunities_text_notstored:plc | coursename_phonetic_notstored:PLK | bittwcworkshop_string_notstored:plc | industries_text_stored:plc | bitisgovsubsidcourse_string_stored:plc | bittwconline_string_notstored:plc | bittwcposted_string_notstored:plc | qualification_phonetic_notstored:PLK | metakeywords_phonetic_notstored:PLK | fcsp_rulecontent_phonetic:PLK | qualification_text_stored:plc | bittafeplus_string_notstored:plc | bitvetfeehelp_string_stored:plc | trainingpackage_text_notstored:plc | nationalcode_text_stored:plc | bitenrolmentsallowed_string_notstored:plc | bitapprenticeship_string_stored:plc | locations_text_stored:plc | bittwc_string_stored:plc | metakeywords_text_notstored:plc | objectid:plc | coursename_text_stored:plc | careerpathways_phonetic_notstored:PLK | bittwcwebconference_string_notstored:plc) +typename:wiCourseInfo +fcsp_sitename:"(test_wi_internet_7 test) wi (internet testwiinternet) 7" +fcsp_benablesearch:T)</str>
<lst name="explain">
	<str name="test_wi_internet_7_A4802AB3-0BB4-4B67-97BC8A8E7E2A960B"> 687.78577 = (MATCH) sum of: 336.3366 = (MATCH) max of: 143.42896 = (MATCH) weight(coursedescription_phonetic_stored:PLK in 447), product of: 0.64945203 = queryWeight(coursedescription_phonetic_stored:PLK), product of: 6.3752785 = idf(docFreq=2, maxDocs=648) 0.10187038 = queryNorm 220.84612 = (MATCH) fieldWeight(coursedescription_phonetic_stored:PLK in 447), product of: 1.7320508 = tf(termFreq(coursedescription_phonetic_stored:PLK)=3) 6.3752785 = idf(docFreq=2, maxDocs=648) 20.0 = fieldNorm(field=coursedescription_phonetic_stored, doc=447) 336.3366 = (MATCH) weight(metakeywords_phonetic_notstored:PLK in 447), product of: 0.597414 = queryWeight(metakeywords_phonetic_notstored:PLK), product of: 5.864453 = idf(docFreq=4, maxDocs=648) 0.10187038 = queryNorm 562.9875 = (MATCH) fieldWeight(metakeywords_phonetic_notstored:PLK in 447), product of: 1.0 = tf(termFreq(metakeywords_phonetic_notstored:PLK)=1) 5.864453 = idf(docFreq=4, maxDocs=648) 96.0 = fieldNorm(field=metakeywords_phonetic_notstored, doc=447) 0.36842465 = (MATCH) weight(typename:wiCourseInfo in 447), product of: 0.19373064 = queryWeight(typename:wiCourseInfo), product of: 1.9017366 = idf(docFreq=262, maxDocs=648) 0.10187038 = queryNorm 1.9017366 = (MATCH) fieldWeight(typename:wiCourseInfo in 447), product of: 1.0 = tf(termFreq(typename:wiCourseInfo)=1) 1.9017366 = idf(docFreq=262, maxDocs=648) 1.0 = fieldNorm(field=typename, doc=447) 350.97913 = (MATCH) weight(fcsp_sitename:"(test_wi_internet_7 test) wi (internet testwiinternet) 7" in 447), product of: 0.6102798 = queryWeight(fcsp_sitename:"(test_wi_internet_7 test) wi (internet testwiinternet) 7"), product of: 5.990748 = idf(fcsp_sitename: test_wi_internet_7=648 test=648 wi=648 internet=648 testwiinternet=648 7=648) 0.10187038 = queryNorm 575.1118 = (MATCH) fieldWeight(fcsp_sitename:"(test_wi_internet_7 test) wi (internet testwiinternet) 7" in 447), product of: 1.0 = tf(phraseFreq=1.0) 5.990748 = idf(fcsp_sitename: test_wi_internet_7=648 test=648 wi=648 internet=648 testwiinternet=648 7=648) 96.0 = fieldNorm(field=fcsp_sitename, doc=447) 0.10155645 = (MATCH) weight(fcsp_benablesearch:T in 447), product of: 0.10171329 = queryWeight(fcsp_benablesearch:T), product of: 0.99845797 = idf(docFreq=648, maxDocs=648) 0.10187038 = queryNorm 0.99845797 = (MATCH) fieldWeight(fcsp_benablesearch:T in 447), product of: 1.0 = tf(termFreq(fcsp_benablesearch:T)=1) 0.99845797 = idf(docFreq=648, maxDocs=648) 1.0 = fieldNorm(field=fcsp_benablesearch, doc=447) </str>
	<str name="test_wi_internet_7_74CC1E76-09E5-426F-89E1590FD2B6537C"> 532.65857 = (MATCH) sum of: 181.20946 = (MATCH) max of: 168.1683 = (MATCH) weight(metakeywords_phonetic_notstored:PLK in 421), product of: 0.597414 = queryWeight(metakeywords_phonetic_notstored:PLK), product of: 5.864453 = idf(docFreq=4, maxDocs=648) 0.10187038 = queryNorm 281.49374 = (MATCH) fieldWeight(metakeywords_phonetic_notstored:PLK in 421), product of: 1.0 = tf(termFreq(metakeywords_phonetic_notstored:PLK)=1) 5.864453 = idf(docFreq=4, maxDocs=648) 48.0 = fieldNorm(field=metakeywords_phonetic_notstored, doc=421) 181.20946 = (MATCH) weight(metakeywords_text_notstored:plc in 421), product of: 0.62014574 = queryWeight(metakeywords_text_notstored:plc), product of: 6.0875964 = idf(docFreq=3, maxDocs=648) 0.10187038 = queryNorm 292.20462 = (MATCH) fieldWeight(metakeywords_text_notstored:plc in 421), product of: 1.0 = tf(termFreq(metakeywords_text_notstored:plc)=1) 6.0875964 = idf(docFreq=3, maxDocs=648) 48.0 = fieldNorm(field=metakeywords_text_notstored, doc=421) 0.36842465 = (MATCH) weight(typename:wiCourseInfo in 421), product of: 0.19373064 = queryWeight(typename:wiCourseInfo), product of: 1.9017366 = idf(docFreq=262, maxDocs=648) 0.10187038 = queryNorm 1.9017366 = (MATCH) fieldWeight(typename:wiCourseInfo in 421), product of: 1.0 = tf(termFreq(typename:wiCourseInfo)=1) 1.9017366 = idf(docFreq=262, maxDocs=648) 1.0 = fieldNorm(field=typename, doc=421) 350.97913 = (MATCH) weight(fcsp_sitename:"(test_wi_internet_7 test) wi (internet testwiinternet) 7" in 421), product of: 0.6102798 = queryWeight(fcsp_sitename:"(test_wi_internet_7 test) wi (internet testwiinternet) 7"), product of: 5.990748 = idf(fcsp_sitename: test_wi_internet_7=648 test=648 wi=648 internet=648 testwiinternet=648 7=648) 0.10187038 = queryNorm 575.1118 = (MATCH) fieldWeight(fcsp_sitename:"(test_wi_internet_7 test) wi (internet testwiinternet) 7" in 421), product of: 1.0 = tf(phraseFreq=1.0) 5.990748 = idf(fcsp_sitename: test_wi_internet_7=648 test=648 wi=648 internet=648 testwiinternet=648 7=648) 96.0 = fieldNorm(field=fcsp_sitename, doc=421) 0.10155645 = (MATCH) weight(fcsp_benablesearch:T in 421), product of: 0.10171329 = queryWeight(fcsp_benablesearch:T), product of: 0.99845797 = idf(docFreq=648, maxDocs=648) 0.10187038 = queryNorm 0.99845797 = (MATCH) fieldWeight(fcsp_benablesearch:T in 421), product of: 1.0 = tf(termFreq(fcsp_benablesearch:T)=1) 0.99845797 = idf(docFreq=648, maxDocs=648) 1.0 = fieldNorm(field=fcsp_benablesearch, doc=421) </str>
	<str name="test_wi_internet_7_867EFBBA-3FD4-4583-AE697C06D0BAD446"> 532.65857 = (MATCH) sum of: 181.20946 = (MATCH) max of: 140.53111 = (MATCH) weight(coursedescription_phonetic_stored:PLK in 432), product of: 0.64945203 = queryWeight(coursedescription_phonetic_stored:PLK), product of: 6.3752785 = idf(docFreq=2, maxDocs=648) 0.10187038 = queryNorm 216.38412 = (MATCH) fieldWeight(coursedescription_phonetic_stored:PLK in 432), product of: 1.4142135 = tf(termFreq(coursedescription_phonetic_stored:PLK)=2) 6.3752785 = idf(docFreq=2, maxDocs=648) 24.0 = fieldNorm(field=coursedescription_phonetic_stored, doc=432) 112.4123 = (MATCH) weight(coursedescription_text_stored:plc in 432), product of: 0.6907569 = queryWeight(coursedescription_text_stored:plc), product of: 6.7807436 = idf(docFreq=1, maxDocs=648) 0.10187038 = queryNorm 162.73785 = (MATCH) fieldWeight(coursedescription_text_stored:plc in 432), product of: 1.0 = tf(termFreq(coursedescription_text_stored:plc)=1) 6.7807436 = idf(docFreq=1, maxDocs=648) 24.0 = fieldNorm(field=coursedescription_text_stored, doc=432) 168.1683 = (MATCH) weight(metakeywords_phonetic_notstored:PLK in 432), product of: 0.597414 = queryWeight(metakeywords_phonetic_notstored:PLK), product of: 5.864453 = idf(docFreq=4, maxDocs=648) 0.10187038 = queryNorm 281.49374 = (MATCH) fieldWeight(metakeywords_phonetic_notstored:PLK in 432), product of: 1.0 = tf(termFreq(metakeywords_phonetic_notstored:PLK)=1) 5.864453 = idf(docFreq=4, maxDocs=648) 48.0 = fieldNorm(field=metakeywords_phonetic_notstored, doc=432) 181.20946 = (MATCH) weight(metakeywords_text_notstored:plc in 432), product of: 0.62014574 = queryWeight(metakeywords_text_notstored:plc), product of: 6.0875964 = idf(docFreq=3, maxDocs=648) 0.10187038 = queryNorm 292.20462 = (MATCH) fieldWeight(metakeywords_text_notstored:plc in 432), product of: 1.0 = tf(termFreq(metakeywords_text_notstored:plc)=1) 6.0875964 = idf(docFreq=3, maxDocs=648) 48.0 = fieldNorm(field=metakeywords_text_notstored, doc=432) 0.36842465 = (MATCH) weight(typename:wiCourseInfo in 432), product of: 0.19373064 = queryWeight(typename:wiCourseInfo), product of: 1.9017366 = idf(docFreq=262, maxDocs=648) 0.10187038 = queryNorm 1.9017366 = (MATCH) fieldWeight(typename:wiCourseInfo in 432), product of: 1.0 = tf(termFreq(typename:wiCourseInfo)=1) 1.9017366 = idf(docFreq=262, maxDocs=648) 1.0 = fieldNorm(field=typename, doc=432) 350.97913 = (MATCH) weight(fcsp_sitename:"(test_wi_internet_7 test) wi (internet testwiinternet) 7" in 432), product of: 0.6102798 = queryWeight(fcsp_sitename:"(test_wi_internet_7 test) wi (internet testwiinternet) 7"), product of: 5.990748 = idf(fcsp_sitename: test_wi_internet_7=648 test=648 wi=648 internet=648 testwiinternet=648 7=648) 0.10187038 = queryNorm 575.1118 = (MATCH) fieldWeight(fcsp_sitename:"(test_wi_internet_7 test) wi (internet testwiinternet) 7" in 432), product of: 1.0 = tf(phraseFreq=1.0) 5.990748 = idf(fcsp_sitename: test_wi_internet_7=648 test=648 wi=648 internet=648 testwiinternet=648 7=648) 96.0 = fieldNorm(field=fcsp_sitename, doc=432) 0.10155645 = (MATCH) weight(fcsp_benablesearch:T in 432), product of: 0.10171329 = queryWeight(fcsp_benablesearch:T), product of: 0.99845797 = idf(docFreq=648, maxDocs=648) 0.10187038 = queryNorm 0.99845797 = (MATCH) fieldWeight(fcsp_benablesearch:T in 432), product of: 1.0 = tf(termFreq(fcsp_benablesearch:T)=1) 0.99845797 = idf(docFreq=648, maxDocs=648) 1.0 = fieldNorm(field=fcsp_benablesearch, doc=432) </str>
	<str name="test_wi_internet_7_015B18C4-EC53-4832-93418309273F51DC"> 532.65857 = (MATCH) sum of: 181.20946 = (MATCH) max of: 168.1683 = (MATCH) weight(metakeywords_phonetic_notstored:PLK in 435), product of: 0.597414 = queryWeight(metakeywords_phonetic_notstored:PLK), product of: 5.864453 = idf(docFreq=4, maxDocs=648) 0.10187038 = queryNorm 281.49374 = (MATCH) fieldWeight(metakeywords_phonetic_notstored:PLK in 435), product of: 1.0 = tf(termFreq(metakeywords_phonetic_notstored:PLK)=1) 5.864453 = idf(docFreq=4, maxDocs=648) 48.0 = fieldNorm(field=metakeywords_phonetic_notstored, doc=435) 181.20946 = (MATCH) weight(metakeywords_text_notstored:plc in 435), product of: 0.62014574 = queryWeight(metakeywords_text_notstored:plc), product of: 6.0875964 = idf(docFreq=3, maxDocs=648) 0.10187038 = queryNorm 292.20462 = (MATCH) fieldWeight(metakeywords_text_notstored:plc in 435), product of: 1.0 = tf(termFreq(metakeywords_text_notstored:plc)=1) 6.0875964 = idf(docFreq=3, maxDocs=648) 48.0 = fieldNorm(field=metakeywords_text_notstored, doc=435) 0.36842465 = (MATCH) weight(typename:wiCourseInfo in 435), product of: 0.19373064 = queryWeight(typename:wiCourseInfo), product of: 1.9017366 = idf(docFreq=262, maxDocs=648) 0.10187038 = queryNorm 1.9017366 = (MATCH) fieldWeight(typename:wiCourseInfo in 435), product of: 1.0 = tf(termFreq(typename:wiCourseInfo)=1) 1.9017366 = idf(docFreq=262, maxDocs=648) 1.0 = fieldNorm(field=typename, doc=435) 350.97913 = (MATCH) weight(fcsp_sitename:"(test_wi_internet_7 test) wi (internet testwiinternet) 7" in 435), product of: 0.6102798 = queryWeight(fcsp_sitename:"(test_wi_internet_7 test) wi (internet testwiinternet) 7"), product of: 5.990748 = idf(fcsp_sitename: test_wi_internet_7=648 test=648 wi=648 internet=648 testwiinternet=648 7=648) 0.10187038 = queryNorm 575.1118 = (MATCH) fieldWeight(fcsp_sitename:"(test_wi_internet_7 test) wi (internet testwiinternet) 7" in 435), product of: 1.0 = tf(phraseFreq=1.0) 5.990748 = idf(fcsp_sitename: test_wi_internet_7=648 test=648 wi=648 internet=648 testwiinternet=648 7=648) 96.0 = fieldNorm(field=fcsp_sitename, doc=435) 0.10155645 = (MATCH) weight(fcsp_benablesearch:T in 435), product of: 0.10171329 = queryWeight(fcsp_benablesearch:T), product of: 0.99845797 = idf(docFreq=648, maxDocs=648) 0.10187038 = queryNorm 0.99845797 = (MATCH) fieldWeight(fcsp_benablesearch:T in 435), product of: 1.0 = tf(termFreq(fcsp_benablesearch:T)=1) 0.99845797 = idf(docFreq=648, maxDocs=648) 1.0 = fieldNorm(field=fcsp_benablesearch, doc=435) </str>
</lst>

mpicker · November 17, 2015, 2:44am

In case it is useful to anyone else, I came across this today which is useful for displaying the SOLR debug output in a more user friendly layout:

http://explain.solr.pl/

GitHub project: https://github.com/solrpl/explain

Regards
Mark

seancoyne · November 17, 2015, 2:38pm

After modifying the synonyms file did you do a reindex?

mpicker · November 17, 2015, 10:36pm

Hi Sean,

I did - I used the Reset and Re-index option many times yesterday

As a side note for anyone else having trouble while debugging SOLR: just to be 100% certain I did the above reset/index at the right time, in UAT this morning I made sure the synonyms file was back to default, and added back in the two phonetic fields I removed yesterday, did a reset and re-index but the search results were working correctly (no bricklaying). It wasn’t until I did a FarCry updateapp that the real search results appeared (with the bricklaying course showing). I assume this didn’t really need an updateapp but just a clearing of the object broker cache.

Anyway, after now getting the index back to the same state as yesterday, I added plc to the synonyms file, reloaded SOLR, did a Reset and Re-index and just for good measure I did an updateapp. Problem still exists.

Here is a link to the SOLR explain report for the above query:

http://explain.solr.pl/explains/172h9kg7

Regards
Mark