Hi,
My PLIP draft :
Title: Plain text search ignores accents.
'''Proposer:''' Thomas Desvenain
'''Seconder:''' Vincent Fretin
== Motivation ==
Most users want the search to ignore accents.
This is a question of comfort :
a search on econometrie should found documents with term "économétrie".
And that would fix an issue, as most users don't use accents with upper case characters :
For example, a search on 'économétrie' doesn't found a document titled as
"Econométrie"
== Assumptions ==
We will improve plone lexicon so that it normalizes indexed and searched terms in plain text indexes (ZCTextIndex).
A document with 'Econométrie' and 'économétrie' words will be indexed for 'econometrie' term.
A search on 'économétrie' or 'econometrie' word will search for 'econometrie' index value.
The normalization will be made on the model of document ids generation in Plone.
To avoid performance issue and extensions at lowest level than Plone, normalization will be independent of site language.
== Proposal & Implementation ==
We have to add a new Case Normalizer named 'I18n Case Normalizer'.
This normalizer will use plone.i18n tools to generate an ascii string from any word to normalize.
== Deliverables ==
* Code
- Add a new class in Products.CMFPlone.UnicodeSplitter, register it as 'I18n Case Normalizer'.
- plone_lexicon will use this.
* Upgrades
- Upgrade plone_lexicon with this normalizer.
- Reindex ZCTextIndex indexes.
* Unit tests
- Test search and found documents containing words 'économétrie', 'Économétrie', 'Econométrie' with criterion 'econometrie' and 'économétrie'.
- Equivalent Unit tests with eastern language
== Risks ==
The main risks are :
- it has to work with all languages, included eastern languages.
- check consequences on general performances.
- indexes have to be updated for backward compatibility.
== Participants ==
Thomas Desvenain
I need volunteers to add tests for eastern languages.
== Progress ==
-------------------------------
NB: what do you think about an
'experimental.plonei18nlexicon' package ?
On Tue, Aug 2, 2011 at 10:30 PM, thomas desvenain
<[hidden email]> wrote:
On Sun, Jul 31, 2011 at 11:04 AM, Hanno Schlichting
<[hidden email]> wrote:
On Mon, Jun 6, 2011 at 3:08 PM, thomas desvenain
> Most users want the search to ignore accents
>
> where "économétrie"
> finds "econometrie", "Econométrie", "Économétrie".
>
> What do you think about a PLIP to give Plone lexicon a casenormalizer
> that would use plone.i18n stuff to normalize ZCTextIndex lexical
> values ?
>
> (I can write the PLIP and implement it... but i know it is a major issue)
+10 :)
The framework team is accepting PLIPs for 4.3 - please get it in :)
Great ! :)
So, I will (after i come back from my holiday place).
Thank you !
Thomas
Hanno
--
Thomas Desvenain
Téléphone : 09 51 37 35 18
--
Thomas Desvenain
Téléphone : 09 51 37 35 18
------------------------------------------------------------------------------
Get a FREE DOWNLOAD! and learn more about uberSVN rich system,
user administration capabilities and model configuration. Take
the hassle out of deploying and managing Subversion and the
tools developers use with it.
http://p.sf.net/sfu/wandisco-dev2dev_______________________________________________
Plone-i18n mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-i18n
Locked
