Suggestion: Provide filter Remove all diacritics

Get help with installation and running here.

Moderators: DataMystic Support, Moderators, DataMystic Support, Moderators, DataMystic Support, Moderators

User avatar
DataMystic Support
Site Admin
Posts: 2227
Joined: Mon Jun 30, 2003 12:32 pm
Location: Melbourne, Australia
Contact:

Re: Suggestion: Provide filter Remove all diacritics

Post by DataMystic Support »

Hi David,

I understand the value, and it might be possible with come caveats.

My concern is that TextPipe has tried hard never to assume the format of incoming files, because auto-detection is generally inprecise. But knowing internally what one filter expects vs one another filter is giving it is do-able.

I can see some potential issues. If two filters were separated by a search/replace, then there is no way of knowing if the search/replace modified the file format. But equally, this might be the express intention of the user, to modify the file format.
dfhtextpipe
Posts: 986
Joined: Sun Dec 09, 2007 2:49 am
Location: UK

Re: Suggestion: Provide filter Remove all diacritics

Post by dfhtextpipe »

See also my recent email (sent yesterday) about grouping the Unicode filters that are UTF-16LE only.

Aside: hasn't the official Unicode terrminology for UTF-16LE been changed to UCS-2 BOM ?

David
David
User avatar
DataMystic Support
Site Admin
Posts: 2227
Joined: Mon Jun 30, 2003 12:32 pm
Location: Melbourne, Australia
Contact:

Re: Suggestion: Provide filter Remove all diacritics

Post by DataMystic Support »

Thanks David - had a look but did not find any reference to this terminology change. Do you have a reference?
dfhtextpipe
Posts: 986
Joined: Sun Dec 09, 2007 2:49 am
Location: UK

Re: Suggestion: Provide filter Remove all diacritics

Post by dfhtextpipe »

I was mistaken. UCS-2 was an earlier form that preceded UTF-16.

I was confused by Notepad++ having changed menu options from UTF-16 LE to UCS-2.

It may be because it doesn't fully support the former.

Ah well.

https://en.wikipedia.org/wiki/Universal_Coded_Character_Set
David
User avatar
DataMystic Support
Site Admin
Posts: 2227
Joined: Mon Jun 30, 2003 12:32 pm
Location: Melbourne, Australia
Contact:

Re: Suggestion: Provide filter Remove all diacritics

Post by DataMystic Support »

Ok, thanks for the clarification!
dfhtextpipe
Posts: 986
Joined: Sun Dec 09, 2007 2:49 am
Location: UK

Re: Suggestion: Provide filter Remove all diacritics

Post by dfhtextpipe »

Please add Help page for the new Remove diacritics filter.

Please add Help page to explain Filter Library\Unicode\UTF-16LE only

Please add See also links in these existing filter help pages to the new page.

David
David
User avatar
DataMystic Support
Site Admin
Posts: 2227
Joined: Mon Jun 30, 2003 12:32 pm
Location: Melbourne, Australia
Contact:

Re: Suggestion: Provide filter Remove all diacritics

Post by DataMystic Support »

Remove diacritics is ready for TP 11.6. It will handle UTF16-LE - any other format can be converted using other TP filters.
Post Reply