Unicode functions and fixes in latest 6.9 beta

[DataMystic Support]

Here's what is new so far -

* XSLT filter
* Unicode (UTF-16LE) search and replace, with options for space
compression and for ignoring non-spacing characters.
* Unicode (UTF-16LE) pattern matching.
* New file grid right-click menu items:
- Analyze file - checks for various Unicode, XML, CSV and Tab
delimited formats, Unix/PC/Mac end of line format and other file
types
- Hex dump - displays a hex dump of the first 200 characters of
the file
* Added \x{...} Unicode character matching for exact matches. This
means that for a Unicode space you can type \x00000020 instead
of \x00\x00\x00\x20.
* Unicode conversion filter is now much simpler, as you just
choose an input and output conversion in one step.
* Synchronized scrolling option for Trial Input and Output.
* Fixed file grid ordering - if you moved rows around, TextPipe
did not remember the new row position.
* Fixed menu items for Remove, Copy and Move tab fields. These did
not add the required filters.
* Fixed help hyperlinks on Trial Run and Files tabs.
* Fixed bugs in Unicode Make Little/Big Endian filters.
* Fixed bug in Convert From Unicode filter, for when it
encountered a character that could not be converted. It now
outputs a space.
* The EasyPattern [year] keyword now checks for a 4 digit year
before checking for a 2 digit year.
* Convert Word Document To Text filter now only closes the Active
Document, avoiding the MS Word error message 'This method or
property is unavailable because a document window is not
active'.
* Fixed line wrapping problem with clipboard export of several
restrict-type filters.

We're also considering
- unicode compose/decompose/normalize functions
- unicode logging
- Replacement of tree view with unicode-enabled tree view

Any and all feedback is welcome! Please post replies to this post.

[DataMystic Support]

We have now made the Unicode conversions far easier, with the following changes -

1. Bundle up the Unicode <=> ANSI filters inside the Convert to/from Unicode filter.
2. Make the Convert to/from Unicode filters use an alphabetically sorted list, instead of sorting by internal Ids.
3. Allow both the input/output format to be specified for Convert to/from Unicode filter, instead of having the 'half-way house' of data moving around in utf-32LE form.

[rain]

Hello

I've a problem with TextPipePro7.0.3demo
When i use the filter "Convert from ANSI to UTF-16"
It works fine when i process one file but when I process *.txt, the header "FEFF" is inserted only at the beginning of the first file of the list.
this header should be inserted at the beginning of each file

regards

[DataMystic Support]

Yes, we've confirmed that this is a bug. We plan to have a 7.0.4 release available shortly with a workaround until we can get a proper fix.

As a workaround until 7.0.4 is available, you could convert to UTF-16LE, then add a Byte Order Mark manually (see the sample filters).

[DataMystic Support]

A 7.0.4 is now available to fix this problem.