Subtle bug in Convert Numeric HTML/XML Entities to text

[dfhtextpipe]

Hi Simon,

I have just encountered some very rare errors in the output of the filter Convert Numeric HTML/XML Entities to text.

My input file contains 31,286 NCRs that are to be converted.
These are scattered through out a UTF-8 text file that has 36,888 lines and a total of 7,219,971 characters.

The vast majority of the NCRs are converted correctly.

However, there were 5 locations where the conversion was incorrect.

I can only conclude that there's a subtle software bug in TextPipe Standard 10.7.2.

The problem is in understanding the root cause, because the same NCRs are converted correctly in other locations in the file.

The errors were detected by comparing the output file with one obtained by using BabelPad version 1.0.0.4 to Convert NCRs to Unicode.

This issue is critical, seeing as the TextPipe conversion is producing these rare but wrong results.

The attached .diff file was generated using WinMerge. Not ideal, but it does provide the context.

NB. I can readily send you a copy of the input file by email so that you might investigate further in detail.

Best regards,

David

[dfhtextpipe]

Since posting the issue, I have found a workaround.

If the filter Convert Numeric HTML/XML Entities to text is plac1ed under a Restrict to each line in turn filter, the output file is without error.

I thereby conclude that the bug must have some relation to data caching or memory management.

David

[dfhtextpipe]

A workaround is one thing, but were you able to confirm whether or not my conjecture about data caching was valid?

If so, is there anything that can be done to prevent the bad output that I observed without such a workaround?

David

[DataMystic Support]

Hi David - can you please send the original file and the TextPipe filter?

The filter is a simple byte-by-byte state machine, expecting

&#<decimal digits>; or
&#x<hex digits>; or

It handles 2-byte unicode characters properly.

I don't understand why *any* changes are being made to your file as the non-&# data should pass straight through.

Simon

[dfhtextpipe]

Email sent just.

David

[DataMystic Support]

Hi David,

This definitely is an issue with TextPipe 10.7.2.

But testing with TextPipe 10.8, the issue is no longer there.

[dfhtextpipe]

That's good news!

I await TextPipe 10.8 with bated breath.

David

[dfhtextpipe]

Related question, but the other way round...

Do any of the new libraries used in TextPipe 11.x support the conversion of Unicode characters to numerical entities?

This would obviate needing to resort to a map.

cf. Help for filter Convert Numeric HTML/XML Entities to text includes:

To convert plain text to HTML entities (ie convert in the opposite direction), use a Map.

Regards,

David