Page 1 of 1

Remove duplicate lines without sorting?

Posted: Wed Feb 22, 2006 6:39 am
by lngzlz
Is it possible for TextPipe to remove duplicate lines without having to sort all the lines first?

If sorted, the original order of lines has been crashed. This is usually not desirable. :oops:

Posted: Wed Feb 22, 2006 8:02 am
by DataMystic Support
We've added a new filter to the next release - 'etext\remove duplicates without sorting.fll'.

Here is the filter export:

|--Comment...
| Add line numbers which we will use to later restore the original line order
|
|--Add line numbers
| Start line 0
| Increment 1
| [ ] Don't increment on blank lines
| [ ] Don't number blank lines
| Format %7.7d
| [ ] Don't reset on end of file
|
|--Comment...
| Compare lines (ignoring line numbers) and discard duplicates
|
|--Ascending ANSI sort (case insensitive), remove duplicates, start column 9, length 50
|
|--Comment...
| Return the lines to the original order
|
|--Ascending numeric sort, length 7
|
|--Comment...
| Now remove the line numbers
|
|--Remove column:8 columns starting at column 1
|

Posted: Wed Feb 22, 2006 2:55 pm
by lngzlz
Compare lines (ignoring line numbers) and discard duplicates
How can I ignore line numbers when comparing lines?

Posted: Wed Feb 22, 2006 3:17 pm
by DataMystic Support
In

Ascending ANSI sort (case insensitive), remove duplicates, start column 9, length 50

set the Start Column (here set to 9).