Remove duplicate lines without sorting?

Get help with installation and running here.

Moderators: DataMystic Support, Moderators, DataMystic Support, Moderators, DataMystic Support, Moderators

Post Reply
lngzlz
Posts: 41
Joined: Wed Sep 21, 2005 10:43 pm

Remove duplicate lines without sorting?

Post by lngzlz »

Is it possible for TextPipe to remove duplicate lines without having to sort all the lines first?

If sorted, the original order of lines has been crashed. This is usually not desirable. :oops:
User avatar
DataMystic Support
Site Admin
Posts: 2227
Joined: Mon Jun 30, 2003 12:32 pm
Location: Melbourne, Australia
Contact:

Post by DataMystic Support »

We've added a new filter to the next release - 'etext\remove duplicates without sorting.fll'.

Here is the filter export:

|--Comment...
| Add line numbers which we will use to later restore the original line order
|
|--Add line numbers
| Start line 0
| Increment 1
| [ ] Don't increment on blank lines
| [ ] Don't number blank lines
| Format %7.7d
| [ ] Don't reset on end of file
|
|--Comment...
| Compare lines (ignoring line numbers) and discard duplicates
|
|--Ascending ANSI sort (case insensitive), remove duplicates, start column 9, length 50
|
|--Comment...
| Return the lines to the original order
|
|--Ascending numeric sort, length 7
|
|--Comment...
| Now remove the line numbers
|
|--Remove column:8 columns starting at column 1
|
lngzlz
Posts: 41
Joined: Wed Sep 21, 2005 10:43 pm

Post by lngzlz »

Compare lines (ignoring line numbers) and discard duplicates
How can I ignore line numbers when comparing lines?
User avatar
DataMystic Support
Site Admin
Posts: 2227
Joined: Mon Jun 30, 2003 12:32 pm
Location: Melbourne, Australia
Contact:

Post by DataMystic Support »

In

Ascending ANSI sort (case insensitive), remove duplicates, start column 9, length 50

set the Start Column (here set to 9).
Post Reply