remove errant CRLF from customer emails

Get help with installation and running here.

Moderators: DataMystic Support, Moderators, DataMystic Support, Moderators, DataMystic Support, Moderators

Post Reply
sheridany
Posts: 36
Joined: Thu Nov 15, 2007 4:20 am

remove errant CRLF from customer emails

Post by sheridany »

I have to remove CRLF from customer emails who write us and organize the data into one row per customer. The CRLF in the text data needs to be removed until we encounter a 3 digits number which starts a new line.
it look like this....

001|DEC2007|I am frustrated with the bank
being able to help me
Please call me.
002|DEC2007| This is craxy. I have never seen
anything
like
this.
User avatar
DataMystic Support
Site Admin
Posts: 2227
Joined: Mon Jun 30, 2003 12:32 pm
Location: Melbourne, Australia
Contact:

Post by DataMystic Support »

Use the EasyPattern:

Code: Select all

[ 3 digits, 1+ chars, mustEndWith( cr, lf, 3 digits ) ]
Replace with:

Code: Select all

$0
Add a subfilter to this to replace EasyPattern

Code: Select all

[cr, lf]
with nothing.
sheridany
Posts: 36
Joined: Thu Nov 15, 2007 4:20 am

Perhaps I am missing something?

Post by sheridany »

I tried what you said but it did not work right. I assumed you meant a Easy Pattern subfilter under the Subfilter that replaces
[ 3 digits, 1+ chars, mustEndWith( cr, lf, 3 digits ) ] with $0?

I must be missing something....

Here is the filter export.

Input from file(s)
| [ ] Confirm before processing each file
| [ ] Confirm before processing read/only files
| [ ] Delete input files after processing
| Process binary files
|
|--Remove multiple whitespace
|
|--EasyPattern [[ 3 digits, 1+ chars, mustEndWith( cr, lf, 3 digits ) ]] with [$0]
| | [ ] Match case
| | [ ] Whole words only
| | [ ] Case sensitive replace
| | [ ] Prompt on replace
| | [ ] Skip prompt if identical
| | [ ] First only
| | [ ] Extract matches
| | Maximum text buffer size 4096
| |
| +--EasyPattern [[cr, lf]] with []
| [ ] Match case
| [ ] Whole words only
| [ ] Case sensitive replace
| [ ] Prompt on replace
| [ ] Skip prompt if identical
| [ ] First only
| [ ] Extract matches
| Maximum text buffer size 4096
User avatar
DataMystic Support
Site Admin
Posts: 2227
Joined: Mon Jun 30, 2003 12:32 pm
Location: Melbourne, Australia
Contact:

Post by DataMystic Support »

Ok, I missed a couple of things. It does not process the last line, and it should be replacing the embedded cr.lf with a space.

I worked around this by adding a ascii(255) (hex \xff) character at the start of each record to prevent it joining all the records together. These get removed at the end

Here is the new filter:

Code: Select all

|   
|--EasyPattern [[ linestart, 3 digits ]] with [\xff$0]
|     [ ] Match case
|     [ ] Whole words only
|     [ ] Case sensitive replace
|     [ ] Prompt on replace
|     [ ] Skip prompt if identical
|     [ ] First only
|     [ ] Extract matches
|     Maximum text buffer size 4096
|   
|--EasyPattern [[ ascii($ff), capture(3 digits, longest 1+ not ascii($ff)) ]] with [$0]
|  |  [ ] Match case
|  |  [ ] Whole words only
|  |  [ ] Case sensitive replace
|  |  [X] Prompt on replace
|  |  [ ] Skip prompt if identical
|  |  [ ] First only
|  |  [ ] Extract matches
|  |  Maximum text buffer size 4096
|  |
|  |--EasyPattern [[cr, lf]] with [ ]
|  |     [ ] Match case
|  |     [ ] Whole words only
|  |     [ ] Case sensitive replace
|  |     [X] Prompt on replace
|  |     [ ] Skip prompt if identical
|  |     [ ] First only
|  |     [ ] Extract matches
|  |     Maximum text buffer size 4096
|  |   
|  +--Add footer [\r\n]
|      
|--EasyPattern [[ ascii($ff) ]] with []
|     [ ] Match case
|     [ ] Whole words only
|     [ ] Case sensitive replace
|     [ ] Prompt on replace
|     [ ] Skip prompt if identical
|     [ ] First only
|     [ ] Extract matches
|     Maximum text buffer size 4096
|   
I can also email you this filter if you drop us an email
sheridany
Posts: 36
Joined: Thu Nov 15, 2007 4:20 am

Not quite yet

Post by sheridany »

All I want to do is get rid of the carriage returns before the start of a new line. The new line always start with the id of 001, 002 003 etc. The customer has imposed the cr lf when they are typing the message to us. Our downstream application can't handle the cr lf because it thinks it is a new line when it is not. This is a pipe delimited file if that helps. The simple thing to do is perhaps use the restrict filter and just remove the cr lf from the third field?

I apologize for not clarifying up front better.

The first and second parsed and processed line would look like this
001|DEC2007|I am frustrated with the bank being able to help.....
002|DEC2007|This is crazy. I have never seen anything like this.


Old
001|DEC2007|I am frustrated with the bank
being able to help me
Please call me.
002|DEC2007| This is crazy. I have never seen
anything
like
this.
User avatar
DataMystic Support
Site Admin
Posts: 2227
Joined: Mon Jun 30, 2003 12:32 pm
Location: Melbourne, Australia
Contact:

Post by DataMystic Support »

Just send us an email and we can send you the filter above, which fixes the problem.
sheridany
Posts: 36
Joined: Thu Nov 15, 2007 4:20 am

did you get my email?

Post by sheridany »

I sent it to simon.carter at datamystic.com. Is that correct?
User avatar
DataMystic Support
Site Admin
Posts: 2227
Joined: Mon Jun 30, 2003 12:32 pm
Location: Melbourne, Australia
Contact:

Post by DataMystic Support »

Yes - we already emailed you the filter (last week). Check your corporate filtering for it.
User avatar
DataMystic Support
Site Admin
Posts: 2227
Joined: Mon Jun 30, 2003 12:32 pm
Location: Melbourne, Australia
Contact:

Post by DataMystic Support »

Hi Sheridan,

Clearly something is wrong with your companies email filtering, and I'm prepared to bet that my company is not the only one with the problem. It is a huge waste of our resources to constantly resend and re-reply to emails because of external filtering issues - and understandably you get frustrated with our apparent lack of response.

Please get a gmail account and use that for contacting us in future. I'll be happy to send you the filter to your gmail account.
sheridany
Posts: 36
Joined: Thu Nov 15, 2007 4:20 am

Email Filtering

Post by sheridany »

At least now we know it is on our side. I have a yahoo account that I can access at work. It is the same as my screen name in the forum here @yahoo.com. Will that work? I at least have access to anything that goes to their spam filter and can flag it as otherwise.
Post Reply