Unicode files with fixed fields and Code Page Conversions

[Ian Dukes]

Hi

My company is processing large volumes of transactions captured from the web for not only English countries, though also countries that use Traditional Chinese, Simplified Chinese, and Thai.

The transactions are captured in an SQL Server database and extracted to file using DTS in ANSI format and finally transferred down to a mainframe. The fields in the file are fixed length and contain names and addresses as well as possibly a mix of characters between Chinese and English. In order to get the Chinese/Thai character exports working I have started testing with extracting the data to text file in Unicode format. We've tested then trying to use FTP to convert the characters to different code pages as well as an add on to DTS to perform the character translation. Though both these methods have issues retainig the fixed length name and address fields.

I've been trying to get the eval version of Text Pipe Pro to perform the conversion with varying amounts of success and am trying to find out if anyone has a suggestion about the best approach. Having read many accounts of people's troubles using various techniques, I realise that there's no apparent simple solution. Though I'm still searching around and was hoping Text Pipe Pro might provide this solution.

What I tried setting up the filter at the end of the posting. The filter performs the code page translation and attempts to force the padding to a width of the fixed length field.
Though it doesn't appear to do what I need it to. Perhaps I'm taking a simplistic approach to the problem and need to break down the file translation into smaller steps. Perhaps write a script that passes the Name and Address fields individually to Text Pipe Pro and concatenate the results to give me the final output record.

Any help or suggestions would be greatly appreciated.

==================================================================================================
TextPipe Evaluation Edition
==================================================================================================
Filter Title: Unicode_Big5.fll

Filter List
-----------
Filter options
| [ ] Log to file
| [X] Append to logfile
| Log filename: textpipe.log
| Threshold 500
|
|--Input from file(s)
| [ ] Confirm before processing each file
| [ ] Confirm before processing read/only files
| [ ] Delete input files after processing
| Process binary files
|
|--Restrict bytes:Byte 1 .. byte 97
| |
| +--Convert from UTF-16LE to 950 (ANSI/OEM - Traditional Chinese Big5)
|
|--Restrict bytes:Byte 98 .. byte 147
| |
| |--Convert from UTF-16LE to 950 (ANSI/OEM - Traditional Chinese Big5)
| |
| +--Pad to width of 50
|
|--Restrict bytes:Byte 148 .. byte 197
| |
| |--Convert from UTF-16LE to 950 (ANSI/OEM - Traditional Chinese Big5)
| |
| +--Pad to width of 50
|
|--Restrict bytes:Byte 198 .. byte 247
| |
| |--Convert from UTF-16LE to 950 (ANSI/OEM - Traditional Chinese Big5)
| |
| +--Pad to width of 50
|
|--Restrict bytes:Byte 248 .. byte 279
| |
| |--Convert from UTF-16LE to 950 (ANSI/OEM - Traditional Chinese Big5)
| |
| +--Pad to width of 50
|
|--Restrict bytes:Byte 280 .. byte 950
| |
| +--Convert from UTF-16LE to 950 (ANSI/OEM - Traditional Chinese Big5)
|
+--Output to file(s)
[ ] Only update date on changed files
[ ] Append mode
[ ] Change extension to: .txt
[ ] Open output file
Only output modified files Backup mode

Files List
----------
IRESP_TW_ORDNGT_20080918_111055a.dat
==================================================================================================

[DataMystic Support]

Hi Ian,

Check the filter we have sent and we'll see how much closer we are!