Friday, April 1, 2016

Netezza: nzload how to handle NUL 0x00 and non-printable characters

PROBLEM:
nzload  responds with errors:

bad #: input row #(byte offset to last char examined) [field #, declaration] diagnostic, "text consumed"[last char examined]
----------------------------------------------------------------------------------------------------------------------------
1: 58351(52) [4, NVARCHAR(1000)] binary zero byte - ignoreZero not enabled, "OOL-US-TDMA"[0x00<NUL>]
2: 59487(104) [4, NVARCHAR(1000)] found control characters and ctrlChars option is off, "SW-BU-BOOST-US:5500000|SW-BU-BOOST-DS:37500000|#K"[0x19]
3: 146957(60) [4, NVARCHAR(1000)] binary zero byte - ignoreZero not enabled, "OOL-US"[0x00<NUL>]
4: 858805(53) [4, NVARCHAR(1000)] binary zero byte - ignoreZero not enabled, "BOOST-US"[0x00<NUL>]



SOLUTION: 

Adding nzload option -ignoreZero YES ingnores loading NUL charactes. They will be stripped out.
Adding nzload option -ctrlChars loads control characters into a table  

nzload
-host 10.10.10.100
-u  db_user
-pw db_password
-db MY_DATABASE
-t  MY_TABLE
-delim ";"
-quotedValue NO
-encoding INTERNAL
-df /usr/local/data/MY_TABLE.data
-lf /usr/local/data/$(date +'%Y%m%d%H%M%S')_$(hostname)_MY_TABLE.nzload.log
-bf /usr/local/data/$(date +'%Y%m%d%H%M%S')_$(hostname)_ MY_TABLE.nzload.bad
-maxErrors 2
-DateStyle 'DMY'
-DateDelim ' '
-TimeStyle '24HOUR'
-TimeDelim ':'
-fillRecord
-ignoreZero YES     <-- This takes care of "binary zero byte - ignoreZero not enabled"
-ctrlChars          <-- This takes care of "found control characters and ctrlChars option is off"



INVESTIGATION AND DETAILS


Initial lookup using cat command does not show NUL characters.


vi shows null characters but it's not conclusive: 


Notepad++ comes at help not only displaying non-printable ASCII characters but also identifying their type. 



No comments:

Post a Comment