Monday, November 8, 2021

Code to Split large file and keep the header row on each file

You can pass a single argument as the file name(Exclude the extension...) or you can modify the file to take extension as the arguments.
fileName=$1
splitFileNamePrefix="${fileName}_"
header=$(head -1 ${fileName}.dat)
split -l 40000 ${fileName}.dat splitFileNamePrefix
n=1
for f in splitFileNamePrefix*
do
    prefixSplitNo=0
    if [[ ${n} -gt 9 ]]; then
        prefixSplitNo=''
    fi
    outputFileName="${splitFileNamePrefix}${prefixSplitNo}${n}.dat"
    echo ${outputFileName}
    if [[ ${n} -ne 1 ]]; then
        echo ${header} > ${outputFileName}
    fi
    cat ${f} >> ${outputFileName}
    rm ${f}
    ((n++))
done

Read More...

Pages

 ©mytechtoday.com 2006-2010

 ©Mytechtoday

TOP