Need Help On Csv Formatting

Hi Linux Experts,

I have the following problem to solve:

-Below is the CSV file give

firstname,lastname,password,username,notes,city,phonenumber
fred,smith, notgood1, fredsmith, this user\, is the first in this file, Brighton,345698
Peter, Bloggs, anotherbad,peterbloggs,,London,987123
Jo, cooper, notmuch, jcooper, this user is Jo, Brighton, 456987
john, carter,nearlyempty,jcarter,This note is actually very long\, but really doesn't say anything very useful,,345777
sam,jones,passing, samjones, Not much of a note really, Manchester, 135790

- capitalise the first letter of the two name fields
- sanitise the formatting
- move the username column to the beginning of each line
- the phone number is missing the area code - look up the city in the following table, and add it to the beginning of the phone number column:

City, Area Code
London, 5
Brighton, 6
Manchester, 7

Provide the corrected CSV file.

One of the problems I have is that whenever I use the comma as FS the output for column 5 is the following

cat Test.csv | awk 'BEGIN { FS = "," } {print $5 }'
notes
this user\

this user is Jo
This note is actually very long\
Not much of a note really

It stops in the middle of the entry because it sees the comma but what I am trying to achieve is to produce the full entry for column 5 like this:

this user\, is the first in this file
This note is actually very long\, but really doesn't say anything very useful

I have to probably escape somehow the FS in the text but so far no joy with completing this task. Also can you kindly help out for the rest of the requirements.
I really appreciate your help in advance.

Ivan

I have an expect script to SSH to a remote host and obtain some user inputs and information about the server/network configuration. The responses are saved in a text file that I then need to copy to my local host so that I can read the lines into variables for use in the parent shell script.

Is there a way to do this without needing to enter the username and password for the local host to use function scp? I have tried the following in my expect script to no avail:
Code:

spawn scp $usr@$host:$flnm .
expect {
	-re "(.*)assword:" { 
		send -s "$pswd\r"
	}
}

I have also tried to directly scp the file and enter the username and password to try to debug the issue, and that doesn't work either:
Code:

spawn scp file.txt user@host:file.txt
expect {
	-re "(.*)assword:" {
		send -s "password\r"
	}
	"you sure you want to continue connecting" {
		send -s "yes\r"
		exp_continue
	}
}

In both scenarios I have used exp_internal 1, and there are no errors. But I do not end up with the file on my local host.

Following the copy, I would like to delete the file from the remote host. Any suggestions on how to accomplish this?

I have a little bash script that cats out a file and tells me if there is a line
where the 11th column has more than 6 characters in it.
It emails me where there is a bad line in a file - bead meaning that it will break a
donwstream process.

anyhow when i get the email saying that there is a bad file i just log in to the pc via
vpn and the I sed out the lines from the file that I get in the email. The bad lines are
always in danny.csv not danny1.csv
It has been the same lines killing the downstream process for a few weeks, so i put the "sed -i's" into
the script and it does it automagically.

[CODE]
for i in danny.csv danny1.csv
do
cat /come/and/play/with/$i | perl -ne 'print if length((split /,/)[10]) > 6' | mail -s "danny.csv bad line" casper@casperr.com
done

#it would be nice to find a perl change the file in place
sed -i '/D,642,0642,UBF,EVL,,M,,S,S,FOREVER,213,213,/d' /come/and/play/with/us/danny.csv
sed -i '/D,642,0642,UBF,EVL,,M,,S,S,QSP-U=C,4,4,/d' /come/and/play/with/us/danny.csv
[CODE]

However when a new line gets put into this file, I am going to have to log in and take out the line.
SO I have been trying to write a perl one liner that will edit the file in place, like sed, and make a
backup of the file. I just need a perl one liner that will delete any line where the 11th columns has more
than 6 characters in it.
[CODE]
perl -p -i.bak -e 's/\,\w{7}\,//g - which does not work.
[CODE]
I tried something like this:

[CODE]
perl -nle 'print if /\,\w{7}\,/' /come/and/play/with/us/danny.csv
[CODE]
but that does not catch the QSP-U=C and it catches more lines than just the
FOREVER. for a solutinog I need to focus on the the 11th column.

Hi Experts,

I am trying to make new enteries in a csv file in new column but am not able to do so.Please help for the same.

Requirement:

There are multiple directories & within those directories i have sub-directories and i want to build a csv file with 2 columns of Directories mapped to their sub-directories. Can you please help me with this. I tried the following code:

Code:

#!/bin/bash

homeDir="$HOME"



ls ~/Parent/ | cut -c1-9 > ~/test_111.csv

while read Child

do

Entry="$(ls $homeDir/Parent/$Child/ABC/XYZ/DEF/PQR)"

echo $Entry

for (( c=1; c<=5; c++ ))

do



sed -i ci"$Entry" test_222.csv

done


done < test_111.csv

Basically i want two columns of csv file , First column should have Child name & Second cloumn should have Sub-Directory name inside PQR Directory.

Any help will be useful on this.

Thanks in Advance!

Best Regards,
Vijay Bhatia

System Info:

I have normal user in CentOS 7 whose name is "mostafa" (the name of the account).

I naturally have another user called root with all privileges. User "mostafa" is put into sudoers file, too.

The OS is installed in VmWare, so the system is all mine.

Problem:

Now I create a file with touch file.sh and put a command in it, but when I want to run it with Code:

sudo ./file.sh

, an error is shown that the command Code:

./file.sh

does not exist. But if I Code:

 sudo chmod 777 ./file.sh

then it gets run. My question is that, why should I use Code:

chmod 777

when I myself have created the file, and I am in sudoers.

Can anyone explain me why shuold I still use Code:

sudo chmod 777

when the creator of the file is me.

hi guys

i am trying to find the "size" of a "block" of data in LARGE data files, the example below test_data.txt is very simplified. by "size" i mean the difference in line numbers of a block, and the "size" will be constant throughout the file so

1234 6.600000 4321
1234 8.500000 4321
1234 1.800000 4321
1234 2.300000 4321
1234 8.500000 4321
1234 2.800000 4321

if i define a block as whenever i find 8.500000 in the second column, then in the example the the block size would be 3 becasue 8.500000 occurs on the 5th line and on the 2nd. right now i am using

Code:

 grep -n "8.500000" test_data.txt | cut -f1 -d:

and/or

Code:

 awk '/8.500000/ {print FNR}' test_data.txt

obviously i don't remeber how to tag text as code?

btw, the grep command is much much faster

both of these commands give an entire list (long list of number for files greater than a gig) of line numbers which i then have to subtract one from another to come up with 3 in the example. not that i'm opposed to doing math, but i would think awk or grep should be able to do this for me

ideas?

tabby

I have created a sql file and it will create a HTML file with query result.But I'm unable to retrieve a html body in my email instead am getting a mail body with html tags.Please help

sql----

--sample HTML report.
--
-- Usage: sqlplus LOGON @script OUTPUT_FILENAME
--

-- Activate HTML output and configure the generated markup.
SET MARKUP HTML ON SPOOL ON -
HEAD '<title>My Report</title> -
<style type="text/css"> -
table { background: #eee; font-size: 80%; } -
th { background: #ccc; } -
td { padding: 0px; } -
</style>'

-- Dump results to the file that is given on the command line.
SPOOL &1

-- Only dump to file, not to the terminal.
SET TERMOUT OFF

-- Titles and formatting of columns.
COLUMN name HEADING 'Name'
--COLUMN job HEADING 'Job Title'
--COLUMN salary HEADING 'Salary' FORMAT $99,990

-- The query
--
select tablespace_name from dba_tablespaces;
/

-- Close file, which also closes the HTML tags.
SPOOL OFF

-- Back to non-HTML output
SET MARKUP HTML OFF


mailx command----

mailx -s 'file system' c.bambarandage@prima.com.lk < sam.html

Thanks,
Charith.

Ok yes this is a homework assignment BUT I am NOT looking to have the answers given to me. I am in the 6th week of my first Linux class ever and we are in our few weeks of beginning scripting. I have some ideas of what to do or where to start but not many and no one to bounce any ideas off...we are using UNIX Bash shell so any others I have no clue. The scenario is that I need a script that searches all my users home directories for bad words. I need the script to report to the screen certain info like username and word found and path. It should ask a user if it is good or bad and if bad be put into a file of list of bad file names, if good remove from list and no longer flagged by the script. What I have so far is wanting to somehow do a loop. I do know that if I do a grep -r -e kill -e steal /home/* I get a list of what I need. I also know that the list is separated by delimiters which I can pipe to get a variable for the things I need. I also know that I can put it to a file with a > filename.txt

What I have no clue is how to start a loop that would do this...

for each line in filename.txt
UNAME=...
LOC=...
TXT=...
echo "Username: $UNAME, Line with bad word found: $TXT, and Path and file name: $LOC. Is this a BAD file? (Y)"
Read YORN

if ["$YORN" = "Y" ]; then
>> (line of text from grep) badfiles.txt
fi

Next or whatever goes there...sorry if this is crazy I just really need some direction. I am trying to learn so please don't give me the answer...that will do nothing for me and I will not be able to explain the code I came up with.

Hello. I am new to Linux and am looking for a solution to remove column 10 from a .csv data file as this column is causing me problems. Any help would be appreciated.
Thank you!

hello Experts!
need some help here with python FTP. so what i'm trying to do is, from a linux machine, ill be sending over a python script to a windows machine via python ftp. that works fine with no problem. now, i want to execute that python script i sent remotely. im having trouble with that. here's my code to just execute the script:

p.s : please dont suggest other options such as python paramiko due to many network security reasons

Code:

ftp = ftplib.FTP("windows machine name")
ftp.login("username", "password")
ftp.cwd("where\\python\\script\\is kept\\")

ftp.sendcmd("python myscript.py")

but looks like it doesn't recognize that command. how do i accomplish this please?