Hello
I have a text file which has blocks like
Code:
dir1/dir2/dir3/name_run_number1:
line1_run_number1_part1
line2_run_number1_part2
line3_run_number1_part3...
Each block is separated with a blank line and there is the ":" in the "header" of each one while each block carries the same "number1" after "run_" suffix
What I want to do is for each block, extract the "number1" as shown in the first line and then for the lines below count from 1-20 and give a message if a "partX" line is missing. Any bash or python would be fine
Thanks
I have two files which has exact same number of lines.
I want first line of first file should be filename of new file and content of this new file should be first line of second file.
Then second line of first file should be filename of again new file and content of this new file should be second line of second file.
then third line of first file should be filename of again new file and content of this new file should be third line of second file.
and so on...
I am trying to do it using for loop but I am not able to create two for loops.
This is what I have done
Code:
IFS=$'\n'
var=$(sed 's/\"http\(.*\)\/\(.*\).wav\"\,\".*/\2/g' 1797.csv) # filenames of all files
var2=$(sed 's/\"http\(.*\)\/\(.*\).wav\"\,\"\(.*\)\"$/\3/g' 1797.csv) # contents of all files
for j in $var;
do
#Here I do not know how to use $var2
done
Please help.
System Info:
I have normal user in CentOS 7 whose name is "mostafa" (the name of the account).
I naturally have another user called root with all privileges. User "mostafa" is put into sudoers file, too.
The OS is installed in VmWare, so the system is all mine.
Problem:
Now I create a file with touch file.sh and put a command in it, but when I want to run it with
Code:
sudo ./file.sh
, an error is shown that the command
Code:
./file.sh
does not exist. But if I
Code:
sudo chmod 777 ./file.sh
then it gets run. My question is that, why should I use
Code:
chmod 777
when I myself have created the file, and I am in sudoers.
Can anyone explain me why shuold I still use
Code:
sudo chmod 777
when the creator of the file is me.
I follow the below URL to install awstats .
Code:
http://thelinuxfaq.com/90-how-to-install-and-configure-awstats-in-centos-fedora-rhel
everything seems works fine , the second last step works fine.
Code:
From data in log file "/var/log/httpd/access_log"...
Phase 1 : First bypass old records, searching new record...
Searching new records from beginning of log file...
Phase 2 : Now process new records (Flush history on disk after 20000 hosts)...
Jumped lines in file: 0
Parsed lines in file: 77
Found 0 dropped records,
Found 0 comments,
Found 0 blank records,
Found 0 corrupted records,
Found 0 old records,
Found 77 new qualified records.
then I tried the below URL , it shows nothing , would advise what is the possible reason ? how can I do the trouble shooting ? thanks
Code:
http://mydomain.com/awstats/awstats.pl?config=mydomain.com
I have an expect script to SSH to a remote host and obtain some user inputs and information about the server/network configuration. The responses are saved in a text file that I then need to copy to my local host so that I can read the lines into variables for use in the parent shell script.
Is there a way to do this without needing to enter the username and password for the local host to use function scp? I have tried the following in my expect script to no avail:
Code:
spawn scp $usr@$host:$flnm .
expect {
-re "(.*)assword:" {
send -s "$pswd\r"
}
}
I have also tried to directly scp the file and enter the username and password to try to debug the issue, and that doesn't work either:
Code:
spawn scp file.txt user@host:file.txt
expect {
-re "(.*)assword:" {
send -s "password\r"
}
"you sure you want to continue connecting" {
send -s "yes\r"
exp_continue
}
}
In both scenarios I have used exp_internal 1, and there are no errors. But I do not end up with the file on my local host.
Following the copy, I would like to delete the file from the remote host. Any suggestions on how to accomplish this?
hi guys
i am trying to find the "size" of a "block" of data in LARGE data files, the example below test_data.txt is very simplified. by "size" i mean the difference in line numbers of a block, and the "size" will be constant throughout the file so
1234 6.600000 4321
1234 8.500000 4321
1234 1.800000 4321
1234 2.300000 4321
1234 8.500000 4321
1234 2.800000 4321
if i define a block as whenever i find 8.500000 in the second column, then in the example the the block size would be 3 becasue 8.500000 occurs on the 5th line and on the 2nd. right now i am using
Code:
grep -n "8.500000" test_data.txt | cut -f1 -d:
and/or
Code:
awk '/8.500000/ {print FNR}' test_data.txt
obviously i don't remeber how to tag text as code?
btw, the grep command is much much faster
both of these commands give an entire list (long list of number for files greater than a gig) of line numbers which i then have to subtract one from another to come up with 3 in the example. not that i'm opposed to doing math, but i would think awk or grep should be able to do this for me
ideas?
tabby
Hello everyone,
Although it seems easy, I've been stuck with this problem for a moment now and I can't figure out a way to get it done.
My problem is the following:
I have a file where each line is a sequence of IP addresses, example :
Line 1: 10.0.01 10.0.0.2
Line 2 : 10.0.0.5 10.0.0.1 10.0.0.2
...
What I'd like to do, is to remove lines that are completely matched in other lines. In the previous example, "Line 1" would be deleted as it is contained in "Line 2".
So far, I've worked with python and set() objects to get the job done but I've got more than 100K lines and sets lookups are becoming time consuming as the program goes :/
Thanks for you help
I have a little bash script that cats out a file and tells me if there is a line
where the 11th column has more than 6 characters in it.
It emails me where there is a bad line in a file - bead meaning that it will break a
donwstream process.
anyhow when i get the email saying that there is a bad file i just log in to the pc via
vpn and the I sed out the lines from the file that I get in the email. The bad lines are
always in danny.csv not danny1.csv
It has been the same lines killing the downstream process for a few weeks, so i put the "sed -i's" into
the script and it does it automagically.
[CODE]
for i in danny.csv danny1.csv
do
cat /come/and/play/with/$i | perl -ne 'print if length((split /,/)[10]) > 6' | mail -s "danny.csv bad line" casper@casperr.com
done
#it would be nice to find a perl change the file in place
sed -i '/D,642,0642,UBF,EVL,,M,,S,S,FOREVER,213,213,/d' /come/and/play/with/us/danny.csv
sed -i '/D,642,0642,UBF,EVL,,M,,S,S,QSP-U=C,4,4,/d' /come/and/play/with/us/danny.csv
[CODE]
However when a new line gets put into this file, I am going to have to log in and take out the line.
SO I have been trying to write a perl one liner that will edit the file in place, like sed, and make a
backup of the file. I just need a perl one liner that will delete any line where the 11th columns has more
than 6 characters in it.
[CODE]
perl -p -i.bak -e 's/\,\w{7}\,//g - which does not work.
[CODE]
I tried something like this:
[CODE]
perl -nle 'print if /\,\w{7}\,/' /come/and/play/with/us/danny.csv
[CODE]
but that does not catch the QSP-U=C and it catches more lines than just the
FOREVER. for a solutinog I need to focus on the the 11th column.
Hi guys,
I am looking to write a script where I need to process many sets of transactions.
I want to process the set if $ 1 == "01" field $5 = "W", and $ 1 == "07" field $3 = "YY" otherwise copy set to output.
Example of the input file:
Code:
01 08 77 78 W 9890
02 08 66 68 0 8554
07 08 YY 85 9 7545
01 08 99 87 X 8787
04 09 85 85 4 8758
09 87 88 78 7 6584
10 84 ZZ 99 8 9887
A new set is always starting with $1 == "01".
Script should only process first set because its 5th value is "W" and put "MATCHED" in the end.
and copy the unmatched set "X" as it is to output.
Code:
Example of output file:
01 08 77 78 W 9890
02 08 66 68 0 8554
07 08 YY 85 9 7545
MATCHED
01 08 99 87 X 8787
04 09 85 85 4 8758
09 87 88 78 7 6584
10 84 ZZ 99 8 9887
and so on..
thanks for your help
Hi
I am using this command to get some info about bots/spiders from my Centos server access.log file:
Code:
grep 'spider\|bot' access.log | sort -u -f >> bots.txt
Result is like this (i know pingdom is not bad):
Code:
141.101.105.102 - - [28/Mar/2015:01:59:56 +0200] "GET / HTTP/1.1" 200 24194 "-" "Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)"
141.101.105.158 - - [28/Mar/2015:02:09:56 +0200] "GET / HTTP/1.1" 200 24260 "-" "Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)"
141.101.105.102 - - [28/Mar/2015:02:19:56 +0200] "GET / HTTP/1.1" 200 24277 "-" "Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)"
108.162.215.53 - - [27/Mar/2015:23:13:21 +0200] "GET /user/74595-tery1/?tab=idm HTTP/1.1" 200 3905 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
108.162.215.53 - - [27/Mar/2015:23:11:59 +0200] "GET /user/275904-ktlk21/ HTTP/1.1" 200 3805 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
108.162.215.75 - - [27/Mar/2015:23:21:31 +0200] "GET /user/74595-tery1/?tab=topics HTTP/1.1" 200 13588 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
Is there any command that can remove duplicate lines if the ip and the user-agent is the same on each line?
To get something like:
Code:
141.101.105.102 - - [28/Mar/2015:01:59:56 +0200] "GET / HTTP/1.1" 200 24194 "-" "Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)"
141.101.105.158 - - [28/Mar/2015:02:09:56 +0200] "GET / HTTP/1.1" 200 24260 "-" "Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)"
108.162.215.53 - - [27/Mar/2015:23:11:59 +0200] "GET /user/275904-ktlk21/ HTTP/1.1" 200 3805 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
108.162.215.75 - - [27/Mar/2015:23:21:31 +0200] "GET /user/74595-tery1/?tab=topics HTTP/1.1" 200 13588 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
Or if there is no way for this then to get only one line (even if different ip's exist for each user agent) like:
Code:
141.101.105.102 - - [28/Mar/2015:01:59:56 +0200] "GET / HTTP/1.1" 200 24194 "-" "Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)"
108.162.215.53 - - [27/Mar/2015:23:11:59 +0200] "GET /user/275904-ktlk21/ HTTP/1.1" 200 3805 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"