How Can I Remove Duplicated Lines On A File?

Hi

I am using this command to get some info about bots/spiders from my Centos server access.log file:

Code:
grep 'spider\|bot' access.log | sort -u -f >> bots.txt

Result is like this (i know pingdom is not bad):

Code:
141.101.105.102 - - [28/Mar/2015:01:59:56 +0200] "GET / HTTP/1.1" 200 24194 "-" "Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)"
141.101.105.158 - - [28/Mar/2015:02:09:56 +0200] "GET / HTTP/1.1" 200 24260 "-" "Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)"
141.101.105.102 - - [28/Mar/2015:02:19:56 +0200] "GET / HTTP/1.1" 200 24277 "-" "Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)"
108.162.215.53 - - [27/Mar/2015:23:13:21 +0200] "GET /user/74595-tery1/?tab=idm HTTP/1.1" 200 3905 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
108.162.215.53 - - [27/Mar/2015:23:11:59 +0200] "GET /user/275904-ktlk21/ HTTP/1.1" 200 3805 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
108.162.215.75 - - [27/Mar/2015:23:21:31 +0200] "GET /user/74595-tery1/?tab=topics HTTP/1.1" 200 13588 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"

Is there any command that can remove duplicate lines if the ip and the user-agent is the same on each line?

To get something like:

Code:
141.101.105.102 - - [28/Mar/2015:01:59:56 +0200] "GET / HTTP/1.1" 200 24194 "-" "Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)"
141.101.105.158 - - [28/Mar/2015:02:09:56 +0200] "GET / HTTP/1.1" 200 24260 "-" "Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)"
108.162.215.53 - - [27/Mar/2015:23:11:59 +0200] "GET /user/275904-ktlk21/ HTTP/1.1" 200 3805 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
108.162.215.75 - - [27/Mar/2015:23:21:31 +0200] "GET /user/74595-tery1/?tab=topics HTTP/1.1" 200 13588 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"

Or if there is no way for this then to get only one line (even if different ip's exist for each user agent) like:

Code:
141.101.105.102 - - [28/Mar/2015:01:59:56 +0200] "GET / HTTP/1.1" 200 24194 "-" "Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)"
108.162.215.53 - - [27/Mar/2015:23:11:59 +0200] "GET /user/275904-ktlk21/ HTTP/1.1" 200 3805 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"



Similar Content



Mutt Does Not Show Full Subject

I use following shell script to send email. But the subject of email always shows "This" instead of "This is L_1.R is finished". You may refer http://www.linuxquestions.org/questions/linux-newbie-8/how-to-define-variable-in-%60ssh%60-4175540566/
Code:
host_list=("c15-0330-01.ad.mtu.edu" "c15-0330-02.ad.mtu.edu" "c15-0330-03.ad.mtu.edu" "c15-0330-04.ad.mtu.edu")
program=("L_1" "L_2" "L_3" "L_4")
subject="The job is finished"
ssh -f c15-0330-01.ad.mtu.edu 'echo' "the job ${program[0]} is finished" '|' 'mutt "zwang10@mtu.edu" -s' "This is "${program[0]}".R is finished";

Delay Calculation From Trace Files Of AODV / LAR Protocols

when i am running delay.awk file on AODV.tr and LAR.tr For AODV.tr , it is giving me the o/p value as some delay
but for LAR , its showing me the follwing error

gawk: e2edelay.awk:93: (FILENAME=larscen5.tr FNR=36516130) fatal: division by zero attempted

I was unable to recify the error. could you please help me.
thank you so much

this is the code of awk file which i am using for calculation of delay. kindly help

# http://205.196.121.184/fnufnnc17mwg/...c/e2edelay.awk
# http://mohittahiliani.blogspot.dk/20...s-for-ns2.html
# ===================================================================

# AWK Script for calculating:

# => Average End-to-End Delay.

# ===================================================================



BEGIN {

seqno = -1;

# droppedPackets = 0;

# receivedPackets = 0;

count = 0;

}

{

if($4 == "AGT" && $1 == "s" && seqno < $6) {

seqno = $6;

}
# else if(($4 == "AGT") && ($1 == "r")) {

# receivedPackets++;

# } else if ($1 == "D" && $7 == "tcp" && $8 > 512){

# droppedPackets++;

# }

#end-to-end delay

if($4 == "AGT" && $1 == "s") {

start_time[$6] = $2;

} else if(($7 == "cbr") && ($1 == "r")) {

end_time[$6] = $2;

} else if($1 == "D" && $7 == "cbr") {

end_time[$6] = -1;

}

}


END {

for(i=0; i<=seqno; i++) {

if(end_time[i] > 0) {

delay[i] = end_time[i] - start_time[i];

count++;

}

else

{

delay[i] = -1;

}

}

for(i=0; i<count; i++) {

if(delay[i] > 0) {

n_to_n_delay = n_to_n_delay + delay[i];

}

}

n_to_n_delay = n_to_n_delay/count;



print "\n";

# print "GeneratedPackets = " seqno+1;

# print "ReceivedPackets = " receivedPackets;

# print "Packet Delivery Ratio = " receivedPackets/(seqno+1)*100
#"%";

# print "Total Dropped Packets = " droppedPackets;

print "Average End-to-End Delay = " n_to_n_delay * 1000 " ms";

print "\n";

}

"vconsole.conf" Show Unicode

Hi!

Here is mine configuration and I want make it show unicode(russian) in console, it works perfect in X11 but not when I logging into tty.

vconsole.conf
Code:
LOCALE="en_US.UTF-8"
KEYMAP="sv-latin1"
FONT_UNIMAP="lat1"
FONT_MAP="ISO8859-1"
CONSOLEFONT="ruscii_8x8"
HARDWARECLOCK="localtime"
TIMEZONE="Europe/Stockholm"
USECOLOR="yes"

Getting The Pattern Of The Value To Print

what can be the pattern to get the value of "q"
below is my JSON file:-
Code:
{"requestDate":"2014-12-17 00:00:13.120","taskId":"558b7f40-014a-1000-803d-970e8333b0c9","appData":{"user_ip":"170.61.236.1, 10.0.255.47","_request_params":{"params":{"uid":["59a07452-8517-4713-c9b3-82bfd408c9f1"],"filtered":["false","false"],"requestId":["55897963-014a-1000-8021-970e8333b0c9"],"sort_by":["price "],"bId":["control"],"userid":["003f9510-2cc3-4925-efaa-3b0779082c77"],"sku":["4532886"],"q":["<U+200B><U+200B>
<U+200B>baby carrier"],"eId":["control"],"action":["click"],"sort_order":["asc"],"z":["1"]},"headers":{"x-bluecoat-via":"87ca26b138e17f23","accept-language":"en-US,en;q\u003d0.8","host":"api.apac-sg.zettata.com","x-forwarded-for":"170.61.236.1, 10.0.255.47","x-forwarded-port":"80","user-agent":"Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36","x-forwarded-proto":"http","cache-control":"max-stale\u003d0","x-real-ip":"10.0.255.47","referer":"http://www.shopclues.com/mum-mee-baby-sling-carrier-1.html","accept-encoding":"gzip, deflate, sdch"},"inCookies":[],"post":false},"cookies":[],"request_params":{"params":{"uid":["59a07452-8517-4713-c9b3-82bfd408c9f1"],"filtered":["false","false"],"requestId":["55897963-014a-1000-8021-970e8333b0c9"],"sort_by":["price "],"bId":["control"],"userid":["003f9510-2cc3-4925-efaa-3b0779082c77"],"sku":["4532886"],"q":["<U+200B><U+200B><U+200B>baby carrier"],"eId":["control"],"action":["click"],"sort_order":["asc"],"z":["1"]},"headers":{"x-bluecoat-via":"87ca26b138e17f23","accept-language":"en-US,en;q\u003d0.8","host":"api.apac-sg.zettata.com","x-forwarded-for":"170.61.236.1, 10.0.255.47","x-forwarded-port":"80","user-agent":"Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36","x-forwarded-proto":"http","cache-control":"max-stale\u003d0","x-real-ip":"10.0.255.47","referer":"http://www.shopclues.com/mum-mee-baby-sling-carrier-1.html","accept-encoding":"gzip, deflate, sdch"},"inCookies":[],"post":false},"user_agent":"Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36"},"cookies":[],"status":200}

from this i want to get the value of q like this:-
Quote:
<U+200B><U+200B><U+200B>baby carrier

Need Help In Bash Scripting

I have two files which has exact same number of lines.
I want first line of first file should be filename of new file and content of this new file should be first line of second file.
Then second line of first file should be filename of again new file and content of this new file should be second line of second file.
then third line of first file should be filename of again new file and content of this new file should be third line of second file.
and so on...
I am trying to do it using for loop but I am not able to create two for loops.
This is what I have done
Code:
IFS=$'\n'
var=$(sed 's/\"http\(.*\)\/\(.*\).wav\"\,\".*/\2/g' 1797.csv) # filenames of all files
var2=$(sed 's/\"http\(.*\)\/\(.*\).wav\"\,\"\(.*\)\"$/\3/g' 1797.csv) # contents of all files
for j in $var;
do
#Here I do not know how to use $var2
done

Please help.

Rsync, Reliable "copy And Paste" Type Of Backup In Case Things Break?

What I did in windows was create images of my drive and restore them.

in linux I am running

Code:
rsync -aAXv --exclude={"/home/*","/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found"} /* /path/to/backup/folder

and this creates a folder for me with all my files, and apparently saves meta data like permissions and paths...

Since I'm using arch and things break sometimes,I'm booted into a CLI with errors and cannot figure my way out since I'm a noob... would I be able to just delete my entire root and replace it with the rsync backup without a problem?

Script Queries

Hi,

Below is the output of the "date" command:
Thu Apr 2 16:07:59 SGT 2015

After assigning the output to a parameter as follows:
DATE=`date`

The output has changed when "echo $DATE":
Thu Apr 2 16:09:40 SGT 2015
^^

The "2" has moved forward.

I don't want the "2" to move forward, please advise how to go about it.

Thanks in advance.

Check Empty Variable Without Comments

Hello,

They gave me this:
Code:
#!/bin/bash

# Write error message on stderr and die
function die() {
  echo "$@" >&2
  exit 1
}

# Load var from properties files
function load() {
  for propertie in "$@" ; do
    [[ -f "${propertie}" ]] || die "load() : \"${propertie}\" don't exist !"
    while read ; do
echo "${REPLY%=*}"
echo "${REPLY#*=}"
      [[ -n "${REPLY%%=*}" && -z "${REPLY#*=}" ]] || die "load() : Variable \"${REPLY%%=*}\" is empty!"
      eval "${REPLY%%=*}"=\'"${REPLY#*=}"\'
    done < "${propertie}"
  done
}

load "manage_srcds.conf"
echo "SRCDS_TICKRATE=$SRCDS_TICKRATE"
echo "SRCDS_SCREEN=$SRCDS_SCREEN"
echo "SRCDS_SRCDS_GAME_NAME=$SRCDS_GAME_NAME"
echo "SRCDS_MAXPLAYERS=$SRCDS_MAXPLAYERS"

but, return:
Quote:
###################################################
###################################################
load() : Variable "###################################################" is empty!

cat manage_srcds.conf

I just wish that there was no empty variable excluding those who do not have integer.

Best regards,

Bash Script For Xfce4 Powermanager Settings Switching Forth And Backward

i need a bash script to switch to zero second on ac the waiting time for turning off the monitor (highlighten in red). same as for on battery (highlighten in purple)

Code:
<?xml version="1.0" encoding="UTF-8"?>

<channel name="xfce4-power-manager" version="1.0">
  <property name="xfce4-power-manager" type="empty">
    <property name="power-button-action" type="empty"/>
    <property name="show-tray-icon" type="empty"/>
    <property name="logind-handle-lid-switch" type="bool" value="true"/> 
    <property name="dpms-on-ac-sleep" type="uint" value="1"/>
    <property name="dpms-on-ac-off" type="uint" value="2"/> 
    <property name="brightness-on-battery" type="uint" value="9"/> 
    <property name="dpms-on-battery-sleep" type="uint" value="1"/>
    <property name="dpms-on-battery-off" type="uint" value="2"/> 
    <property name="lock-screen-suspend-hibernate" type="bool" value="false"/>
    <property name="dpms-enabled" type="bool" value="true"/>
  </property>
</channel>

Network Configuration For Redhat 7

I just installed Redhat 7 and try to config the network , add the IP address to network card , but I got the problem.

I modify the file /etc/sysconfig/network-scripts/ifcfg-enp0s3 , the configuration is as below , and then reboot .

DEVICE="enp0s3"
NETBOOT="yes"
HWADDR="08:00:27:15:38:B7"
TYPE="Ethernet"
BOOTPROTO="none"
NAME="enp0s3"
UUID="462f4834-4fe7-43a7-84e7-83b2722e94c1"
ONBOOT="yes"
IPADDR="192.168.1.1"
NETMASK="255.255.255.0"

After reboot , I use "ip addr show" still not shown the ip address , the ip address is still blank , would advise what is wrong ? thanks