Web Scrape
Tools
Pull URLS from a file
https://github.com/jobertabma/relative-url-extractor cat demo-file.js | ./extract.rb curl -s
https://hackerone.com/hacktivity | ./extract.rb
Screenshots
Check list of Hosts for Websites and take Screenshot:
aquatone
cat hosts.txt | aquatone -out ~/lab/aqua/
aquatone_report.html
CutyCapt - pull screenshots from a website
> cutycapt --url=https://pentest.mxhx.org/ --out=mxhx.png View Tons of Webpages - Grab and Compile
Quickly view tons of webpages
Nmap open webports on a network
Loop the IP's through cutycapt to png
Create html and link/view all png
> sudo nmap -A -p80 --open 10.x.x.x/24 -oG results
> cat results | grep 80 | grep -v "Nmap" | awk '{print $2}'
> for i in $(cat results | grep 80 | grep -v "Nmap" | awk '{print $2}'); do cutycapt --url=$i --out=$i.png;done
> cat htmlshot.sh
#!/bin/bash
echo "<HTML><BODY><BR>" > web.html
ls -1 *.png | awk -F : '{ print $1":\n<BR><IMG SRC=\""$1""$2"\" width=600><BR>"}' >> web.html
echo "</BODY></HTML>" >> web.html
> chmod +x ./htmlshot.sh
> ./htmlshot.sh
> firefox web.htmlLast updated
Was this helpful?