Web Scrape

Tools

Pull URLS from a file
- https://github.com/jobertabma/relative-url-extractor cat demo-file.js | ./extract.rb curl -s
- https://hackerone.com/hacktivity | ./extract.rb

Screenshots

Check list of Hosts for Websites and take Screenshot:
- aquatone
- cat hosts.txt | aquatone -out ~/lab/aqua/
- aquatone_report.html
CutyCapt - pull screenshots from a website

> cutycapt --url=https://pentest.mxhx.org/ --out=mxhx.png

View Tons of Webpages - Grab and Compile

Quickly view tons of webpages
- Nmap open webports on a network
- Loop the IP's through cutycapt to png
- Create html and link/view all png

> sudo nmap -A -p80 --open 10.x.x.x/24 -oG results
> cat results | grep 80 | grep -v "Nmap" | awk '{print $2}'
> for i in $(cat results | grep 80 | grep -v "Nmap" | awk '{print $2}'); do cutycapt --url=$i --out=$i.png;done

> cat htmlshot.sh
#!/bin/bash
echo "<HTML><BODY><BR>" > web.html
ls -1 *.png | awk -F : '{ print $1":\n<BR><IMG SRC=\""$1""$2"\" width=600><BR>"}' >> web.html
echo "</BODY></HTML>" >> web.html

> chmod +x ./htmlshot.sh
> ./htmlshot.sh
> firefox web.html

PreviousWebmin NextWordpress

Last updated 2 years ago

Was this helpful?

Web Scrape

Tools

Pull URLS from a file
- https://github.com/jobertabma/relative-url-extractor cat demo-file.js | ./extract.rb curl -s
- https://hackerone.com/hacktivity | ./extract.rb

Screenshots

Check list of Hosts for Websites and take Screenshot:
- aquatone
- cat hosts.txt | aquatone -out ~/lab/aqua/
- aquatone_report.html
CutyCapt - pull screenshots from a website

> cutycapt --url=https://pentest.mxhx.org/ --out=mxhx.png

View Tons of Webpages - Grab and Compile

Quickly view tons of webpages
- Nmap open webports on a network
- Loop the IP's through cutycapt to png
- Create html and link/view all png

> sudo nmap -A -p80 --open 10.x.x.x/24 -oG results
> cat results | grep 80 | grep -v "Nmap" | awk '{print $2}'
> for i in $(cat results | grep 80 | grep -v "Nmap" | awk '{print $2}'); do cutycapt --url=$i --out=$i.png;done

> cat htmlshot.sh
#!/bin/bash
echo "<HTML><BODY><BR>" > web.html
ls -1 *.png | awk -F : '{ print $1":\n<BR><IMG SRC=\""$1""$2"\" width=600><BR>"}' >> web.html
echo "</BODY></HTML>" >> web.html

> chmod +x ./htmlshot.sh
> ./htmlshot.sh
> firefox web.html

PreviousWebmin NextWordpress

Last updated 2 years ago

Was this helpful?