# Web Scrape ## Tools * Pull URLS from a file * cat demo-file.js | ./extract.rb curl -s * | ./extract.rb * ## Screenshots * Check list of Hosts for Websites and take Screenshot: * aquatone * cat hosts.txt | aquatone -out \~/lab/aqua/ * aquatone\_report.html * CutyCapt - pull screenshots from a website ``` > cutycapt --url=https://pentest.mxhx.org/ --out=mxhx.png ``` ## View Tons of Webpages - Grab and Compile * Quickly view tons of webpages * Nmap open webports on a network * Loop the IP's through cutycapt to png * Create html and link/view all png ``` > sudo nmap -A -p80 --open 10.x.x.x/24 -oG results > cat results | grep 80 | grep -v "Nmap" | awk '{print $2}' > for i in $(cat results | grep 80 | grep -v "Nmap" | awk '{print $2}'); do cutycapt --url=$i --out=$i.png;done > cat htmlshot.sh #!/bin/bash echo "
" > web.html ls -1 *.png | awk -F : '{ print $1":\n

"}' >> web.html echo "" >> web.html > chmod +x ./htmlshot.sh > ./htmlshot.sh > firefox web.html ```