Page 1 of 1

web scraping with python

Posted: Mon Sep 04, 2017 3:20 pm
by witnesone
Anyone had any experience with web scraping or python and databases? I don't have much experience with either and my understanding of python is pretty limited. I use the code below to extract what I want from the webpage and print it on the screen, which is about 200 lines. I understand bash scripting and crontab enough to write a small script to run the code below daily. I want to save the output and I'm guessing it would be best to save to a database but I've got no experience with databases at all

import re
import urllib2,sys
import lxml
from lxml import etree
from lxml.html.soupparser import fromstring
from lxml.etree import tostring
from lxml.cssselect import CSSSelector
from BeautifulSoup import BeautifulSoup, NavigableString

address='xxx'
html = urllib2.urlopen(address).read()
soup = BeautifulSoup(html)
thetds = soup.findAll('td', attrs={'class': 'name'})
for thetd in thetds:
print thetd.string