web scraping with python

Creativity Corner
Forum rules
witnesone
Small Maggot
Small Maggot
Posts: 33
Joined: Thu Jan 02, 2014 7:54 am

web scraping with python

Postby witnesone » Mon Sep 04, 2017 3:20 pm

Anyone had any experience with web scraping or python and databases? I don't have much experience with either and my understanding of python is pretty limited. I use the code below to extract what I want from the webpage and print it on the screen, which is about 200 lines. I understand bash scripting and crontab enough to write a small script to run the code below daily. I want to save the output and I'm guessing it would be best to save to a database but I've got no experience with databases at all

import re
import urllib2,sys
import lxml
from lxml import etree
from lxml.html.soupparser import fromstring
from lxml.etree import tostring
from lxml.cssselect import CSSSelector
from BeautifulSoup import BeautifulSoup, NavigableString

address='xxx'
html = urllib2.urlopen(address).read()
soup = BeautifulSoup(html)
thetds = soup.findAll('td', attrs={'class': 'name'})
for thetd in thetds:
print thetd.string

Return to “Programming”

Who is online

Users browsing this forum: No registered users and 0 guests