web scraping with python

Creativity Corner
Post Reply
Small Maggot
Small Maggot
Posts: 36
Joined: Thu Jan 02, 2014 7:54 am

web scraping with python

Post by witnesone » Mon Sep 04, 2017 3:20 pm

Anyone had any experience with web scraping or python and databases? I don't have much experience with either and my understanding of python is pretty limited. I use the code below to extract what I want from the webpage and print it on the screen, which is about 200 lines. I understand bash scripting and crontab enough to write a small script to run the code below daily. I want to save the output and I'm guessing it would be best to save to a database but I've got no experience with databases at all

import re
import urllib2,sys
import lxml
from lxml import etree
from lxml.html.soupparser import fromstring
from lxml.etree import tostring
from lxml.cssselect import CSSSelector
from BeautifulSoup import BeautifulSoup, NavigableString

html = urllib2.urlopen(address).read()
soup = BeautifulSoup(html)
thetds = soup.findAll('td', attrs={'class': 'name'})
for thetd in thetds:
print thetd.string

Posts: 2
Joined: Wed May 09, 2018 7:01 pm

Re: web scraping with python

Post by Susanut » Wed May 09, 2018 7:16 pm

Thank you for the web I'm looking for this site. It helped me a lot.

Post Reply