python – Trouble with hidden rows web scraping html table-ThrowExceptions

Exception or error:

Im relatively new to web scraping and I really can’t figure out why this is not working, I think it is something to do with hidden rows within the table I want to scrape.

I am trying to extract a fairly simple table into a a pandas dataframe, by way of a for loop. However when I try to use find_all(‘tr’) to extract the rows, I end up with a completely empty set.

Here is my code:

url = 'https://pjammcycling.com/home;mode=search;sort=worldRank;view=table;asc=true;f1=France;dsply=100'

html = requests.get(url)
soup = BeautifulSoup(html.text, 'lxml')
table_body = soup.find('tbody')
rows = table_body.find_all('tr')

When I print(rows) I just get:

[]

The for loop I want to use is:

labels = []
data = []

for row in rows:
    labels.append(str(row.find_all('td')[0].text))
    data.append(str(row.find_all('td')[1].text))

cols = {'Field': labels, 'Data': data}
df = pd.DataFrame(cols)

Any help would be very much appreciated since I have been banging my head against the wall all morning trying to get this to work 🙁
None of my google searches have cleared this up for me

Thank You in advance!

How to solve:

Your tbody is empty when retrieved. In the browser it is filled by javascript. Here is the extract of the raw html:

<tbody _ngcontent-sc18=""><!----></tbody>

It is useless as is for BeautifulSoup. Maybe you could try that: https://pythonprogramming.net/javascript-dynamic-scraping-parsing-beautiful-soup-tutorial/

Leave a Reply

Your email address will not be published. Required fields are marked *