python – Trouble with hidden rows web scraping html table-ThrowExceptions

Exception or error:

Im relatively new to web scraping and I really can’t figure out why this is not working, I think it is something to do with hidden rows within the table I want to scrape.

I am trying to extract a fairly simple table into a a pandas dataframe, by way of a for loop. However when I try to use find_all(‘tr’) to extract the rows, I end up with a completely empty set.

Here is my code:

url = ';mode=search;sort=worldRank;view=table;asc=true;f1=France;dsply=100'

html = requests.get(url)
soup = BeautifulSoup(html.text, 'lxml')
table_body = soup.find('tbody')
rows = table_body.find_all('tr')

When I print(rows) I just get:


The for loop I want to use is:

labels = []
data = []

for row in rows:

cols = {'Field': labels, 'Data': data}
df = pd.DataFrame(cols)

Any help would be very much appreciated since I have been banging my head against the wall all morning trying to get this to work 🙁
None of my google searches have cleared this up for me

Thank You in advance!

How to solve:

Your tbody is empty when retrieved. In the browser it is filled by javascript. Here is the extract of the raw html:

<tbody _ngcontent-sc18=""><!----></tbody>

It is useless as is for BeautifulSoup. Maybe you could try that:

Leave a Reply

Your email address will not be published. Required fields are marked *