4 2 2783
For relevant code visit http://stackoverflow.com/questions/21006940/how-to-load-all-entries-in-an-infinite-scroll-at-once-to-parse-the-html-in-pytho/21008335#21008335
By anonymous 2017-09-20
This you won't be able to do with requests and BeautifulSoup as the page that you want to extract the information from loads the rest of the entries through JS when you scroll down. You can do this using selenium which opens a real browser and you can pass page down key press events programmatically. Watch this video to see the action. http://www.youtube.com/watch?v=g54xYVMojos
Below is the script that extracts all the 100 post titles using selenium.
import time from selenium import webdriver from selenium.webdriver.common.keys import Keys browser = webdriver.Chrome() browser.get("https://medium.com/top-100/december-2013") time.sleep(1) elem = browser.find_element_by_tag_name("body") no_of_pagedowns = 20 while no_of_pagedowns: elem.send_keys(Keys.PAGE_DOWN) time.sleep(0.2) no_of_pagedowns-=1 post_elems = browser.find_elements_by_class_name("post-item-title") for post in post_elems: print post.text