Selenium

Selenium 4.9.0 Chrome 112 ChromeDriver 112.0.5615.49 Python 3.11 import time import json from selenium.webdriver import ( Chrome, DesiredCapabilities, ) desired_capabilities = DesiredCapabilities.CHROME desired_capabilities["goog:loggingPrefs"] = { "performance": "ALL", } driver = Chrome( desired_capabilities=desired_capabilities, ) driver.implicitly_wait(5) driver.get("https://www.google.com/") known_url_set = set() while True: performance_log_entries = driver.get_log("performance") for log_entry in performance_log_entries: log_message = json.loads(log_entry.get("message", "{}")).get("message", {}) method = log_message.get("method") params = log_message.get("params", {}) if method == "Network.responseReceived": response = params.get("response", {}) url = response.get("url") if url in known_url_set: continue known_url_set.add(url) print(url) time.sleep(1) Python+SeleniumでChromeデベロッパーツールのNetworkタブ相当の情報を取得する - Qiita java - Using Selenium how to get network request - Stack Overflow

Selenium 4.9.0 Chrome 112 ChromeDriver 112.0.5615.49 Python 3.11 from selenium.webdriver import ( Chrome, ChromeOptions, ) download_dir = "./downloads" os.makedirs(download_dir, exist_ok=True) options = ChromeOptions() options.add_experimental_option("prefs", { "profile.default_content_settings.popups": 0, "download.default_directory": os.path.realpath(download_dir), "download.prompt_for_download": False, "download.directory_upgrade": True, }) driver = Chrome( options=options, ) python - How to change download directory location path in Selenium using Chrome? - Stack Overflow

2022-08-06 追記 Seleniumのバージョンが上がって一部非推奨化したりしたので、そちらに対応した版を作成しました。 リポジトリ: https://github.com/aoirint/compose-selenium-python-template 2020-09-28 docker-compose.yml version: '3.8' services: app: build: ./app/ entrypoint: [ "wait-for-it", "selenium:4444", "--", "python3", "/code/main.py" ] volumes: - ./work:/work environment: SELENIUM_URL: http://selenium:4444/wd/hub depends_on: - selenium selenium: image: selenium/standalone-chrome volumes: - /dev/shm:/dev/shm app/Dockerfile FROM python:3 WORKDIR /work RUN apt update && apt install -y \ wait-for-it ADD requirements.txt /tmp/ RUN pip3 install -r /tmp/requirements.txt ADD code/ /code app/requirements.txt requests >= 2.24.0 selenium app/code/main.py from selenium import webdriver from selenium.common.exceptions import NoSuchElementException from selenium.webdriver.common.desired_capabilities import DesiredCapabilities selenium_url = os.environ['SELENIUM_URL'] website_url: str = None driver = webdriver.Remote( command_executor=selenium_url, desired_capabilities=DesiredCapabilities.CHROME, ) driver.get(website_url)