This example demonstrates how to handle pages with dynamic content. For this it uses Google’s search engine page.
Keep in mind that Google’s search engine page is highly dynamic in the sense that you must expect it to change rapidly, possibly even several times per day. Therefore this example simply demonstrates how to cope with the status quo at the time of writing this example.
Your own web pages will be very unlikely to change as often and as dramatically (internally) as Google’s page, therefore it should be much more “stable” for you.
Note how the solution makes assumptions along the way:
The real name pattern to retrieve the result link objects:
The “next page” button real name:
“Sub-hits” (links shown below the first hit/link at the time of this writing) have a different DOM path depth than the “normal” result links.
All “normal” result links have the same DOM path depth.