Fix a web scraper program created in Python

We have a Web Scraping program, created in Python, that stopped working properly. I need someone to go through the program files, and fix the part that is no longer working properly. We need to end up with a working program.

The purpose of the program is to:

1. Spider through a website ad download all files that result from the spidering

2. Format each file downloaded to a specific format

The program appears to be working properly, and goes through the same steps as it always did, but there are no output files created. There is a good chance that there is change need in the 'search and replace' file; "[login to view URL]"

I have attached the program files, including source code.

The following is the program description:

1. Spider through a website ad download all files that result from the spidering

2. Format each file downloaded to a specific format

Part one:

You will be given a batch of starting URL's that look like this:

[login to view URL]

You will follow each of these URL's that will lead to another page with links that look like this:

[login to view URL]

You will follow each of these URL's that will lead again to another page with links that look like this:

[login to view URL]

You will now follow each of these links that leads to a page that links to specific documents. The links within the pages tend to look like this:

<table width="480"><tr>

<td><table width="120">

<tr><td>

<a class="tpl" href="/cgi/t/text/text-idx?c=ecfr&SID=f68f503ab8017206c54fb367aaaa7851&amp;rgn=div8&amp;view=text&amp;node=10:1.0.1.1.4.1.56.1&amp;idno=10">

&sect;5.100</a></td></tr>

</table></td>

<td><table width="354">

<tr><td>Purpose and effective date.</td></tr>

</table></td>

</tr></table>

each of these links leads to a page that needs to be saved with the following naming structure that looks like this:

[login to view URL]

other examples of naming structures:

6cfrAppendix A to Part [login to view URL]

Part two of this project:

After you have downloaded each file, you will need to put each file into a specific html page structure.

1. You will first strip all of the information before <!-- startDynamic --> and after the <!-- endDynamic -->

2. You will now need to create a header for each record that looks like the files that are part of the samples.

3. You will need to replace the string in the text when it comes across a graphic:

example string:

Please replace:

<img src="/graphics/

With this string:

<img src="[login to view URL]

AND replace this string:

<a href="/graphics/pdfs/

With this string:

<a href="[login to view URL]

4. You will need to create a footer at the bottom of each section, after the p class=” cita, that looks this this example:

<p class="cita">[54 FR 53314, Dec. 28, 1989]</p>

<br><p><center>Copyright 2013 Compliance Publishing Corporation (877) 500-6737</center>

</body>

</html>

5. You must be able to accommodate both regular regulations and the Appendix sections

6. Some of the titles have one less level. This program must be able selectable to how many levels deep the individual text is located.

The project must be completed in 7 days.

.

Python Web Scraping

Project ID: #6552649

About the project