How To Increase Your Chances Of A Successful Scrape Using Skyscraper

Skyscraper is a fantastic scraping programme for RetroPie, but let’s be honest, using the basic setup, you are more often than not kicked out of the scraping process half way through. There are a number of potential reasons for your scrape being unsuccessful, such as a drop or time out in your internet connection, however by far the most likely reason is that you have not signed up to the ScreenScraper website and added your user details to Skyscraper config file.

In the following article we will walk you through the in’s and out’s of signing up to ScreenScraper as well as a few other pitfalls that may disrupt your scraping experience. One thing to note is that this article assumes that you use ScreeScraper as your scraping database when performing a scrape with Skyscraper. This is recommended as it is by far the most diverse and complete games metadata and media resource on offer at this time. Read on and all your scraping headaches should disappear.

How To Register On The ScreenScraper Website

If you attempt to scrape your ROMs using just the basic settings in Skyscraper, you will invariably notice that you get kicked out of the process midway through and you see a message stating something along the lines that the ScreenScraper service has stopped running as the server is overloaded and priority is given to registered users.

The reason behind this is that screenscraper.fr (the source from which you are pulling your assets) is by far the best repository for ROM images, video clips, marquees etc. This being the case the sites resources are in high demand, which consistently puts strain on the site’s servers. The more visitors a site receives, the harder it has to work. These servers then have to work even harder if the visitors are downloading assets from the server. It is easy to find out if the ScreenScraper server is under stress. The ScreenScraper site actually displays the server loads they are currently experiencing at the top of their homepage so you can keep up to date on it 24/7.

So, one way to avoid, or at least reduce your chances of you being kicked out of the scraping process halfway through, is to sign up for a free account with ScreenScraper and then enter your login details into the Skyscraper config.ini file in RetroPie. ScreenScraper gives preferential scraping privileges to anyone who is a member of their site. Even more privileges are given to members who contribute to ScreenScraper through Patreon. Extra privileges include an increased number of scrape allocations per day (typically one ROM requires roughly 4 requests to acquire all available metadata and media) as well as granting users more streams through which to download the assets. This helps increase the speed of the scraping process by essentially allowing you to perform multiple scrapes simultaneously. Below is a table which illustrates which privileges are granted to each level. Also, you should bear in mind that each contribution level has priority over the contribution levels beneath them.

Contribution	Scrape requests per day	No. of scraping threads available	Status
Not registered	10,000	1	None
Registered	20,000	1	None
1€ Donation per month	50,000	1	Bronze Financial Contributor
2€ Donation per month	50,000	1	Silver Financial Contributor
5€ Donation per month	50,000	2	Gold Financial Contributor
10€ Donation per month	50,000	5	Carbon Financial Contributor

So, how do you go about signing up to ScreenScraper? First off, as mentioned, you need to sign up for an account on the ScreenScraper website. This may sound like a straightforward process however there are some elements of the site that are a bit misleading. You can’t register on the home page but this is not made very clear on the site (I tried registering my details a number of times using the form at the top right of the home page which is actually the sign in form). The page you need to visit in order to sign up is screenscraper.fr/membership. On this page you then need to scroll down below the fold until you see the ‘Sign up on ScreenScraper’ box.

Enter your details into the form and click send to register. You will then be sent an activation email where you will need to copy and paste a link into your browser (apparently ScreenScaper haven’t yet discovered hyperlinks). That’s this step completed. Next you will need to get your hands dirty with a small amount of coding, but don’t worry. We’ll walk you through every step of the way.

Adding Your ScreenScraper Info To Skyscraper

We’re done with websites for the time being. Now it’s time to return to your Raspberry Pi. In RetroPie you will need to hit F4 in order to access the RetroPie terminal. this is the command line of RetroPie where you can access many features not available through EmulationStation. Once in the terminal enter the line of code below to open the Skyscraper config.ini file in the nano text editor.

Once in the editor use the down arrow key on your keyboard to move the cursor to the bottom of the file. Press the Enter key to add an extra line and enter your user credentials exactly as illustrated below where USER is your nickname and PASSWORD is obviously your password:

N.B. Make sure you enter your nickname and not your email address as this will not work. I fell into this trap and it took me several hours to work out why Skyscraper was failing to connect to my account.

That’s all you need to do. Exit the nano editor by pressing Ctrl+x, answer the save question with y, and press Enter to confirm the file name. You will then be transferred back to the RetroPie terminal.

At this point, you are ready to scrape your ROMs again, with the added security that your scrape now has a better chance of succeeding.

How To Avoid Cache Overflow When Scraping

If you have a large number of ROMs to scrape (I’m talking in the region of several thousand plus) it is very possible that at some stage during your scraping process you will notice that you are seeing an error message similar to that below and your ROMs are no longer being scraped, although it would appear that the process is still ongoing.

Error message – Warning! Couldn’t add resource to cache. Have you run out of disk space?

The reason behind this is that the available disk space used for storing cached media has been exhausted. What’s that? You say that you store your ROMs on a mammoth sized external hard drive? Well that doesn’t matter. By default the cache files are stored on the Raspberry Pi’s user home directory and not with the ROMs, meaning the cache storage folder resides on that tiny micro SD card that you only intended for the RetroPie operating software.

Fear not! There is a solution to this issue, and if you are storing your ROMs on an external hard drive then the hard bit is already done. All you need to do is specify the folder to which your cache files are written to. This can be done in the Skyscraper config.ini file.

Specifying a new folder for your cache files is simple. Bring up the Skyscraper config.ini file again using the method outlined earlier in this article, and just above the code that you entered to specify your ScreenScraper user credentials enter the code below:

So, if you have followed the steps to add your ScreenScraper user credentials, your completed code should look similar to that below:

From now on, whenever you scrape any ROMs, your cache files will be stored in the skyscrapercache folder which resides within the retropie-mount folder. The same retropie-mount folder that you created when you set up your external USB storage device. Issue sorted!

Skipped ROMs And Where To Find A List Of Them

The more ROMs you scrape, the more likely it is that some of them will fall through the Skyscraper net and not have any metadata or media attributed to them. The reasons behind this can be numerous but it is generally to do with an inaccurate file name or that the game is the product of someone’s home brew efforts and is not actually listed in any system game lists. It is also possible, although unlikely that the game has not yet been indexed into the ScreenScraper databases.

Regardless of what the reason may be, the issue remains the same; you have some ROMs which still need to be scraped. But how do you find out which ROMs are missing their info and media, other than to wade through every single game title in your EmulationStation games lists? Well conveniently enough, at the end of every scrape, Skyscraper creates a text file that contains a list of every ROM that it has had to skip. These files can be found in the /home/USER/.skyscraper/ folder and are categorised by platform. An example of a filename would be skipped-amstradcpc-screenscraper.txt. All you need to do is open the txt file to view the list. You can then decide if you want to manually scrape the skipped ROMs or try scraping with a different database.

Related Questions

What else can I do to customise my RetroPie build? There are a vast number of features you can configure in RetroPie to turn a generic device into your own personalised retro gaming machine. These features range from adding any number of playable retro games to expand your portfolio, to changing the theme of your front end menus and even adding music to enhance your experience. Generally if you can think of something you want to change, its likely that it can be done.

Can I use the native RetroPie scraper alongside of Skyscraper? Yes. As a matter of fact I would recommend this as part of your scraping process. While Skyscraper is, in my opinion, the best scraper available it is not adapted well to scraping individual ROMs. I would suggest using Skyscraper for scraping the bulk of your ROMs then use the RetroPie scraper to mop up any ROMs that Skyscraper has had to skip.