27 Commits

Author SHA1 Message Date
Godzil
141bdccf02 1.3.7 2018-07-30 22:47:38 +01:00
Godzil
4990effa1c Try to fix #81 and probably some other issues when the URL is not valid to properly display that the URL is not valid. Also change a bit on how error are handled 2018-07-30 22:47:38 +01:00
Godzil
2459f342c5 Force debug file to be written synchronously 2018-07-30 22:47:38 +01:00
Godzil
d68a2b7bce Update dependencies 2018-07-30 22:47:38 +01:00
Godzil
69d5ceac36 Remove useless ignore in .gitignore 2018-07-30 22:47:38 +01:00
Manoël Trapier
cf7039400c Update readme with more usefull examples 2018-07-30 22:47:38 +01:00
Godzil
02a9d763cd Add the episode title in the default file name template. 2018-07-30 22:47:38 +01:00
Godzil
d549d46979 1.3.6 2018-07-30 22:47:37 +01:00
Godzil
3f5b4b2585 Update readme 2018-07-30 22:47:37 +01:00
Godzil
1d596b02f7 Cleaning up the command line parameter to properly use default values 2018-07-30 22:47:37 +01:00
Godzil
cee53fb113 Fix for #78 (and a bit of cleanup) 2018-07-30 22:47:37 +01:00
Godzil
1e56cab73f Move error displaying when downloading an episode fail. 2018-07-30 22:47:37 +01:00
Godzil
0dc3c1e8e2 Update a bit the bug report template
(Commit #200!)
2018-07-30 22:47:37 +01:00
Godzil
0124e38a89 1.3.5 2018-07-30 22:47:36 +01:00
Godzil
6765b517ec Add a new episode filter and completely remove some dependencies on the config object. 2018-07-30 22:47:36 +01:00
Godzil
8c1e0f2e0c Stop messing with the config objet 2018-07-30 22:47:36 +01:00
Godzil
817843c40c Add more output to debug.txt 2018-07-30 22:47:36 +01:00
Godzil
04b22fdce5 Update readme file 2018-07-30 22:47:36 +01:00
Godzil
eb15d7d854 1.3.4 2018-07-30 22:47:36 +01:00
Godzil
66670547b9 Add a crude debug mechanism 2018-07-30 22:47:36 +01:00
Godzil
987e424324 Force an exit if authentication failed! 2018-07-30 22:47:35 +01:00
Godzil
523c780b18 Force to use a user account 2018-07-30 22:47:35 +01:00
Godzil
6c2100fbff Give access to the config objet to the subtitles for future changes. 2018-07-30 22:47:35 +01:00
Godzil
f10bead0dc Remove episode and volume filter, they were buggy and useless.
Use the @URL syntax do download a single episode.
2018-07-30 22:47:35 +01:00
Godzil
6448f4ec97 1.3.3 2018-07-30 22:47:35 +01:00
Godzil
829bb080ee Small update on the bug report template 2018-07-30 22:47:35 +01:00
Godzil
5edd7cf05a Fix 4 silly bugs 2018-07-30 22:47:35 +01:00
16 changed files with 2381 additions and 355 deletions

View File

@@ -14,9 +14,12 @@ A clear and concise description of what you expected to happen.
If applicable, add screenshots to help explain your problem.
**Please fill theses informations:**
(Add a X between brackets to make them ticked)
(Add a X between brackets to make them ticked if relevant)
- OS: [e.g:. Windows 10, Mac OS X 10.13, ...]
- [ ] I'm using the latest version of Crunchy
- [ ] I have a premium accrount on CR
- [ ] I am using a VPN
- My region in the world (country or continent):
- Serie you get a problem with (and specify which episode if it is specific to one):
- The command line you are running Crunchy with:
- The message Crunchy is giving you, if any:
@@ -28,4 +31,5 @@ If applicable, add screenshots to help explain your problem.
Add any other context about the problem here.
_Also don't hesitate to add labels you feel apropriate on your report._
_Also don't hesitate to add labels you feel apropriate on your report._
_Please don't edit logs if you are adding them, apart from removing sensitive informations like login/password_

1
.gitignore vendored
View File

@@ -1,3 +1,2 @@
dist/
node_modules/
typings/

View File

@@ -12,11 +12,11 @@
This application is not endorsed or affliated with *CrunchyRoll*. The usage of this application enables episodes to be downloaded for offline convenience which may be forbidden by law in your country. Usage of this application may also cause a violation of the agreed *Terms of Service* between you and the stream provider. A tool is not responsible for your actions; please make an informed decision prior to using this application.
**PLEASE _ONLY_ USE THIS TOOL IF YOU HAVE A _PREMIUM ACCOUNT_**
**_ONLY_ USE THIS TOOL IF YOU HAVE A _PREMIUM ACCOUNT_**
## Configuration
It is recommended to enable authentication (`-p` and `-u`) so your account permissions and settings are available for use. It is not possible to download non-free material without an account and premium subscription. Furthermore, the default account settings are used when downloading. If you want the highest quality videos, configure these preferences at https://www.crunchyroll.com/acct/?action=video.
You need to authentication (`-p` and `-u`) to use Crunchy so you need to have an account on *CrunchyRool*. It is not possible to download non-free material without an account and premium subscription.
## Prerequisites
@@ -66,42 +66,75 @@ The [command-line interface](http://en.wikipedia.org/wiki/Command-line_interface
Options:
-V, --version output the version number
-p, --pass <s> The password.
-u, --user <s> The e-mail address or username.
-c, --cache Disables the cache.
-m, --merge Disables merging subtitles and videos.
-e, --episode <i> The episode filter.
-v, --volume <i> The volume filter.
-f, --format <s> The subtitle format. (Default: ass)
-o, --output <s> The output path.
-s, --series <s> The series override.
-n, --filename <s> The name override.
-t, --tag <s> The subgroup. (Default: CrunchyRoll) (default: CrunchyRoll)
-r, --resolution <s> The video resolution. (Default: 1080 (360, 480, 720, 1080)) (default: 1080)
-g, --rebuildcrp Rebuild the crpersistant file.
-b, --batch <s> Batch file (default: CrunchyRoll.txt)
--verbose Make tool verbose
--retry <i> Number or time to retry fetching an episode. Default: 5 (default: 5)
-h, --help output usage information
-V, --version output the version number
-p, --pass <s> The password.
-u, --user <s> The e-mail address or username.
-c, --cache Disables the cache.
-m, --merge Disables merging subtitles and videos.
-e, --episodes <s> Episode list. Read documentation on how to use
-f, --format <s> The subtitle format. (default: ass)
-o, --output <s> The output path.
-s, --series <s> The series name override.
-n, --nametmpl <s> Output name template (default: {SERIES_TITLE} - s{SEASON_NUMBER}e{EPISODE_NUMBER} - [{TAG}])
-t, --tag <s> The subgroup. (default: CrunchyRoll)
-r, --resolution <s> The video resolution. (valid: 360, 480, 720, 1080) (default: 1080)
-b, --batch <s> Batch file (default: CrunchyRoll.txt)
--verbose Make tool verbose
--rebuildcrp Rebuild the crpersistant file.
--retry <i> Number or time to retry fetching an episode. (default: 5)
-h, --help output usage information
#### Batch-mode
When no sequence of series addresses is provided, the batch-mode source file will be read (which is *CrunchyRoll.txt* in the current work directory. Each line in this file is processed as a seperate command-line statement. This makes it ideal to manage a large sequence of series addresses with variating command-line options or incremental episode updates.
When no sequence of series addresses is provided, the batch-mode source file will be read (which is *CrunchyRoll.txt* in the current work directory. Each line in this file is processed contain the URL of a series and can support some of the command line parameter (like `-e`). This makes it ideal to manage a large sequence of series addresses.
#### Examples
Download in batch-mode:
crunchy
You will need to create the batch file (default name is `CrunchyRoll.txt`):
Download *Fairy Tail* to the current work directory:
http://www.cr.com/tail-fairy
http://www.cr.com/gin-mama
http://www.cr.com/two-parts
// Just download episodes 3 to 42
http://www.cr.com/defense-of-dwarfs -e 3-42
crunchy http://www.crunchyroll.com/fairy-tail
Then launch crunchy:
crunchy -u login -p password http://www.cr.com/tail-fairy
Download *Tail Fairy* to the current work directory:
crunchy -u login -p password http://www.cr.com/tail-fairy
Download *Tail Fairy* to `C:\Anime`:
crunchy -u login -p password --output C:\Anime http://www.cr.com/tail-fairy
Download episode 42 of *Tail Fairy* to `C:\Anime`:
crunchy -u login -p password --output C:\Anime @http://www.cr.com/tail-fairy/episode-42-the-episode-which-dont-exist-665544
*Notice the '@' in front of the URL, it is there to tell Crunchy that the URL is an episode URL and not a series URL.*
or
crunchy -u login -p password --output C:\Anime http://www.cr.com/tail-fairy -e 42
Download episode 10 to 42 (both included) of *Tail Fairy*:
crunchy -u login -p password http://www.cr.com/tail-fairy -e 10-42
Download episode up to 42 (included) of *Tail Fairy*:
crunchy -u login -p password http://www.cr.com/tail-fairy -e -42
Download episodes starting from 42 to the last available of *Tail Fairy*:
crunchy -u login -p password http://www.cr.com/tail-fairy -e 42-
Download *Fairy Tail* to `C:\Anime`:
crunchy --output C:\Anime http://www.crunchyroll.com/fairy-tail
#### Command line parameters
@@ -117,15 +150,9 @@ Download *Fairy Tail* to `C:\Anime`:
* `-c or --cache` disables the cache in batch mode.
* `-m or --merge` disables merging subtitles and videos.
##### Filters
* `-e or --episode <i>` filters episodes (positive is greater than, negative is smaller than).
* `-v or --volume <i>` filters volumes (positive is greater than, negative is smaller than).
_These parameters are probably extremely buggy at the moment..._
##### Settings
* `-e or --episodes <s>` set an episode
* `-f or --format <s>` sets the subtitle format. (Default: ass)
* `-o or --output <s>` sets the output path.
* `-s or --series <s>` sets the series override.

2257
package-lock.json generated

File diff suppressed because it is too large Load Diff

View File

@@ -15,19 +15,18 @@
"engines": {
"node": ">=5.0"
},
"version": "1.3.2",
"version": "1.3.7",
"bin": {
"crunchy": "./bin/crunchy",
"crunchy.sh": "./bin/crunchy.sh"
},
"dependencies": {
"@types/node": "^10.3.3",
"big-integer": "^1.6.31",
"big-integer": "^1.6.32",
"bluebird": "^3.5.1",
"cheerio": "^0.22.0",
"cloudscraper": "^1.5.0",
"commander": "^2.15.1",
"fs-extra": "^6.0.1",
"commander": "^2.16.0",
"fs-extra": "^7.0.0",
"mkdirp": "^0.5.0",
"pjson": "^1.0.9",
"request": "^2.87.0",
@@ -35,15 +34,17 @@
"xml2js": "^0.4.5"
},
"devDependencies": {
"@types/bluebird": "^3.5.20",
"@types/cheerio": "^0.22.7",
"@types/fs-extra": "^5.0.3",
"@types/bluebird": "^3.5.23",
"@types/cheerio": "^0.22.8",
"@types/fs-extra": "^5.0.4",
"@types/mkdirp": "^0.5.2",
"@types/node": "^10.5.3",
"@types/request": "^2.47.1",
"@types/request-promise": "^4.1.41",
"@types/request-promise": "^4.1.42",
"@types/xml2js": "^0.4.3",
"npm-check": "^5.7.1",
"tsconfig-lint": "^0.12.0",
"tslint": "^5.10.0",
"tslint": "^5.11.0",
"typescript": "^2.9.2"
},
"scripts": {

View File

@@ -44,6 +44,17 @@ export default function(args: string[], done: (err?: Error) => void)
config.video_quality = resol_table['1080'].quality;
}
if (config.debug)
{
/* Ugly but meh */
const tmp = JSON.parse(JSON.stringify(config));
tmp.pass = 'obfuscated';
tmp.user = 'obfustated';
tmp.rawArgs = undefined;
tmp.options = undefined;
log.dumpToDebug('Config', JSON.stringify(tmp), true);
}
tasks(config, batchPath, (err, tasksArr) =>
{
if (err)
@@ -51,6 +62,11 @@ export default function(args: string[], done: (err?: Error) => void)
return done(err);
}
if (tasksArr[0].address === '')
{
return done();
}
let i = 0;
(function next()
@@ -60,22 +76,43 @@ export default function(args: string[], done: (err?: Error) => void)
return done();
}
series(tasksArr[i].config, tasksArr[i].address, (errin) =>
if (config.debug)
{
log.dumpToDebug('Task ' + i, JSON.stringify(tasksArr[i]));
}
series(config, tasksArr[i], (errin) =>
{
if (errin)
{
if (errin.error)
{
/* Error from the request, so ignore it */
tasksArr[i].retry = 0;
}
if (tasksArr[i].retry <= 0)
{
console.error(err);
log.error(JSON.stringify(errin));
if (config.debug)
{
log.dumpToDebug('BatchGiveUp', JSON.stringify(errin));
}
log.error('Cannot get episodes from "' + tasksArr[i].address + '", please rerun later');
/* Go to the next on the list */
i += 1;
}
else
{
if (config.verbose)
{
console.error(err);
log.error(JSON.stringify(errin));
}
log.warn('Retrying to fetch episodes ' + tasksArr[i].retry + ' / ' + config.retry);
if (config.debug)
{
log.dumpToDebug('BatchRetry', JSON.stringify(errin));
}
log.warn('Retrying to fetch episodes list from' + tasksArr[i].retry + ' / ' + config.retry);
tasksArr[i].retry -= 1;
}
}
@@ -123,6 +160,73 @@ function split(value: string): string[]
return pieces;
}
function get_min_filter(filter: string): number
{
if (filter !== undefined)
{
const tok = filter.split('-');
if (tok.length > 2)
{
log.error('Invalid episode filter \'' + filter + '\'');
process.exit(-1);
}
if (tok[0] !== '')
{
return parseInt(tok[0], 10);
}
}
return 0;
}
function get_max_filter(filter: string): number
{
if (filter !== undefined)
{
const tok = filter.split('-');
if (tok.length > 2)
{
log.error('Invalid episode filter \'' + filter + '\'');
process.exit(-1);
}
if ((tok.length > 1) && (tok[1] !== ''))
{
/* We have a max value */
return parseInt(tok[1], 10);
}
else if ((tok.length === 1) && (tok[0] !== ''))
{
/* A single episode has been requested */
return parseInt(tok[0], 10);
}
}
return +Infinity;
}
/**
* Check that URL start with http:// or https://
* As for some reason request just return an error but a useless one when that happen so check it
* soon enough.
*/
function checkURL(address: string): boolean
{
if (address.startsWith('http:\/\/'))
{
return true;
}
if (address.startsWith('http:\/\/'))
{
return true;
}
log.error('URL ' + address + ' miss \'http:\/\/\' or \'https:\/\/\' => will be ignored');
return false;
}
/**
* Parses the configuration or reads the batch-mode file for tasks.
*/
@@ -130,11 +234,15 @@ function tasks(config: IConfigLine, batchPath: string, done: (err: Error, tasks?
{
if (config.args.length)
{
const configIn = config;
return done(null, config.args.map((addressIn) =>
{
return {address: addressIn, config: configIn, retry: config.retry};
if (checkURL(addressIn))
{
return {address: addressIn, retry: config.retry,
episode_min: get_min_filter(config.episodes), episode_max: get_max_filter(config.episodes)};
}
return {address: '', retry: 0, episode_min: 0, episode_max: 0};
}));
}
@@ -170,7 +278,11 @@ function tasks(config: IConfigLine, batchPath: string, done: (err: Error, tasks?
return;
}
map.push({address: addressIn, config: lineConfig, retry: config.retry});
if (checkURL(addressIn))
{
map.push({address: addressIn, retry: lineConfig.retry,
episode_min: get_min_filter(lineConfig.episodes), episode_max: get_max_filter(lineConfig.episodes)});
}
});
});
done(null, map);
@@ -190,20 +302,19 @@ function parse(args: string[]): IConfigLine
// Disables
.option('-c, --cache', 'Disables the cache.')
.option('-m, --merge', 'Disables merging subtitles and videos.')
// Filters
.option('-e, --episode <i>', 'The episode filter.')
.option('-v, --volume <i>', 'The volume filter.')
// Episode filter
.option('-e, --episodes <s>', 'Episode list. Read documentation on how to use')
// Settings
.option('-f, --format <s>', 'The subtitle format. (Default: ass)')
.option('-f, --format <s>', 'The subtitle format.', 'ass')
.option('-o, --output <s>', 'The output path.')
.option('-s, --series <s>', 'The series override.')
.option('-n, --filename <s>', 'The name override.')
.option('-t, --tag <s>', 'The subgroup. (Default: CrunchyRoll)', 'CrunchyRoll')
.option('-r, --resolution <s>', 'The video resolution. (Default: 1080 (360, 480, 720, 1080))',
'1080')
.option('-g, --rebuildcrp', 'Rebuild the crpersistant file.')
.option('-s, --series <s>', 'The series name override.')
.option('-n, --nametmpl <s>', 'Output name template', '{SERIES_TITLE} - s{SEASON_NUMBER}e{EPISODE_NUMBER} - {EPISODE_TITLE} - [{TAG}]')
.option('-t, --tag <s>', 'The subgroup.', 'CrunchyRoll')
.option('-r, --resolution <s>', 'The video resolution. (valid: 360, 480, 720, 1080)', '1080')
.option('-b, --batch <s>', 'Batch file', 'CrunchyRoll.txt')
.option('--verbose', 'Make tool verbose')
.option('--retry <i>', 'Number or time to retry fetching an episode. Default: 5', 5)
.option('--debug', 'Create a debug file. Use only if requested!')
.option('--rebuildcrp', 'Rebuild the crpersistant file.')
.option('--retry <i>', 'Number or time to retry fetching an episode.', 5)
.parse(args);
}

View File

@@ -74,11 +74,10 @@ function sanitiseFileName(str: string)
*/
function download(config: IConfig, page: IEpisodePage, player: IEpisodePlayer, done: (err: Error, ign: boolean) => void)
{
let series = config.series || page.series;
const serieFolder = sanitiseFileName(config.series || page.series);
series = sanitiseFileName(series);
let fileName = sanitiseFileName(name(config, page, series, ''));
let filePath = path.join(config.output || process.cwd(), series, fileName);
let fileName = sanitiseFileName(generateName(config, page));
let filePath = path.join(config.output || process.cwd(), serieFolder, fileName);
if (fileExist(filePath + '.mkv'))
{
@@ -95,13 +94,13 @@ function download(config: IConfig, page: IEpisodePage, player: IEpisodePlayer, d
do
{
count = count + 1;
fileName = sanitiseFileName(name(config, page, series, '-' + count));
filePath = path.join(config.output || process.cwd(), series, fileName);
fileName = sanitiseFileName(generateName(config, page, '-' + count));
filePath = path.join(config.output || process.cwd(), serieFolder, fileName);
} while (fileExist(filePath + '.mkv'));
log.warn('Renaming to \'' + fileName + '\'...');
config.filename = fileName;
page.filename = fileName;
}
if (config.rebuildcrp)
@@ -114,6 +113,7 @@ function download(config: IConfig, page: IEpisodePage, player: IEpisodePlayer, d
{
if (errM)
{
log.dispEpisode(fileName, 'Error...', true);
return done(errM, false);
}
@@ -122,6 +122,7 @@ function download(config: IConfig, page: IEpisodePage, player: IEpisodePlayer, d
{
if (errDS)
{
log.dispEpisode(fileName, 'Error...', true);
return done(errDS, false);
}
@@ -133,6 +134,7 @@ function download(config: IConfig, page: IEpisodePage, player: IEpisodePlayer, d
{
if (errDV)
{
log.dispEpisode(fileName, 'Error...', true);
return done(errDV, false);
}
@@ -148,6 +150,7 @@ function download(config: IConfig, page: IEpisodePage, player: IEpisodePlayer, d
{
if (errVM)
{
log.dispEpisode(fileName, 'Error...', true);
return done(errVM, false);
}
@@ -186,7 +189,7 @@ function downloadSubtitle(config: IConfig, player: IEpisodePlayer, filePath: str
const formats = subtitle.formats;
const format = formats[config.format] ? config.format : 'ass';
formats[format](data, (errF: Error, decodedSubtitle: string) =>
formats[format](config, data, (errF: Error, decodedSubtitle: string) =>
{
if (errF)
{
@@ -211,19 +214,16 @@ function downloadVideo(config: IConfig, page: IEpisodePage, player: IEpisodePla
/**
* Names the file based on the config, page, series and tag.
*/
function name(config: IConfig, page: IEpisodePage, series: string, extra: string)
function generateName(config: IConfig, page: IEpisodePage, extra = '')
{
const episodeNum = parseInt(page.episode, 10);
const volumeNum = parseInt(page.volume, 10);
const episode = (episodeNum < 10 ? '0' : '') + page.episode;
const volume = (volumeNum < 10 ? '0' : '') + page.volume;
const tag = config.tag || 'CrunchyRoll';
const series = config.series || page.series;
if (!config.filename) {
return page.series + ' - s' + volume + 'e' + episode + ' - [' + tag + ']' + extra;
}
return config.filename
return config.nametmpl
.replace(/{EPISODE_ID}/g, page.id.toString())
.replace(/{EPISODE_NUMBER}/g, episode)
.replace(/{SEASON_NUMBER}/g, volume)
@@ -277,10 +277,21 @@ function scrapePage(config: IConfig, address: string, done: (err: Error, page?:
const episodeTitle = $('#showmedia_about_name').text().replace(/[“”]/g, '');
const data = regexp.exec(look);
if (config.debug)
{
log.dumpToDebug('episode page', $.html());
}
if (!swf || !data)
{
log.warn('Somethig unexpected in the page at ' + address + ' (data are: ' + look + ')');
log.warn('Setting Season to 0 and episode to 0...');
if (config.debug)
{
log.dumpToDebug('episode unexpected', look);
}
done(null, {
episode: '0',
id: epId,
@@ -289,6 +300,7 @@ function scrapePage(config: IConfig, address: string, done: (err: Error, page?:
title: episodeTitle,
swf: swf[1],
volume: '0',
filename: '',
});
}
else
@@ -301,6 +313,7 @@ function scrapePage(config: IConfig, address: string, done: (err: Error, page?:
title: episodeTitle,
swf: swf[1],
volume: data[2] || '1',
filename: '',
});
}
});
@@ -367,6 +380,11 @@ function scrapePlayer(config: IConfig, address: string, id: number, done: (err:
});
} catch (parseError)
{
if (config.debug)
{
log.dumpToDebug('player scrape', parseError);
}
done(parseError);
}
});

View File

@@ -5,14 +5,12 @@ interface IConfig {
// Disables
cache?: boolean;
merge?: boolean;
// Filters
episode?: number;
volume?: number;
episodes?: string;
// Settings
format?: string;
output?: string;
series?: string;
filename?: string;
nametmpl?: string;
tag?: string;
resolution?: string;
video_format?: string;
@@ -20,5 +18,6 @@ interface IConfig {
rebuildcrp?: boolean;
batch?: string;
verbose?: boolean;
debug?: boolean;
retry?: number;
}

View File

@@ -1,5 +1,6 @@
interface IConfigTask {
address: string;
config: IConfigLine;
retry: number;
episode_min: number;
episode_max: number;
}

View File

@@ -6,4 +6,5 @@ interface IEpisodePage {
season: string;
title: string;
swf: string;
filename: string;
}

View File

@@ -1,3 +1,3 @@
interface IFormatterTable {
[key: string]: (input: string|Buffer, done: (err: Error, subtitle?: string) => void) => void;
[key: string]: (config: IConfig, input: string|Buffer, done: (err: Error, subtitle?: string) => void) => void;
}

View File

@@ -1,7 +1,8 @@
'use strict';
import os = require('os');
import fs = require('fs-extra');
export function error(str: string)
export function error(str: string|Error)
{
/* Do fancy output */
console.error(' \x1B[1;31m* ERROR\x1B[0m: ' + str);
@@ -35,3 +36,14 @@ export function dispEpisode(name: string, status: string, addNL: boolean)
console.log('');
}
}
export function dumpToDebug(what: string, data: any, create = false)
{
if (create)
{
fs.writeFileSync('debug.txt', '>>>>>>>> ' + what + ':\n' + data + '\n<<<<<<<<\n');
return;
}
fs.appendFileSync('debug.txt', '>>>>>>>> ' + what + ':\n' + data + '\n<<<<<<<<\n');
}

View File

@@ -82,7 +82,7 @@ function login(sessionId: string, user: string, pass: string): Promise<any>
/**
* Performs a GET request for the resource.
*/
export function get(config: IConfig, options: string|request.Options, done: (err: Error, result?: string) => void)
export function get(config: IConfig, options: string|request.Options, done: (err: any, result?: string) => void)
{
authenticate(config, (err) =>
{
@@ -91,7 +91,7 @@ export function get(config: IConfig, options: string|request.Options, done: (err
return done(err);
}
cloudscraper.request(modify(options, 'GET'), (error: Error, response: any, body: any) =>
cloudscraper.request(modify(options, 'GET'), (error: any, response: any, body: any) =>
{
if (error) return done(error);
done(null, typeof body === 'string' ? body : String(body));
@@ -127,9 +127,15 @@ export function post(config: IConfig, options: request.Options, done: (err: Erro
*/
function authenticate(config: IConfig, done: (err: Error) => void)
{
if (isAuthenticated || !config.pass || !config.user)
if (isAuthenticated)
{
return done(null);
return done(null);
}
if (!config.pass || !config.user)
{
log.error('You need to give login/password to use Crunchy');
process.exit(-1);
}
startSession()
@@ -181,7 +187,8 @@ function authenticate(config: IConfig, done: (err: Error) => void)
if (isAuthenticated === false)
{
const error = $('ul.message, li.error').text();
return done(new Error('Authentication failed: ' + error));
log.error('Authentication failed: ' + error);
process.exit(-1);
}
if (isPremium === false)
@@ -190,7 +197,7 @@ function authenticate(config: IConfig, done: (err: Error) => void)
}
else
{
log.info('You have a premium account! Good!');
log.info('You have a premium account! Good!');
}
done(null);
});
@@ -216,4 +223,4 @@ function modify(options: string|request.Options, reqMethod: string): request.Opt
url: options.toString(),
method: reqMethod
};
}
}

View File

@@ -27,7 +27,7 @@ function fileExist(path: string)
/**
* Streams the series to disk.
*/
export default function(config: IConfig, address: string, done: (err: Error) => void)
export default function(config: IConfig, task: IConfigTask, done: (err: any) => void)
{
const persistentPath = path.join(config.output || process.cwd(), persistent);
@@ -41,34 +41,59 @@ export default function(config: IConfig, address: string, done: (err: Error) =>
{
const cache = config.cache ? {} : JSON.parse(contents || '{}');
page(config, address, (errP, page) =>
pageScrape(config, task, (errP, page) =>
{
if (errP)
{
const reqErr = errP.error;
if ((reqErr.syscall === 'getaddrinfo') && (reqErr.errno === 'ENOTFOUND'))
{
log.error('The URL \'' + task.address + '\' is invalid, please check => I\'m ignoring it.');
}
return done(errP);
}
let i = 0;
(function next()
{
if (config.debug)
{
log.dumpToDebug('Episode ' + i, JSON.stringify(page.episodes[i]));
}
if (i >= page.episodes.length) return done(null);
download(cache, config, address, page.episodes[i], (errD, ignored) =>
download(cache, config, task, page.episodes[i], (errD, ignored) =>
{
if (errD)
{
/* Check if domain is valid */
const reqErr = errD.error;
if ((reqErr.syscall === 'getaddrinfo') && (reqErr.errno === 'ENOTFOUND'))
{
page.episodes[i].retry = 0;
log.error('The URL \'' + task.address + '\' is invalid, please check => I\'m ignoring it.');
}
if (page.episodes[i].retry <= 0)
{
log.dispEpisode(config.filename, 'Error...', true);
console.error(err);
log.error(JSON.stringify(errD));
log.error('Cannot fetch episode "s' + page.episodes[i].volume + 'e' + page.episodes[i].episode +
'", please rerun later');
/* Go to the next on the list */
i += 1;
}
else
{
log.dispEpisode(config.filename, 'Error...', true);
if (config.verbose)
if ((config.verbose) || (config.debug))
{
console.error(errD);
if (config.debug)
{
log.dumpToDebug('series address', task.address);
log.dumpToDebug('series error', JSON.stringify(errD));
log.dumpToDebug('series data', JSON.stringify(page));
}
log.error(errD);
}
log.warn('Retrying to fetch episode "s' + page.episodes[i].volume + 'e' + page.episodes[i].episode +
'" - Retry ' + page.episodes[i].retry + ' / ' + config.retry);
@@ -108,15 +133,18 @@ export default function(config: IConfig, address: string, done: (err: Error) =>
* Downloads the episode.
*/
function download(cache: {[address: string]: number}, config: IConfig,
baseAddress: string, item: ISeriesEpisode,
done: (err: Error, ign: boolean) => void)
task: IConfigTask, item: ISeriesEpisode,
done: (err: any, ign: boolean) => void)
{
if (!filter(config, item))
const episodeNumber = parseInt(item.episode, 10);
if ( (episodeNumber < task.episode_min) ||
(episodeNumber > task.episode_max) )
{
return done(null, false);
}
const address = url.resolve(baseAddress, item.address);
const address = url.resolve(task.address, item.address);
if (cache[address])
{
@@ -135,41 +163,17 @@ function download(cache: {[address: string]: number}, config: IConfig,
});
}
/**
* Filters the item based on the configuration.
*/
function filter(config: IConfig, item: ISeriesEpisode)
{
// Filter on chapter.
const episodeFilter = config.episode;
// Filter on volume.
const volumeFilter = config.volume;
const currentEpisode = parseInt(item.episode, 10);
const currentVolume = item.volume;
if ( ( (episodeFilter > 0) && (currentEpisode <= episodeFilter) ) ||
( (episodeFilter < 0) && (currentEpisode >= -episodeFilter) ) ||
( (volumeFilter > 0) && (currentVolume <= volumeFilter ) ) ||
( (volumeFilter < 0) && (currentVolume >= -volumeFilter ) ) )
{
return false;
}
return true;
}
/**
* Requests the page and scrapes the episodes and series.
*/
function page(config: IConfig, address: string, done: (err: Error, result?: ISeries) => void)
function pageScrape(config: IConfig, task: IConfigTask, done: (err: any, result?: ISeries) => void)
{
if (address[0] === '@')
if (task.address[0] === '@')
{
log.info('Trying to fetch from ' + address.substr(1));
log.info('Trying to fetch from ' + task.address.substr(1));
const episodes: ISeriesEpisode[] = [];
episodes.push({
address: address.substr(1),
address: task.address.substr(1),
episode: '',
volume: 0,
retry: config.retry,
@@ -179,16 +183,26 @@ function page(config: IConfig, address: string, done: (err: Error, result?: ISer
else
{
let episodeCount = 0;
my_request.get(config, address, (err, result) => {
if (err) {
my_request.get(config, task.address, (err, result) => {
if (err)
{
return done(err);
}
const $ = cheerio.load(result);
const title = $('span[itemprop=name]').text();
if (config.debug)
{
log.dumpToDebug('serie page', $.html());
}
if (!title) {
return done(new Error('Invalid page.(' + address + ')'));
if (config.debug)
{
log.dumpToDebug('missing title', task.address);
}
return done(new Error('Invalid page.(' + task.address + ')'));
}
log.info('Checking availability for ' + title);

View File

@@ -4,7 +4,7 @@ import xml2js = require('xml2js');
/**
* Converts an input buffer to a SubStation Alpha subtitle.
*/
export default function(input: string|Buffer, done: (err: Error, subtitle?: string) => void)
export default function(config: IConfig, input: string|Buffer, done: (err: Error, subtitle?: string) => void)
{
xml2js.parseString(input.toString(), {
explicitArray: false,
@@ -18,9 +18,9 @@ export default function(input: string|Buffer, done: (err: Error, subtitle?: stri
try
{
done(null, script(xml) + '\n' +
done(null, script(config, xml) + '\n' +
style(xml.styles) + '\n' +
event(xml.events));
event(config, xml.events));
} catch (err)
{
done(err);
@@ -31,7 +31,7 @@ export default function(input: string|Buffer, done: (err: Error, subtitle?: stri
/**
* Converts the event block.
*/
function event(block: ISubtitleEvent): string
function event(config: IConfig, block: ISubtitleEvent): string
{
const format = 'Layer,Start,End,Style,Name,MarginL,MarginR,MarginV,Effect,Text';
@@ -51,10 +51,11 @@ function event(block: ISubtitleEvent): string
/**
* Converts the script block.
*/
function script(block: ISubtitle): string
function script(config: IConfig, block: ISubtitle): string
{
return '[Script Info]\n' +
'Origin: Downloaded from Crunchyroll.com by ' + config.user + '\n' +
'Title: ' + block.$.title + '\n' +
'ScriptType: v4.00+\n' +
'WrapStyle: ' + block.$.wrap_style + '\n' +

View File

@@ -4,7 +4,7 @@ import xml2js = require('xml2js');
/**
* Converts an input buffer to a SubRip subtitle.
*/
export default function(input: Buffer|string, done: (err: Error, subtitle?: string) => void)
export default function(config: IConfig, input: Buffer|string, done: (err: Error, subtitle?: string) => void)
{
const options = {explicitArray: false, explicitRoot: false};