Compare commits
18 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
141bdccf02 | ||
|
|
4990effa1c | ||
|
|
2459f342c5 | ||
|
|
d68a2b7bce | ||
|
|
69d5ceac36 | ||
|
|
cf7039400c | ||
|
|
02a9d763cd | ||
|
|
d549d46979 | ||
|
|
3f5b4b2585 | ||
|
|
1d596b02f7 | ||
|
|
cee53fb113 | ||
|
|
1e56cab73f | ||
|
|
0dc3c1e8e2 | ||
|
|
0124e38a89 | ||
|
|
6765b517ec | ||
|
|
8c1e0f2e0c | ||
|
|
817843c40c | ||
|
|
04b22fdce5 |
7
.github/ISSUE_TEMPLATE/bug_report.md
vendored
7
.github/ISSUE_TEMPLATE/bug_report.md
vendored
@@ -14,10 +14,12 @@ A clear and concise description of what you expected to happen.
|
|||||||
If applicable, add screenshots to help explain your problem.
|
If applicable, add screenshots to help explain your problem.
|
||||||
|
|
||||||
**Please fill theses informations:**
|
**Please fill theses informations:**
|
||||||
(Add a X between brackets to make them ticked)
|
(Add a X between brackets to make them ticked if relevant)
|
||||||
- OS: [e.g:. Windows 10, Mac OS X 10.13, ...]
|
- OS: [e.g:. Windows 10, Mac OS X 10.13, ...]
|
||||||
- [ ] I'm using the latest version of Crunchy
|
- [ ] I'm using the latest version of Crunchy
|
||||||
- [ ] I have a premium accrount on CR
|
- [ ] I have a premium accrount on CR
|
||||||
|
- [ ] I am using a VPN
|
||||||
|
- My region in the world (country or continent):
|
||||||
- Serie you get a problem with (and specify which episode if it is specific to one):
|
- Serie you get a problem with (and specify which episode if it is specific to one):
|
||||||
- The command line you are running Crunchy with:
|
- The command line you are running Crunchy with:
|
||||||
- The message Crunchy is giving you, if any:
|
- The message Crunchy is giving you, if any:
|
||||||
@@ -29,4 +31,5 @@ If applicable, add screenshots to help explain your problem.
|
|||||||
Add any other context about the problem here.
|
Add any other context about the problem here.
|
||||||
|
|
||||||
|
|
||||||
_Also don't hesitate to add labels you feel apropriate on your report._
|
_Also don't hesitate to add labels you feel apropriate on your report._
|
||||||
|
_Please don't edit logs if you are adding them, apart from removing sensitive informations like login/password_
|
||||||
1
.gitignore
vendored
1
.gitignore
vendored
@@ -1,3 +1,2 @@
|
|||||||
dist/
|
dist/
|
||||||
node_modules/
|
node_modules/
|
||||||
typings/
|
|
||||||
|
|||||||
84
README.md
84
README.md
@@ -12,11 +12,11 @@
|
|||||||
|
|
||||||
This application is not endorsed or affliated with *CrunchyRoll*. The usage of this application enables episodes to be downloaded for offline convenience which may be forbidden by law in your country. Usage of this application may also cause a violation of the agreed *Terms of Service* between you and the stream provider. A tool is not responsible for your actions; please make an informed decision prior to using this application.
|
This application is not endorsed or affliated with *CrunchyRoll*. The usage of this application enables episodes to be downloaded for offline convenience which may be forbidden by law in your country. Usage of this application may also cause a violation of the agreed *Terms of Service* between you and the stream provider. A tool is not responsible for your actions; please make an informed decision prior to using this application.
|
||||||
|
|
||||||
**PLEASE _ONLY_ USE THIS TOOL IF YOU HAVE A _PREMIUM ACCOUNT_**
|
**_ONLY_ USE THIS TOOL IF YOU HAVE A _PREMIUM ACCOUNT_**
|
||||||
|
|
||||||
## Configuration
|
## Configuration
|
||||||
|
|
||||||
It is recommended to enable authentication (`-p` and `-u`) so your account permissions and settings are available for use. It is not possible to download non-free material without an account and premium subscription. Furthermore, the default account settings are used when downloading. If you want the highest quality videos, configure these preferences at https://www.crunchyroll.com/acct/?action=video.
|
You need to authentication (`-p` and `-u`) to use Crunchy so you need to have an account on *CrunchyRool*. It is not possible to download non-free material without an account and premium subscription.
|
||||||
|
|
||||||
|
|
||||||
## Prerequisites
|
## Prerequisites
|
||||||
@@ -66,40 +66,75 @@ The [command-line interface](http://en.wikipedia.org/wiki/Command-line_interface
|
|||||||
|
|
||||||
Options:
|
Options:
|
||||||
|
|
||||||
-V, --version output the version number
|
-V, --version output the version number
|
||||||
-p, --pass <s> The password.
|
-p, --pass <s> The password.
|
||||||
-u, --user <s> The e-mail address or username.
|
-u, --user <s> The e-mail address or username.
|
||||||
-c, --cache Disables the cache.
|
-c, --cache Disables the cache.
|
||||||
-m, --merge Disables merging subtitles and videos.
|
-m, --merge Disables merging subtitles and videos.
|
||||||
-f, --format <s> The subtitle format. (Default: ass)
|
-e, --episodes <s> Episode list. Read documentation on how to use
|
||||||
-o, --output <s> The output path.
|
-f, --format <s> The subtitle format. (default: ass)
|
||||||
-s, --series <s> The series override.
|
-o, --output <s> The output path.
|
||||||
-n, --filename <s> The name override.
|
-s, --series <s> The series name override.
|
||||||
-t, --tag <s> The subgroup. (Default: CrunchyRoll) (default: CrunchyRoll)
|
-n, --nametmpl <s> Output name template (default: {SERIES_TITLE} - s{SEASON_NUMBER}e{EPISODE_NUMBER} - [{TAG}])
|
||||||
-r, --resolution <s> The video resolution. (Default: 1080 (360, 480, 720, 1080)) (default: 1080)
|
-t, --tag <s> The subgroup. (default: CrunchyRoll)
|
||||||
-g, --rebuildcrp Rebuild the crpersistant file.
|
-r, --resolution <s> The video resolution. (valid: 360, 480, 720, 1080) (default: 1080)
|
||||||
-b, --batch <s> Batch file (default: CrunchyRoll.txt)
|
-b, --batch <s> Batch file (default: CrunchyRoll.txt)
|
||||||
--verbose Make tool verbose
|
--verbose Make tool verbose
|
||||||
--retry <i> Number or time to retry fetching an episode. Default: 5 (default: 5)
|
--rebuildcrp Rebuild the crpersistant file.
|
||||||
-h, --help output usage information
|
--retry <i> Number or time to retry fetching an episode. (default: 5)
|
||||||
|
-h, --help output usage information
|
||||||
|
|
||||||
#### Batch-mode
|
#### Batch-mode
|
||||||
|
|
||||||
When no sequence of series addresses is provided, the batch-mode source file will be read (which is *CrunchyRoll.txt* in the current work directory. Each line in this file is processed as a seperate command-line statement. This makes it ideal to manage a large sequence of series addresses with variating command-line options or incremental episode updates.
|
When no sequence of series addresses is provided, the batch-mode source file will be read (which is *CrunchyRoll.txt* in the current work directory. Each line in this file is processed contain the URL of a series and can support some of the command line parameter (like `-e`). This makes it ideal to manage a large sequence of series addresses.
|
||||||
|
|
||||||
#### Examples
|
#### Examples
|
||||||
|
|
||||||
Download in batch-mode:
|
Download in batch-mode:
|
||||||
|
|
||||||
crunchy
|
You will need to create the batch file (default name is `CrunchyRoll.txt`):
|
||||||
|
|
||||||
Download *Fairy Tail* to the current work directory:
|
http://www.cr.com/tail-fairy
|
||||||
|
http://www.cr.com/gin-mama
|
||||||
|
http://www.cr.com/two-parts
|
||||||
|
// Just download episodes 3 to 42
|
||||||
|
http://www.cr.com/defense-of-dwarfs -e 3-42
|
||||||
|
|
||||||
crunchy http://www.crunchyroll.com/fairy-tail
|
Then launch crunchy:
|
||||||
|
|
||||||
|
crunchy -u login -p password http://www.cr.com/tail-fairy
|
||||||
|
|
||||||
|
Download *Tail Fairy* to the current work directory:
|
||||||
|
|
||||||
|
crunchy -u login -p password http://www.cr.com/tail-fairy
|
||||||
|
|
||||||
|
Download *Tail Fairy* to `C:\Anime`:
|
||||||
|
|
||||||
|
crunchy -u login -p password --output C:\Anime http://www.cr.com/tail-fairy
|
||||||
|
|
||||||
|
Download episode 42 of *Tail Fairy* to `C:\Anime`:
|
||||||
|
|
||||||
|
crunchy -u login -p password --output C:\Anime @http://www.cr.com/tail-fairy/episode-42-the-episode-which-dont-exist-665544
|
||||||
|
|
||||||
|
*Notice the '@' in front of the URL, it is there to tell Crunchy that the URL is an episode URL and not a series URL.*
|
||||||
|
|
||||||
|
or
|
||||||
|
|
||||||
|
crunchy -u login -p password --output C:\Anime http://www.cr.com/tail-fairy -e 42
|
||||||
|
|
||||||
|
Download episode 10 to 42 (both included) of *Tail Fairy*:
|
||||||
|
|
||||||
|
crunchy -u login -p password http://www.cr.com/tail-fairy -e 10-42
|
||||||
|
|
||||||
|
Download episode up to 42 (included) of *Tail Fairy*:
|
||||||
|
|
||||||
|
crunchy -u login -p password http://www.cr.com/tail-fairy -e -42
|
||||||
|
|
||||||
|
Download episodes starting from 42 to the last available of *Tail Fairy*:
|
||||||
|
|
||||||
|
crunchy -u login -p password http://www.cr.com/tail-fairy -e 42-
|
||||||
|
|
||||||
Download *Fairy Tail* to `C:\Anime`:
|
|
||||||
|
|
||||||
crunchy --output C:\Anime http://www.crunchyroll.com/fairy-tail
|
|
||||||
|
|
||||||
#### Command line parameters
|
#### Command line parameters
|
||||||
|
|
||||||
@@ -117,6 +152,7 @@ Download *Fairy Tail* to `C:\Anime`:
|
|||||||
|
|
||||||
##### Settings
|
##### Settings
|
||||||
|
|
||||||
|
* `-e or --episodes <s>` set an episode
|
||||||
* `-f or --format <s>` sets the subtitle format. (Default: ass)
|
* `-f or --format <s>` sets the subtitle format. (Default: ass)
|
||||||
* `-o or --output <s>` sets the output path.
|
* `-o or --output <s>` sets the output path.
|
||||||
* `-s or --series <s>` sets the series override.
|
* `-s or --series <s>` sets the series override.
|
||||||
|
|||||||
2257
package-lock.json
generated
2257
package-lock.json
generated
File diff suppressed because it is too large
Load Diff
21
package.json
21
package.json
@@ -15,19 +15,18 @@
|
|||||||
"engines": {
|
"engines": {
|
||||||
"node": ">=5.0"
|
"node": ">=5.0"
|
||||||
},
|
},
|
||||||
"version": "1.3.4",
|
"version": "1.3.7",
|
||||||
"bin": {
|
"bin": {
|
||||||
"crunchy": "./bin/crunchy",
|
"crunchy": "./bin/crunchy",
|
||||||
"crunchy.sh": "./bin/crunchy.sh"
|
"crunchy.sh": "./bin/crunchy.sh"
|
||||||
},
|
},
|
||||||
"dependencies": {
|
"dependencies": {
|
||||||
"@types/node": "^10.3.3",
|
"big-integer": "^1.6.32",
|
||||||
"big-integer": "^1.6.31",
|
|
||||||
"bluebird": "^3.5.1",
|
"bluebird": "^3.5.1",
|
||||||
"cheerio": "^0.22.0",
|
"cheerio": "^0.22.0",
|
||||||
"cloudscraper": "^1.5.0",
|
"cloudscraper": "^1.5.0",
|
||||||
"commander": "^2.15.1",
|
"commander": "^2.16.0",
|
||||||
"fs-extra": "^6.0.1",
|
"fs-extra": "^7.0.0",
|
||||||
"mkdirp": "^0.5.0",
|
"mkdirp": "^0.5.0",
|
||||||
"pjson": "^1.0.9",
|
"pjson": "^1.0.9",
|
||||||
"request": "^2.87.0",
|
"request": "^2.87.0",
|
||||||
@@ -35,15 +34,17 @@
|
|||||||
"xml2js": "^0.4.5"
|
"xml2js": "^0.4.5"
|
||||||
},
|
},
|
||||||
"devDependencies": {
|
"devDependencies": {
|
||||||
"@types/bluebird": "^3.5.20",
|
"@types/bluebird": "^3.5.23",
|
||||||
"@types/cheerio": "^0.22.7",
|
"@types/cheerio": "^0.22.8",
|
||||||
"@types/fs-extra": "^5.0.3",
|
"@types/fs-extra": "^5.0.4",
|
||||||
"@types/mkdirp": "^0.5.2",
|
"@types/mkdirp": "^0.5.2",
|
||||||
|
"@types/node": "^10.5.3",
|
||||||
"@types/request": "^2.47.1",
|
"@types/request": "^2.47.1",
|
||||||
"@types/request-promise": "^4.1.41",
|
"@types/request-promise": "^4.1.42",
|
||||||
"@types/xml2js": "^0.4.3",
|
"@types/xml2js": "^0.4.3",
|
||||||
|
"npm-check": "^5.7.1",
|
||||||
"tsconfig-lint": "^0.12.0",
|
"tsconfig-lint": "^0.12.0",
|
||||||
"tslint": "^5.10.0",
|
"tslint": "^5.11.0",
|
||||||
"typescript": "^2.9.2"
|
"typescript": "^2.9.2"
|
||||||
},
|
},
|
||||||
"scripts": {
|
"scripts": {
|
||||||
|
|||||||
130
src/batch.ts
130
src/batch.ts
@@ -62,6 +62,11 @@ export default function(args: string[], done: (err?: Error) => void)
|
|||||||
return done(err);
|
return done(err);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if (tasksArr[0].address === '')
|
||||||
|
{
|
||||||
|
return done();
|
||||||
|
}
|
||||||
|
|
||||||
let i = 0;
|
let i = 0;
|
||||||
|
|
||||||
(function next()
|
(function next()
|
||||||
@@ -71,13 +76,28 @@ export default function(args: string[], done: (err?: Error) => void)
|
|||||||
return done();
|
return done();
|
||||||
}
|
}
|
||||||
|
|
||||||
series(tasksArr[i].config, tasksArr[i].address, (errin) =>
|
if (config.debug)
|
||||||
|
{
|
||||||
|
log.dumpToDebug('Task ' + i, JSON.stringify(tasksArr[i]));
|
||||||
|
}
|
||||||
|
|
||||||
|
series(config, tasksArr[i], (errin) =>
|
||||||
{
|
{
|
||||||
if (errin)
|
if (errin)
|
||||||
{
|
{
|
||||||
|
if (errin.error)
|
||||||
|
{
|
||||||
|
/* Error from the request, so ignore it */
|
||||||
|
tasksArr[i].retry = 0;
|
||||||
|
}
|
||||||
|
|
||||||
if (tasksArr[i].retry <= 0)
|
if (tasksArr[i].retry <= 0)
|
||||||
{
|
{
|
||||||
console.error(errin);
|
log.error(JSON.stringify(errin));
|
||||||
|
if (config.debug)
|
||||||
|
{
|
||||||
|
log.dumpToDebug('BatchGiveUp', JSON.stringify(errin));
|
||||||
|
}
|
||||||
log.error('Cannot get episodes from "' + tasksArr[i].address + '", please rerun later');
|
log.error('Cannot get episodes from "' + tasksArr[i].address + '", please rerun later');
|
||||||
/* Go to the next on the list */
|
/* Go to the next on the list */
|
||||||
i += 1;
|
i += 1;
|
||||||
@@ -86,7 +106,11 @@ export default function(args: string[], done: (err?: Error) => void)
|
|||||||
{
|
{
|
||||||
if (config.verbose)
|
if (config.verbose)
|
||||||
{
|
{
|
||||||
console.error(errin);
|
log.error(JSON.stringify(errin));
|
||||||
|
}
|
||||||
|
if (config.debug)
|
||||||
|
{
|
||||||
|
log.dumpToDebug('BatchRetry', JSON.stringify(errin));
|
||||||
}
|
}
|
||||||
log.warn('Retrying to fetch episodes list from' + tasksArr[i].retry + ' / ' + config.retry);
|
log.warn('Retrying to fetch episodes list from' + tasksArr[i].retry + ' / ' + config.retry);
|
||||||
tasksArr[i].retry -= 1;
|
tasksArr[i].retry -= 1;
|
||||||
@@ -136,6 +160,73 @@ function split(value: string): string[]
|
|||||||
return pieces;
|
return pieces;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function get_min_filter(filter: string): number
|
||||||
|
{
|
||||||
|
if (filter !== undefined)
|
||||||
|
{
|
||||||
|
const tok = filter.split('-');
|
||||||
|
|
||||||
|
if (tok.length > 2)
|
||||||
|
{
|
||||||
|
log.error('Invalid episode filter \'' + filter + '\'');
|
||||||
|
process.exit(-1);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (tok[0] !== '')
|
||||||
|
{
|
||||||
|
return parseInt(tok[0], 10);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
function get_max_filter(filter: string): number
|
||||||
|
{
|
||||||
|
if (filter !== undefined)
|
||||||
|
{
|
||||||
|
const tok = filter.split('-');
|
||||||
|
|
||||||
|
if (tok.length > 2)
|
||||||
|
{
|
||||||
|
log.error('Invalid episode filter \'' + filter + '\'');
|
||||||
|
process.exit(-1);
|
||||||
|
}
|
||||||
|
|
||||||
|
if ((tok.length > 1) && (tok[1] !== ''))
|
||||||
|
{
|
||||||
|
/* We have a max value */
|
||||||
|
return parseInt(tok[1], 10);
|
||||||
|
}
|
||||||
|
else if ((tok.length === 1) && (tok[0] !== ''))
|
||||||
|
{
|
||||||
|
/* A single episode has been requested */
|
||||||
|
return parseInt(tok[0], 10);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return +Infinity;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Check that URL start with http:// or https://
|
||||||
|
* As for some reason request just return an error but a useless one when that happen so check it
|
||||||
|
* soon enough.
|
||||||
|
*/
|
||||||
|
function checkURL(address: string): boolean
|
||||||
|
{
|
||||||
|
if (address.startsWith('http:\/\/'))
|
||||||
|
{
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
if (address.startsWith('http:\/\/'))
|
||||||
|
{
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
log.error('URL ' + address + ' miss \'http:\/\/\' or \'https:\/\/\' => will be ignored');
|
||||||
|
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Parses the configuration or reads the batch-mode file for tasks.
|
* Parses the configuration or reads the batch-mode file for tasks.
|
||||||
*/
|
*/
|
||||||
@@ -143,11 +234,15 @@ function tasks(config: IConfigLine, batchPath: string, done: (err: Error, tasks?
|
|||||||
{
|
{
|
||||||
if (config.args.length)
|
if (config.args.length)
|
||||||
{
|
{
|
||||||
const configIn = config;
|
|
||||||
|
|
||||||
return done(null, config.args.map((addressIn) =>
|
return done(null, config.args.map((addressIn) =>
|
||||||
{
|
{
|
||||||
return {address: addressIn, config: configIn, retry: config.retry};
|
if (checkURL(addressIn))
|
||||||
|
{
|
||||||
|
return {address: addressIn, retry: config.retry,
|
||||||
|
episode_min: get_min_filter(config.episodes), episode_max: get_max_filter(config.episodes)};
|
||||||
|
}
|
||||||
|
|
||||||
|
return {address: '', retry: 0, episode_min: 0, episode_max: 0};
|
||||||
}));
|
}));
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -183,7 +278,11 @@ function tasks(config: IConfigLine, batchPath: string, done: (err: Error, tasks?
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
map.push({address: addressIn, config: lineConfig, retry: config.retry});
|
if (checkURL(addressIn))
|
||||||
|
{
|
||||||
|
map.push({address: addressIn, retry: lineConfig.retry,
|
||||||
|
episode_min: get_min_filter(lineConfig.episodes), episode_max: get_max_filter(lineConfig.episodes)});
|
||||||
|
}
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
done(null, map);
|
done(null, map);
|
||||||
@@ -203,18 +302,19 @@ function parse(args: string[]): IConfigLine
|
|||||||
// Disables
|
// Disables
|
||||||
.option('-c, --cache', 'Disables the cache.')
|
.option('-c, --cache', 'Disables the cache.')
|
||||||
.option('-m, --merge', 'Disables merging subtitles and videos.')
|
.option('-m, --merge', 'Disables merging subtitles and videos.')
|
||||||
|
// Episode filter
|
||||||
|
.option('-e, --episodes <s>', 'Episode list. Read documentation on how to use')
|
||||||
// Settings
|
// Settings
|
||||||
.option('-f, --format <s>', 'The subtitle format. (Default: ass)')
|
.option('-f, --format <s>', 'The subtitle format.', 'ass')
|
||||||
.option('-o, --output <s>', 'The output path.')
|
.option('-o, --output <s>', 'The output path.')
|
||||||
.option('-s, --series <s>', 'The series override.')
|
.option('-s, --series <s>', 'The series name override.')
|
||||||
.option('-n, --filename <s>', 'The name override.')
|
.option('-n, --nametmpl <s>', 'Output name template', '{SERIES_TITLE} - s{SEASON_NUMBER}e{EPISODE_NUMBER} - {EPISODE_TITLE} - [{TAG}]')
|
||||||
.option('-t, --tag <s>', 'The subgroup. (Default: CrunchyRoll)', 'CrunchyRoll')
|
.option('-t, --tag <s>', 'The subgroup.', 'CrunchyRoll')
|
||||||
.option('-r, --resolution <s>', 'The video resolution. (Default: 1080 (360, 480, 720, 1080))',
|
.option('-r, --resolution <s>', 'The video resolution. (valid: 360, 480, 720, 1080)', '1080')
|
||||||
'1080')
|
|
||||||
.option('-g, --rebuildcrp', 'Rebuild the crpersistant file.')
|
|
||||||
.option('-b, --batch <s>', 'Batch file', 'CrunchyRoll.txt')
|
.option('-b, --batch <s>', 'Batch file', 'CrunchyRoll.txt')
|
||||||
.option('--verbose', 'Make tool verbose')
|
.option('--verbose', 'Make tool verbose')
|
||||||
.option('--debug', 'Create a debug file. Use only if requested!')
|
.option('--debug', 'Create a debug file. Use only if requested!')
|
||||||
.option('--retry <i>', 'Number or time to retry fetching an episode. Default: 5', 5)
|
.option('--rebuildcrp', 'Rebuild the crpersistant file.')
|
||||||
|
.option('--retry <i>', 'Number or time to retry fetching an episode.', 5)
|
||||||
.parse(args);
|
.parse(args);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -74,11 +74,10 @@ function sanitiseFileName(str: string)
|
|||||||
*/
|
*/
|
||||||
function download(config: IConfig, page: IEpisodePage, player: IEpisodePlayer, done: (err: Error, ign: boolean) => void)
|
function download(config: IConfig, page: IEpisodePage, player: IEpisodePlayer, done: (err: Error, ign: boolean) => void)
|
||||||
{
|
{
|
||||||
let series = config.series || page.series;
|
const serieFolder = sanitiseFileName(config.series || page.series);
|
||||||
|
|
||||||
series = sanitiseFileName(series);
|
let fileName = sanitiseFileName(generateName(config, page));
|
||||||
let fileName = sanitiseFileName(name(config, page, series, ''));
|
let filePath = path.join(config.output || process.cwd(), serieFolder, fileName);
|
||||||
let filePath = path.join(config.output || process.cwd(), series, fileName);
|
|
||||||
|
|
||||||
if (fileExist(filePath + '.mkv'))
|
if (fileExist(filePath + '.mkv'))
|
||||||
{
|
{
|
||||||
@@ -95,13 +94,13 @@ function download(config: IConfig, page: IEpisodePage, player: IEpisodePlayer, d
|
|||||||
do
|
do
|
||||||
{
|
{
|
||||||
count = count + 1;
|
count = count + 1;
|
||||||
fileName = sanitiseFileName(name(config, page, series, '-' + count));
|
fileName = sanitiseFileName(generateName(config, page, '-' + count));
|
||||||
filePath = path.join(config.output || process.cwd(), series, fileName);
|
filePath = path.join(config.output || process.cwd(), serieFolder, fileName);
|
||||||
} while (fileExist(filePath + '.mkv'));
|
} while (fileExist(filePath + '.mkv'));
|
||||||
|
|
||||||
log.warn('Renaming to \'' + fileName + '\'...');
|
log.warn('Renaming to \'' + fileName + '\'...');
|
||||||
|
|
||||||
config.filename = fileName;
|
page.filename = fileName;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (config.rebuildcrp)
|
if (config.rebuildcrp)
|
||||||
@@ -114,6 +113,7 @@ function download(config: IConfig, page: IEpisodePage, player: IEpisodePlayer, d
|
|||||||
{
|
{
|
||||||
if (errM)
|
if (errM)
|
||||||
{
|
{
|
||||||
|
log.dispEpisode(fileName, 'Error...', true);
|
||||||
return done(errM, false);
|
return done(errM, false);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -122,6 +122,7 @@ function download(config: IConfig, page: IEpisodePage, player: IEpisodePlayer, d
|
|||||||
{
|
{
|
||||||
if (errDS)
|
if (errDS)
|
||||||
{
|
{
|
||||||
|
log.dispEpisode(fileName, 'Error...', true);
|
||||||
return done(errDS, false);
|
return done(errDS, false);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -133,6 +134,7 @@ function download(config: IConfig, page: IEpisodePage, player: IEpisodePlayer, d
|
|||||||
{
|
{
|
||||||
if (errDV)
|
if (errDV)
|
||||||
{
|
{
|
||||||
|
log.dispEpisode(fileName, 'Error...', true);
|
||||||
return done(errDV, false);
|
return done(errDV, false);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -148,6 +150,7 @@ function download(config: IConfig, page: IEpisodePage, player: IEpisodePlayer, d
|
|||||||
{
|
{
|
||||||
if (errVM)
|
if (errVM)
|
||||||
{
|
{
|
||||||
|
log.dispEpisode(fileName, 'Error...', true);
|
||||||
return done(errVM, false);
|
return done(errVM, false);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -211,19 +214,16 @@ function downloadVideo(config: IConfig, page: IEpisodePage, player: IEpisodePla
|
|||||||
/**
|
/**
|
||||||
* Names the file based on the config, page, series and tag.
|
* Names the file based on the config, page, series and tag.
|
||||||
*/
|
*/
|
||||||
function name(config: IConfig, page: IEpisodePage, series: string, extra: string)
|
function generateName(config: IConfig, page: IEpisodePage, extra = '')
|
||||||
{
|
{
|
||||||
const episodeNum = parseInt(page.episode, 10);
|
const episodeNum = parseInt(page.episode, 10);
|
||||||
const volumeNum = parseInt(page.volume, 10);
|
const volumeNum = parseInt(page.volume, 10);
|
||||||
const episode = (episodeNum < 10 ? '0' : '') + page.episode;
|
const episode = (episodeNum < 10 ? '0' : '') + page.episode;
|
||||||
const volume = (volumeNum < 10 ? '0' : '') + page.volume;
|
const volume = (volumeNum < 10 ? '0' : '') + page.volume;
|
||||||
const tag = config.tag || 'CrunchyRoll';
|
const tag = config.tag || 'CrunchyRoll';
|
||||||
|
const series = config.series || page.series;
|
||||||
|
|
||||||
if (!config.filename) {
|
return config.nametmpl
|
||||||
return page.series + ' - s' + volume + 'e' + episode + ' - [' + tag + ']' + extra;
|
|
||||||
}
|
|
||||||
|
|
||||||
return config.filename
|
|
||||||
.replace(/{EPISODE_ID}/g, page.id.toString())
|
.replace(/{EPISODE_ID}/g, page.id.toString())
|
||||||
.replace(/{EPISODE_NUMBER}/g, episode)
|
.replace(/{EPISODE_NUMBER}/g, episode)
|
||||||
.replace(/{SEASON_NUMBER}/g, volume)
|
.replace(/{SEASON_NUMBER}/g, volume)
|
||||||
@@ -277,10 +277,21 @@ function scrapePage(config: IConfig, address: string, done: (err: Error, page?:
|
|||||||
const episodeTitle = $('#showmedia_about_name').text().replace(/[“”]/g, '');
|
const episodeTitle = $('#showmedia_about_name').text().replace(/[“”]/g, '');
|
||||||
const data = regexp.exec(look);
|
const data = regexp.exec(look);
|
||||||
|
|
||||||
|
if (config.debug)
|
||||||
|
{
|
||||||
|
log.dumpToDebug('episode page', $.html());
|
||||||
|
}
|
||||||
|
|
||||||
if (!swf || !data)
|
if (!swf || !data)
|
||||||
{
|
{
|
||||||
log.warn('Somethig unexpected in the page at ' + address + ' (data are: ' + look + ')');
|
log.warn('Somethig unexpected in the page at ' + address + ' (data are: ' + look + ')');
|
||||||
log.warn('Setting Season to ’0’ and episode to ’0’...');
|
log.warn('Setting Season to ’0’ and episode to ’0’...');
|
||||||
|
|
||||||
|
if (config.debug)
|
||||||
|
{
|
||||||
|
log.dumpToDebug('episode unexpected', look);
|
||||||
|
}
|
||||||
|
|
||||||
done(null, {
|
done(null, {
|
||||||
episode: '0',
|
episode: '0',
|
||||||
id: epId,
|
id: epId,
|
||||||
@@ -289,6 +300,7 @@ function scrapePage(config: IConfig, address: string, done: (err: Error, page?:
|
|||||||
title: episodeTitle,
|
title: episodeTitle,
|
||||||
swf: swf[1],
|
swf: swf[1],
|
||||||
volume: '0',
|
volume: '0',
|
||||||
|
filename: '',
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
else
|
else
|
||||||
@@ -301,6 +313,7 @@ function scrapePage(config: IConfig, address: string, done: (err: Error, page?:
|
|||||||
title: episodeTitle,
|
title: episodeTitle,
|
||||||
swf: swf[1],
|
swf: swf[1],
|
||||||
volume: data[2] || '1',
|
volume: data[2] || '1',
|
||||||
|
filename: '',
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
@@ -367,6 +380,11 @@ function scrapePlayer(config: IConfig, address: string, id: number, done: (err:
|
|||||||
});
|
});
|
||||||
} catch (parseError)
|
} catch (parseError)
|
||||||
{
|
{
|
||||||
|
if (config.debug)
|
||||||
|
{
|
||||||
|
log.dumpToDebug('player scrape', parseError);
|
||||||
|
}
|
||||||
|
|
||||||
done(parseError);
|
done(parseError);
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|||||||
3
src/interface/IConfig.d.ts
vendored
3
src/interface/IConfig.d.ts
vendored
@@ -5,11 +5,12 @@ interface IConfig {
|
|||||||
// Disables
|
// Disables
|
||||||
cache?: boolean;
|
cache?: boolean;
|
||||||
merge?: boolean;
|
merge?: boolean;
|
||||||
|
episodes?: string;
|
||||||
// Settings
|
// Settings
|
||||||
format?: string;
|
format?: string;
|
||||||
output?: string;
|
output?: string;
|
||||||
series?: string;
|
series?: string;
|
||||||
filename?: string;
|
nametmpl?: string;
|
||||||
tag?: string;
|
tag?: string;
|
||||||
resolution?: string;
|
resolution?: string;
|
||||||
video_format?: string;
|
video_format?: string;
|
||||||
|
|||||||
3
src/interface/IConfigTask.d.ts
vendored
3
src/interface/IConfigTask.d.ts
vendored
@@ -1,5 +1,6 @@
|
|||||||
interface IConfigTask {
|
interface IConfigTask {
|
||||||
address: string;
|
address: string;
|
||||||
config: IConfigLine;
|
|
||||||
retry: number;
|
retry: number;
|
||||||
|
episode_min: number;
|
||||||
|
episode_max: number;
|
||||||
}
|
}
|
||||||
|
|||||||
1
src/interface/IEpisodePage.d.ts
vendored
1
src/interface/IEpisodePage.d.ts
vendored
@@ -6,4 +6,5 @@ interface IEpisodePage {
|
|||||||
season: string;
|
season: string;
|
||||||
title: string;
|
title: string;
|
||||||
swf: string;
|
swf: string;
|
||||||
|
filename: string;
|
||||||
}
|
}
|
||||||
|
|||||||
12
src/log.ts
12
src/log.ts
@@ -41,15 +41,9 @@ export function dumpToDebug(what: string, data: any, create = false)
|
|||||||
{
|
{
|
||||||
if (create)
|
if (create)
|
||||||
{
|
{
|
||||||
fs.writeFile('debug.txt', '>>>>>>>> ' + what + ':\n' + data + '\n<<<<<<<<\n', (err) =>
|
fs.writeFileSync('debug.txt', '>>>>>>>> ' + what + ':\n' + data + '\n<<<<<<<<\n');
|
||||||
{
|
|
||||||
if (err) throw err;
|
|
||||||
});
|
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
fs.appendFile('debug.txt', '>>>>>>>> ' + what + ':\n' + data + '\n<<<<<<<<\n', (err) =>
|
fs.appendFileSync('debug.txt', '>>>>>>>> ' + what + ':\n' + data + '\n<<<<<<<<\n');
|
||||||
{
|
}
|
||||||
if (err) throw err;
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -82,7 +82,7 @@ function login(sessionId: string, user: string, pass: string): Promise<any>
|
|||||||
/**
|
/**
|
||||||
* Performs a GET request for the resource.
|
* Performs a GET request for the resource.
|
||||||
*/
|
*/
|
||||||
export function get(config: IConfig, options: string|request.Options, done: (err: Error, result?: string) => void)
|
export function get(config: IConfig, options: string|request.Options, done: (err: any, result?: string) => void)
|
||||||
{
|
{
|
||||||
authenticate(config, (err) =>
|
authenticate(config, (err) =>
|
||||||
{
|
{
|
||||||
@@ -91,7 +91,7 @@ export function get(config: IConfig, options: string|request.Options, done: (err
|
|||||||
return done(err);
|
return done(err);
|
||||||
}
|
}
|
||||||
|
|
||||||
cloudscraper.request(modify(options, 'GET'), (error: Error, response: any, body: any) =>
|
cloudscraper.request(modify(options, 'GET'), (error: any, response: any, body: any) =>
|
||||||
{
|
{
|
||||||
if (error) return done(error);
|
if (error) return done(error);
|
||||||
done(null, typeof body === 'string' ? body : String(body));
|
done(null, typeof body === 'string' ? body : String(body));
|
||||||
@@ -223,4 +223,4 @@ function modify(options: string|request.Options, reqMethod: string): request.Opt
|
|||||||
url: options.toString(),
|
url: options.toString(),
|
||||||
method: reqMethod
|
method: reqMethod
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -27,7 +27,7 @@ function fileExist(path: string)
|
|||||||
/**
|
/**
|
||||||
* Streams the series to disk.
|
* Streams the series to disk.
|
||||||
*/
|
*/
|
||||||
export default function(config: IConfig, address: string, done: (err: Error) => void)
|
export default function(config: IConfig, task: IConfigTask, done: (err: any) => void)
|
||||||
{
|
{
|
||||||
const persistentPath = path.join(config.output || process.cwd(), persistent);
|
const persistentPath = path.join(config.output || process.cwd(), persistent);
|
||||||
|
|
||||||
@@ -41,25 +41,43 @@ export default function(config: IConfig, address: string, done: (err: Error) =>
|
|||||||
{
|
{
|
||||||
const cache = config.cache ? {} : JSON.parse(contents || '{}');
|
const cache = config.cache ? {} : JSON.parse(contents || '{}');
|
||||||
|
|
||||||
page(config, address, (errP, page) =>
|
pageScrape(config, task, (errP, page) =>
|
||||||
{
|
{
|
||||||
if (errP)
|
if (errP)
|
||||||
{
|
{
|
||||||
|
const reqErr = errP.error;
|
||||||
|
if ((reqErr.syscall === 'getaddrinfo') && (reqErr.errno === 'ENOTFOUND'))
|
||||||
|
{
|
||||||
|
log.error('The URL \'' + task.address + '\' is invalid, please check => I\'m ignoring it.');
|
||||||
|
}
|
||||||
|
|
||||||
return done(errP);
|
return done(errP);
|
||||||
}
|
}
|
||||||
|
|
||||||
let i = 0;
|
let i = 0;
|
||||||
(function next()
|
(function next()
|
||||||
{
|
{
|
||||||
|
if (config.debug)
|
||||||
|
{
|
||||||
|
log.dumpToDebug('Episode ' + i, JSON.stringify(page.episodes[i]));
|
||||||
|
}
|
||||||
|
|
||||||
if (i >= page.episodes.length) return done(null);
|
if (i >= page.episodes.length) return done(null);
|
||||||
download(cache, config, address, page.episodes[i], (errD, ignored) =>
|
download(cache, config, task, page.episodes[i], (errD, ignored) =>
|
||||||
{
|
{
|
||||||
if (errD)
|
if (errD)
|
||||||
{
|
{
|
||||||
|
/* Check if domain is valid */
|
||||||
|
const reqErr = errD.error;
|
||||||
|
if ((reqErr.syscall === 'getaddrinfo') && (reqErr.errno === 'ENOTFOUND'))
|
||||||
|
{
|
||||||
|
page.episodes[i].retry = 0;
|
||||||
|
log.error('The URL \'' + task.address + '\' is invalid, please check => I\'m ignoring it.');
|
||||||
|
}
|
||||||
|
|
||||||
if (page.episodes[i].retry <= 0)
|
if (page.episodes[i].retry <= 0)
|
||||||
{
|
{
|
||||||
log.dispEpisode(config.filename, 'Error...', true);
|
log.error(JSON.stringify(errD));
|
||||||
log.error(errD);
|
|
||||||
log.error('Cannot fetch episode "s' + page.episodes[i].volume + 'e' + page.episodes[i].episode +
|
log.error('Cannot fetch episode "s' + page.episodes[i].volume + 'e' + page.episodes[i].episode +
|
||||||
'", please rerun later');
|
'", please rerun later');
|
||||||
/* Go to the next on the list */
|
/* Go to the next on the list */
|
||||||
@@ -67,13 +85,12 @@ export default function(config: IConfig, address: string, done: (err: Error) =>
|
|||||||
}
|
}
|
||||||
else
|
else
|
||||||
{
|
{
|
||||||
log.dispEpisode(config.filename, 'Error...', true);
|
|
||||||
if ((config.verbose) || (config.debug))
|
if ((config.verbose) || (config.debug))
|
||||||
{
|
{
|
||||||
if (config.debug)
|
if (config.debug)
|
||||||
{
|
{
|
||||||
log.dumpToDebug('series address', address);
|
log.dumpToDebug('series address', task.address);
|
||||||
log.dumpToDebug('series error', errD.stack || errD);
|
log.dumpToDebug('series error', JSON.stringify(errD));
|
||||||
log.dumpToDebug('series data', JSON.stringify(page));
|
log.dumpToDebug('series data', JSON.stringify(page));
|
||||||
}
|
}
|
||||||
log.error(errD);
|
log.error(errD);
|
||||||
@@ -116,10 +133,18 @@ export default function(config: IConfig, address: string, done: (err: Error) =>
|
|||||||
* Downloads the episode.
|
* Downloads the episode.
|
||||||
*/
|
*/
|
||||||
function download(cache: {[address: string]: number}, config: IConfig,
|
function download(cache: {[address: string]: number}, config: IConfig,
|
||||||
baseAddress: string, item: ISeriesEpisode,
|
task: IConfigTask, item: ISeriesEpisode,
|
||||||
done: (err: Error, ign: boolean) => void)
|
done: (err: any, ign: boolean) => void)
|
||||||
{
|
{
|
||||||
const address = url.resolve(baseAddress, item.address);
|
const episodeNumber = parseInt(item.episode, 10);
|
||||||
|
|
||||||
|
if ( (episodeNumber < task.episode_min) ||
|
||||||
|
(episodeNumber > task.episode_max) )
|
||||||
|
{
|
||||||
|
return done(null, false);
|
||||||
|
}
|
||||||
|
|
||||||
|
const address = url.resolve(task.address, item.address);
|
||||||
|
|
||||||
if (cache[address])
|
if (cache[address])
|
||||||
{
|
{
|
||||||
@@ -141,14 +166,14 @@ function download(cache: {[address: string]: number}, config: IConfig,
|
|||||||
/**
|
/**
|
||||||
* Requests the page and scrapes the episodes and series.
|
* Requests the page and scrapes the episodes and series.
|
||||||
*/
|
*/
|
||||||
function page(config: IConfig, address: string, done: (err: Error, result?: ISeries) => void)
|
function pageScrape(config: IConfig, task: IConfigTask, done: (err: any, result?: ISeries) => void)
|
||||||
{
|
{
|
||||||
if (address[0] === '@')
|
if (task.address[0] === '@')
|
||||||
{
|
{
|
||||||
log.info('Trying to fetch from ' + address.substr(1));
|
log.info('Trying to fetch from ' + task.address.substr(1));
|
||||||
const episodes: ISeriesEpisode[] = [];
|
const episodes: ISeriesEpisode[] = [];
|
||||||
episodes.push({
|
episodes.push({
|
||||||
address: address.substr(1),
|
address: task.address.substr(1),
|
||||||
episode: '',
|
episode: '',
|
||||||
volume: 0,
|
volume: 0,
|
||||||
retry: config.retry,
|
retry: config.retry,
|
||||||
@@ -158,21 +183,26 @@ function page(config: IConfig, address: string, done: (err: Error, result?: ISer
|
|||||||
else
|
else
|
||||||
{
|
{
|
||||||
let episodeCount = 0;
|
let episodeCount = 0;
|
||||||
my_request.get(config, address, (err, result) => {
|
my_request.get(config, task.address, (err, result) => {
|
||||||
if (err) {
|
if (err)
|
||||||
|
{
|
||||||
return done(err);
|
return done(err);
|
||||||
}
|
}
|
||||||
|
|
||||||
const $ = cheerio.load(result);
|
const $ = cheerio.load(result);
|
||||||
const title = $('span[itemprop=name]').text();
|
const title = $('span[itemprop=name]').text();
|
||||||
|
|
||||||
|
if (config.debug)
|
||||||
|
{
|
||||||
|
log.dumpToDebug('serie page', $.html());
|
||||||
|
}
|
||||||
|
|
||||||
if (!title) {
|
if (!title) {
|
||||||
if (config.debug)
|
if (config.debug)
|
||||||
{
|
{
|
||||||
log.dumpToDebug('inval page addr', address);
|
log.dumpToDebug('missing title', task.address);
|
||||||
log.dumpToDebug('inval page data', $);
|
|
||||||
}
|
}
|
||||||
return done(new Error('Invalid page.(' + address + ')'));
|
return done(new Error('Invalid page.(' + task.address + ')'));
|
||||||
}
|
}
|
||||||
|
|
||||||
log.info('Checking availability for ' + title);
|
log.info('Checking availability for ' + title);
|
||||||
|
|||||||
Reference in New Issue
Block a user